"Missing input file" error during Bazel build

jtilahun commented 1 year ago

Description of the bug:

We noticed an unexplained failure in CI during a Bazel build. The build failed with an error message stating that a Python cache file was missing. The excerpt is:

ERROR: /buildbot/astranis-monorepo-sw/build/sw/BUILD:3:25: Middleman _middlemen/sw_Srequirements_Utest-runfiles failed: missing input file '@pypi__pip//:pip/_internal/models/__pycache__/candidate.cpython-38.pyc.140279245062608'
ERROR: /buildbot/astranis-monorepo-sw/build/sw/BUILD:3:25: Middleman _middlemen/sw_Srequirements_Utest-runfiles failed: 1 input file(s) do not exist
ERROR: /buildbot/astranis-monorepo-sw/build/sw/BUILD:3:25 Middleman _middlemen/sw_Srequirements_Utest-runfiles failed: 1 input file(s) do not exist

Logs and corresponding BUILD file attached.

missing_input_file.txt BUILD

Which category does this issue belong to?

Python Rules

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

This bug has been observed but has not been reproduced. See description above.

Which operating system are you running Bazel on?

Linux Ubuntu 20.04

What is the output of `bazel info release`?

release 6.3.0

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

No response

What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

pauliusjwix commented 9 months ago

Facing this issue on bazel 6.4.0 as well

jfirebaugh commented 7 months ago

Very similar errors in 7.1.1:

ERROR: /home/circleci/figma/fullscreen/scripts/BUILD.bazel:34:10: Middleman _middlemen/fullscreen_Sscripts_Sgenerate_Ustyles-runfiles failed: missing input file '@@python3_x86_64-unknown-linux-gnu//:lib/python3.11/encodings/__pycache__/__init__.cpython-311.pyc.139763498763728'
ERROR: /home/circleci/figma/fullscreen/scripts/BUILD.bazel:34:10: Middleman _middlemen/fullscreen_Sscripts_Sgenerate_Ustyles-runfiles failed: 1 input file(s) do not exist
ERROR: /home/circleci/figma/fullscreen/scripts/BUILD.bazel:34:10 Middleman _middlemen/fullscreen_Sscripts_Sgenerate_Ustyles-runfiles failed: 1 input file(s) do not exist

There seems to be something non-hermetic or otherwise buggy related to __pycache__ and pyc files.

jfirebaugh commented 7 months ago

On Bazel slack I was pointed to: https://rules-python.readthedocs.io/en/latest/api/extensions/python.html#python-toolchain-ignore-root-user-error

The WORKSPACE equivalent is the python_register_toolchains attribute of python_register_toolchains, which we had set to True. As the docs say:

If True, the Python runtime installation is read-write. This allows the interpreter to create pyc files for the standard library, but, because they are created as needed, it adversely affects Bazel’s ability to cache the runtime and can result in spurious build failures.

jtilahun commented 6 months ago

The WORKSPACE equivalent is the python_register_toolchains attribute of python_register_toolchains, which we had set to True.

I think you meant to refer to the ignore_root_user_error attribute of python_register_toolchains, if I'm not mistaken?

At the time of this CI failure, we did not set ignore_root_user_error to True ourselves. Here is the relevant excerpt from WORKSPACE at the time:

http_archive(
    name = "rules_python",
    sha256 = "9fcf91dbcc31fde6d1edb15f117246d912c33c36f44cf681976bd886538deba6",
    strip_prefix = "rules_python-0.8.0",
    url = "https://github.com/bazelbuild/rules_python/archive/refs/tags/0.8.0.tar.gz",
)

load("@rules_python//python:repositories.bzl", "python_register_toolchains")

python_register_toolchains(
    name = "python3_8",
    python_version = "3.8.12",
)

Also, as indicated in the excerpt above, the version of rules_python at the time was 0.8.0.

jfirebaugh commented 6 months ago

I think you meant to refer to the ignore_root_user_error attribute of python_register_toolchains, if I'm not mistaken?

Yes, thank you for catching that.

I can also now confirm that this error can occur even with the default (False) value of ignore_root_user_error.

ERROR: /home/[redacted]/BUILD.bazel:190:25: Middleman _middlemen/[redacted].update-runfiles failed: missing input file '@@pypi__pip//:pip/_internal/commands/__pycache__/__init__.cpython-311.pyc.140035533023856'
ERROR: /home/[redacted]/BUILD.bazel:190:25: Middleman _middlemen/[redacted].update-runfiles failed: 1 input file(s) do not exist

Bazel 7.1.1, rules_python 0.20.0.

jfirebaugh commented 3 months ago

Based on a tip in Bazel Slack, I upgraded to rules_python 0.34.0 and have not seen the issue since then.

bazelbuild / bazel