search

theoremlp / rules_mypy

a Bazel mypy aspect
Apache License 2.0
18 stars 6 forks source link

Not finding bazel-installed python packages #21

Open michael-christen opened 2 months ago

michael-christen commented 2 months ago

I've noticed that packages installed outside of the typical requirements_lock.txt path, purely from other bazel repositories aren't being found when running mypy. If you look at the python_imports for a py_binary it seems like we'd want to append that list to MYPY_PATH.

Curious if others are seeing this or if it's just an issue with my setup.

mark-thm commented 2 months ago

When you say python_imports do you mean the values provided to the imports attribute of a py_binary?

Could you clarify Bazel, rules_python and rules_mypy versions you're running? If possible, a small repro would also be helpful (feel free to make a PR that adds another example or modifies the existing one to illustrate the issue).

michael-christen commented 2 months ago

Of course, thanks for the quick response.

Check out https://github.com/michael-christen/toolbox/tree/reproduce_bazel_dependency_issue and run

 bazel build examples/pigweed/tools:console

Here's an example of the output I'm getting:

INFO: Analyzed target //examples/pigweed/tools:console (0 packages loaded, 0 targets configured).
ERROR: /home/mchristen/devel/toolbox/examples/pigweed/tools/BUILD:7:10: mypy //examples/pigweed/tools:console failed: (Exit 1): mypy_cli failed: error executing mypy command (from target //examples/pigweed/tools:console)
  (cd /home/mchristen/.local/tmp/bazel/_bazel_mchristen/eabc9c58e7a2790b61df5bad4df6e1e8/sandbox/linux-sandbox/6/execroot/_main && \
  exec env - \
    MYPYPATH=external/pigweed~/site-packages:bazel-out/k8-fastbuild/bin \
    MYPY_FORCE_COLOR=1 \
    PATH=/bin:/usr/bin:/usr/local/bin \
    PYTHONNOUSERSITE=1 \
    TERM=xterm-256color \
  bazel-out/k8-opt-exec-ST-718e85fced9c/bin/tools/mypy_cli --output bazel-out/k8-fastbuild/bin/examples/pigweed/tools/console.mypy_stdout --cache-dir bazel-out/k8-fastbuild/bin/examples/pigweed/tools/console.mypy_cache examples/pigweed/tools/console.py --mypy-ini mypy.ini)
# Configuration: fcff3044cd2b035f8297cc21e327a3585539f89acaf5f2fa0123017cdf3e85a3
# Execution platform: @@platforms//host:host

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
examples/pigweed/tools/console.py:20: error: Cannot find implementation or library stub for module named "pw_cli"  [import-not-found]
examples/pigweed/tools/console.py:21: error: Cannot find implementation or library stub for module named "pw_system.console"  [import-not-found]
examples/pigweed/tools/console.py:21: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
examples/pigweed/tools/console.py:21: error: Cannot find implementation or library stub for module named "pw_system"  [import-not-found]
examples/pigweed/tools/console.py:22: error: Cannot find implementation or library stub for module named "pw_system.device"  [import-not-found]
examples/pigweed/tools/console.py:23: error: Cannot find implementation or library stub for module named "pw_system.device_connection"  [import-not-found]
Found 5 errors in 1 file (checked 1 source file)
INFO: Elapsed time: 3.154s, Critical Path: 2.85s
INFO: 2 processes: 2 internal.
ERROR: Build did NOT complete successfully

If you look at the generated binary bazel-out/k8-fastbuild/bin/examples/pigweed/tools/console, it has a reference to the paths needed, eg)

...
  python_imports = 'protobuf~/python/python:protobuf~/python:_main/external/pigweed~/pw_protobuf/_virtual_imports/common_proto:rules_python~~pip~python_packages_311_wcwidth/site-packages:rules_python~~pip~python_packages_311_prompt_toolkit/site-packages:rules_python~~pip~python_packages_311_psutil/site-packages:pigweed~/pw_cli/py:rules_python~~pip~python_packages_311_pyyaml/site-packages:pigweed~/pw_config_loader/py:pigweed~/pw_log_tokenized/py:rules_python~~pip~python_packages_311_frozenlist/site-packages:rules_python~~pip~python_packages_311_aiosignal/site-packages:rules_python~~pip~python_packages_311_attrs/site-packages:rules_python~~pip~python_packages_311_multidict/site-packages:rules_python~~pip~python_packages_311_idna/site-packages:rules_python~~pip~python_packages_311_yarl/site-packages:rules_python~~pip~python_packages_311_aiohttp/site-packages:rules_python~~pip~python_packages_311_markupsafe/site-packages:rules_python~~pip~python_packages_311_jinja2/site-packages:rules_python~~pip~python_packages_311_appdirs/site-packages:rules_python~~pip~python_packages_311_parso/site-packages:rules_python~~pip~python_packages_311_jedi/site-packages:rules_python~~pip~python_packages_311_pygments/site-packages:rules_python~~pip~python_packages_311_ptpython/site-packages:rules_python~~pip~python_packages_311_pyperclip/site-packages:rules_python~~pip~python_packages_311_websockets/site-packages:pigweed~/pw_console/py:pigweed~/pw_protobuf_compiler/py:pigweed~/pw_status/py:pigweed~/pw_rpc/py:rules_python~~pip~python_packages_311_pyserial/site-packages:pigweed~/pw_hdlc/py:_main/external/protobuf~/src/google/protobuf/_virtual_imports/descriptor_proto:_main/external/pigweed~/pw_tokenizer/_virtual_imports/tokenizer_proto:_main/external/pigweed~/pw_log/_virtual_imports/log_proto:pigweed~/pw_tokenizer/py:pigweed~/pw_log/py:pigweed~/pw_log_rpc/py:_main/external/pigweed~/pw_metric/_virtual_imports/metric_proto:pigweed~/pw_metric/py:_main/external/pigweed~/pw_snapshot/_virtual_imports/metadata_proto:pigweed~/pw_snapshot/py:_main/external/pigweed~/pw_chrono/_virtual_imports/chrono_proto:pigweed~/pw_chrono/py:_main/external/pigweed~/pw_cpu_exception_cortex_m/_virtual_imports/cpu_state_protos:rules_python~:pigweed~/pw_symbolizer/py:pigweed~/pw_cpu_exception_cortex_m/py:pigweed~/pw_cpu_exception_cortex_m:_main/external/pigweed~/pw_cpu_exception_risc_v/_virtual_imports/cpu_state_protos:pigweed~/pw_cpu_exception_risc_v/py:pigweed~/pw_cpu_exception_risc_v:_main/external/pigweed~/pw_thread/_virtual_imports/thread_proto:_main/external/pigweed~/pw_snapshot/_virtual_imports/snapshot_proto:pigweed~/pw_thread/py:rules_python~~pip~python_packages_311_pyelftools/site-packages:pigweed~/pw_build_info/py:_main/external/pigweed~/pw_system/_virtual_imports/device_service_proto:_main/external/pigweed~/pw_thread/_virtual_imports/thread_snapshot_service_proto:pigweed~/pw_trace/py:_main/external/pigweed~/pw_trace_tokenized/_virtual_imports/protos:pigweed~/pw_trace_tokenized/py:pigweed~/pw_transfer/py:_main/external/pigweed~/pw_unit_test/_virtual_imports/unit_test_proto:pigweed~/pw_unit_test/py:rules_python~~pip~python_packages_311_backcall/site-packages:rules_python~~pip~python_packages_311_decorator/site-packages:rules_python~~pip~python_packages_311_traitlets/site-packages:rules_python~~pip~python_packages_311_matplotlib_inline/site-packages:rules_python~~pip~python_packages_311_ptyprocess/site-packages:rules_python~~pip~python_packages_311_pexpect/site-packages:rules_python~~pip~python_packages_311_pickleshare/site-packages:rules_python~~pip~python_packages_311_six/site-packages:rules_python~~pip~python_packages_311_asttokens/site-packages:rules_python~~pip~python_packages_311_executing/site-packages:rules_python~~pip~python_packages_311_pure_eval/site-packages:rules_python~~pip~python_packages_311_stack_data/site-packages:rules_python~~pip~python_packages_311_ipython/site-packages:pigweed~/pw_system/py'
...

But, if you look at the mypy_cache, it doesn't seem to be picking this bazel installed python repository up.

± ls bazel-out/k8-fastbuild/bin/examples/pigweed/tools/console.mypy_cache/3.12/                                                                                                                            1s
abc.data.json       codecs.data.json            dataclasses.meta.json  google                  posixpath.data.json      sre_constants.meta.json  threading.data.json  typing_extensions.data.json
abc.meta.json       _codecs.meta.json           email                  importlib               posixpath.meta.json      sre_parse.data.json      threading.meta.json  typing_extensions.meta.json
argparse.data.json  codecs.meta.json            enum.data.json         io.data.json            re.data.json             sre_parse.meta.json      _thread.meta.json    typing.meta.json
argparse.meta.json  collections                 enum.meta.json         io.meta.json            re.meta.json             string.data.json         time.data.json       zipfile
_ast.data.json      _collections_abc.data.json  examples               logging                 resource.data.json       string.meta.json         time.meta.json
_ast.meta.json      _collections_abc.meta.json  examples.data.json     os                      resource.meta.json       subprocess.data.json     types.data.json
builtins.data.json  contextlib.data.json        examples.meta.json     pathlib.data.json       sre_compile.data.json    subprocess.meta.json     _typeshed
builtins.meta.json  contextlib.meta.json        genericpath.data.json  pathlib.meta.json       sre_compile.meta.json    sys                      types.meta.json
_codecs.data.json   dataclasses.data.json       genericpath.meta.json  @plugins_snapshot.json  sre_constants.data.json  _thread.data.json        typing.data.json

Bazel Version is ~ 7, a specific commit 0ddcfd327ffd012d348deeae08ec0836409706ad rules_python is 0.34.0 rules_mypy is 0.6.0

Thanks for the help, lmk if there's anything I can do.

keith commented 2 weeks ago

if the issue is the imports, you can test with https://github.com/theoremlp/rules_mypy/pull/37