[bazel] [rules_foreign_cc] ModuleNotFoundError: No module named 'tools'

[allsey87]

When building a simple hello world binary with Meson, I am hitting the following error

mesonbuild.utils.universal.EnvironmentException: Unknown compiler(s): [['/home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/sandbox/processwrapper-sandbox/9/execroot/_main/external/emsdk/emscripten_toolchain/emcc.sh']]

Digging a bit deeper, it seems emcc.py --version is failing and unable to find the tools module.

The Meson build system
Version: 1.1.1
Source dir: /home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/sandbox/processwrapper-sandbox/9/execroot/_main/testing/hellomeson
Build dir: /home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/sandbox/processwrapper-sandbox/9/execroot/_main/bazel-out/k8-fastbuild/bin/testing/test_meson.build_tmpdir
Build type: native build
Project name: tutorial
Project version: undefined
running cmd: ['/home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/sandbox/processwrapper-sandbox/9/execroot/_main/external/emsdk/emscripten_toolchain/emcc.sh', '--version']
out: 
err: Traceback (most recent call last):
  File "/home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/sandbox/processwrapper-sandbox/9/execroot/_main/external/emscripten_bin_linux/emscripten/emcc.py", line 23, in <module>
    from tools.toolchain_profiler import ToolchainProfiler
ModuleNotFoundError: No module named 'tools'

Perhaps this isn't included in the compiler_files filegroup which is passed to the toolchain?

[allsey87]

Perhaps this isn't included in the compiler_files filegroup which is passed to the toolchain?

This can't be the case, this module import is the first statement in emcc.py...

I checked the contents of external/emscripten_bin_linux/emscripten/ inside the sandbox and emcc.py + the tools directory is there. I also tried running emcc.py from my workspace and it didn't seem to have a problem with importing the tools module (although it complains that there is no config file or LLVM_ROOT set as expected).

python3 /home/develop/home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/sandbox/processwrapper-sandbox/10/execroot/_main/external/emscripten_bin_linux/emscripten/emcc.py --version
emcc: warning: config file not found: /home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/external/emscripten_bin_linux/emscripten/.emscripten.  You can create one by hand or run `emcc --generate-config`
emcc: error: LLVM_ROOT not set in config (/home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/external/emscripten_bin_linux/emscripten/.emscripten), and `clang` not found in PATH

[walkingeyerobot]

I don't quite understand here. My understanding is that Meson is a build system like bazel. Are you using both at once? Is one invoking the other?

[allsey87]

Indeed, there is often a need in Bazel to build software that is primarily built using other build systems. In this case, I am looking at building numpy which uses the Meson build system. rules_foreign_cc allows you to build this software in Bazel by integrating support for calling into other build systems such as automake, cmake, meson etc and exporting the build artefacts from them for use in other rules.

[allsey87]

Finally got to the root cause of why the configure_make rule from rules_foreign_cc works but the meson rule does not.

Since Meson is written in Python, it uses Bazel's py_binary which sets PYTHONSAFEPATH by default. This environment variable is inherited by emcc.sh and emcc.py, the latter relying on having the containing directory of emcc.py added to sys.path. However PYTHONSAFEPATH disables this behaviour and hence the import of the tools module fails.

[allsey87]

@walkingeyerobot I think the solution to this problem is to unset the PYTHONSAFEPATH environment variable in env.sh and env.bat. I have tested this and it works. See #1423.

[kleisauke]

FWIW, I see a similar issue when trying to run Emscripten's test suite with Python 3.12.

$ python -V
Python 3.12.4
$ ./test/runner other.*code_size* other.*metadce* --rebase
Traceback (most recent call last):
  File "/home/kleisauke/emscripten/./test/runner.py", line 38, in <module>
    import jsrun
  File "/home/kleisauke/emscripten/test/jsrun.py", line 6, in <module>
    import common
  File "/home/kleisauke/emscripten/test/common.py", line 34, in <module>
    import clang_native
  File "/home/kleisauke/emscripten/test/clang_native.py", line 10, in <module>
    from tools.shared import PIPE, run_process, CLANG_CC, CLANG_CXX
ModuleNotFoundError: No module named 'tools.shared'

I'm currently working around this by removing the -E here: https://github.com/emscripten-core/emscripten/blob/060c3fc4f8a8fd1040fc25107d200df38fdb5202/test/runner#L35

And by setting the PYTHONPATH=. env variable.

[sbc100]

@kleisauke, do you think that is somehow related? Is this related to PYTHONSAFEPATH? The runner.py script contains the following lines near the top.. can you figure out why they are not working:

__rootpath__ = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))    
sys.path.append(__rootpath__)    

[sbc100]

3.12.2 seem to work find for me..

$ python3.12 -V
Python 3.12.2
$ python3.12 ./test/runner.py 
Test suites:
[]

[kleisauke]

@sbc100 Sorry for the noise. I initially thought my issue was similar to this one, but it turns out to be different.

It seems that installing Detectron2 triggers this issue:

$ pip install 'git+https://github.com/facebookresearch/detectron2.git'

After installation, it creates the following files in Python's per user site-packages directory (site-packages/tools/*).

$ ls $(python3 -m site --user-site)/tools
analyze_model.py  benchmark.py  convert-torchvision-to-d2.py  __init__.py  lazyconfig_train_net.py  lightning_train_net.py  plain_train_net.py  __pycache__  train_net.py  visualize_data.py  visualize_json_results.py

Since this path is included in Python's default search path, it conflicts with the tools directory in Emscripten.

Here's a standalone reproducer:

$ git clone https://github.com/emscripten-core/emscripten.git
$ cd emscripten
$ docker run -v $(pwd):/app -w /app -it fedora:40 sh -c 'mkdir -p $(python3 -m site --user-site)/tools && touch $(python3 -m site --user-site)/tools/__init__.py && python3 ./test/runner.py'
Traceback (most recent call last):
  File "/app/./test/runner.py", line 38, in <module>
    import jsrun
  File "/app/test/jsrun.py", line 6, in <module>
    import common
  File "/app/test/common.py", line 34, in <module>
    import clang_native
  File "/app/test/clang_native.py", line 10, in <module>
    from tools.shared import PIPE, run_process, CLANG_CC, CLANG_CXX
ModuleNotFoundError: No module named 'tools.shared'

[sbc100]

I guess that solution to that problem is to put our directory first in sys.path?

[sbc100]

See https://github.com/emscripten-core/emscripten/pull/22230