Cannot use ninja from a venv

[stephanlachnit]

I'm currently having an issue using meson-python in an environment where I cannot install ninja system-wide.

In particular, I'm in a venv

python3 -m venv venv
source venv/bin/activate
pip install -e . # has `requires = ["meson-python"]`

Which results in the following ninja:

Found ninja-1.11.1.git.kitware.jobserver-1 at /tmp/pip-build-env-3_c2bhz3/normal/bin/ninja

However, this doesn't work in the venv:

>>> import constellation
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1138, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1078, in _find_spec
  File "/home/stephan/Projects/constellation/venv/lib/python3.11/site-packages/_constellation_editable_loader.py", line 271, in find_spec
    tree = self.rebuild()
           ^^^^^^^^^^^^^^
  File "/home/stephan/Projects/constellation/venv/lib/python3.11/site-packages/_constellation_editable_loader.py", line 312, in rebuild
    subprocess.run(self._build_cmd, cwd=self._build_path, env=env, stdout=stdout, check=True)
  File "/usr/lib/python3.11/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.11/subprocess.py", line 1953, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pip-build-env-3_c2bhz3/normal/bin/ninja'

I also tried installing ninja in the venv and setting the NINJA env var, but this also doesn't work:

meson-python: error: Could not find ninja version 1.8.2 or newer.

which I think is because this is called outside of the venv - ninja --version in the venv is fine.

[eli-schwartz]

pip install creates a new venv to use instead, due to "build isolation".

meson-python will anyways install ninja as a pip-based build dependency if you don't have one on PATH, so no need to do so manually.

That's actually the problem. Since you're using an editable install, the ninja it installs inside the build isolation venv no longer exists when you try to rebuild the package.

You CANNOT use build isolation in combination with editable installs. Sadly, pip defaults to doing this so you have to magically know it doesn't work and disable it.

Editable installs are... advanced use cases, basically.

[rgommers]

You have to use:

pip install -e . --no-build-isolation

See the docs in https://mesonbuild.com/meson-python/how-to-guides/editable-installs.html for the background on why --no-build-isolation is necessary.

[stephanlachnit]

Ah ok - so even when I want to use ninja from a venv, I need to pass --no-build-isolation?

[stephanlachnit]

Ah right, and with --no-build-isolation I need to install meson-python beforehand - quite annoying, thanks for your help!

[eli-schwartz]

Ah right, and with --no-build-isolation I need to install meson-python beforehand - quite annoying, thanks for your help!

Yeah but if you don't pass no-build-isolation then pip installs meson-python plus meson, compiles the project once, and uninstalls meson-python plus meson.

You might have meson already installed. I guarantee many other users of editable installs don't. :D

[stephanlachnit]

Yeah but if you don't pass no-build-isolation then pip installs meson-python plus meson, compiles the project once, and uninstalls meson-python plus meson.

I guess this is the annoying part - it would be nice if one could tell pip to leave those package persistently

[rgommers]

it would be nice if one could tell pip to leave those package persistently

pip devs don't seem to have much interest in this, but someone should just write a small wrapper to install build deps listed in pyproject.toml into the development environment and then invoke pip install -e . --no-build-isolation. This would be the most sensible behavior for any project with compiled code, independent of which build backend is used.