Document that using meson installed in a virtual environment requires the venv to be activated

[inklesspen]

I create a venv (but do not activate it); instead I explicitly specify the path to the python binary: ../myvenv/bin/python3 -m pip --no-build-isolation -e .

The build fails because it cannot find the meson command. The meson command does exist in the venv; it's just not on the PATH. Note that the venv docs say that activating a venv is not required in order to use it.

Since meson-python expects to be able to use the meson package it includes as a dependency, it should have some way of finding it in the venv that does not depend on the path.

Reproducible example (via Docker)

FROM buildpack-deps

RUN apt update && apt install -y python3-venv python3-dev fonts-texgyre
RUN adduser --disabled-password --comment "" builder

USER builder
WORKDIR /home/builder

RUN python3 -m venv myvenv && /home/builder/myvenv/bin/python3 -m pip install meson ninja meson-python cffi
RUN git clone https://github.com/inklesspen/meson-python-cffi-example.git

WORKDIR /home/builder/meson-python-cffi-example
# specific known commit (current HEAD)
RUN git checkout b83508b99c0edd719475d4f4d5b1cb35e2ef2a50
RUN /home/builder/myvenv/bin/python3 -m pip install --no-build-isolation -e .

(Changing the final line to RUN PATH=/home/builder/myvenv/bin:$PATH /home/builder/myvenv/bin/python3 -m pip install --no-build-isolation -e . allows it to build properly.)

[dnicolodi]

The venv documentation says that activating the venv is not necessary if the only thing you want to do is to run the Python interpreter linked into the venv. If you want to use executables installed into the venv you need to either invoke them with the full path, or you need to add the bin directory inside the venv to $PATH, ie yu need to activate the venv. meson-python and Meson look for executables in the $PATH. This is true for all executables required for the build which may or may not be installed in the venv. meson-python has no way to know which executable are required, thus it cannot go check for them in paths outside $PATH.

I don't understand why you don't want to set $PATH as required, but if you really want to use a meson executable that is not on the $PATH you can use the $MESON environment variable to point meson-python to it.

[inklesspen]

At the very least it could be documented that the venv bin directory has to be on the PATH for meson-python to work.

[rgommers]

At the very least it could be documented that the venv bin directory has to be on the PATH for meson-python to work.

I agree - given how much trouble the venv docs go through to say that you can use an env without activation (which is very strange to me, because there's nothing to gain and lots of issues like this one to run into), I'd be happy to add a sentence stating that activation is needed.

The one place where we already mention a virtual environment is at https://github.com/mesonbuild/meson-python/blob/0d2df77cf1d3f0ed08ae65372175946d8e50a73f/docs/tutorials/introduction.rst#L236, which seems like an appropriate section to add a note on this.

[dnicolodi]

AFAICT the venv documentation says that you can avoid activating a venv if the only thing you want to do is to run the Python linked in the the venv by invoking it with the full path. Emphasis on the only thing you want to do is to run Python. I consider this advanced usage and I am surprised that someone that does this (or went all the way to read the documentation to learn it is a possibility) does not realize that this does not allow to run other executables installed in the venv. The documentation is pretty clear about how a venv works and what activation does. The fact that other executables installed in the venv would not be found logically follows.

I'm not opposed to add a sentence to the documentation (although anticipating all the ways in which people can shoot themself in the foot makes for very hard to read documentation). However, the title of this ticked need to be changed, there is nothing wrong with what meson-python does.

[inklesspen]

Sorry, but that page does say

You don’t specifically need to activate a virtual environment, as you can just specify the full path to that environment’s Python interpreter when invoking Python. Furthermore, all scripts installed in the environment should be runnable without activating it.

and

Since explicitly activating a virtual environment is not required to use it

and

scripts installed in environments should not expect the environment to be activated

Maybe this is a bonkers way to use the venv, but that's what the venv docs say.

[rgommers]

Yeah, it is confusing. The venv docs are really poor here imho. I've also never seen this come up before - environments should be activated, not doing so is just asking for trouble.

My random guess is that it's emphasized because the activation command is platform-dependent (at least Windows is different), which is annoying.

I was just working on a documentation update now.

[rgommers]

there is nothing wrong with what meson-python does.

I do agree with this. Any Python package may rely on an executable being on the path. E.g., an IDE like Spyder may run a CLI tool like fzf or ripgrep, and that just won't work without env activation if it's packaged as part of a wheel, or as a separate wheel. Since any random Python package is allowed to depend on site-packages/bin and a user cannot know whether this is the case, not activating an env can break any package.

[rgommers]

PR to address this: gh-634.

[eli-schwartz]

Sorry, but that page does say

You don’t specifically need to activate a virtual environment, as you can just specify the full path to that environment’s Python interpreter when invoking Python. Furthermore, all scripts installed in the environment should be runnable without activating it.

And you're trying to do something that is neither invoking a script in the venv, nor trying to directly run the venv python.

I agree with @rgommers that the documentation is just really poor here.

Fundamentally, the venv docs are attempting to make a statement about reality. They aren't saying all software is supposed to make it work - they are saying that software inherently works whether you try to make it work or not, "because shebangs". The problem is that having a shebang just means if you do run a script, the script doesn't execute using a wrong non-venv python. It doesn't mean that scripts are in PATH and usable to other software.

Part of the issue with venv's robustness of documentation is that the target userbase of the docs is not people who do complex things. It's people who don't know much about python, and are being told by the same documentation writers that the only way to safely use python is to have one venv per script and use tools like pipx to make sure they are all in PATH because it's too dangerous to try to manually get it correct. They're being told it's dangerous to use a venv for multiple purposes (like having a single copy of pytorch or SciPy/NumPy used in multiple contexts) and that they should just bite the bullet and have 16 different copies in as many venvs scattered around their home directory.

They're also told "editable installs are something you should not use. They're only for developers."