Long env path names cause weird errors after activiting virtual environment

[HaraldNordgren]

After creating a virtualenv in the path

/home/harald/git-repos/sandbox/stackoverflow/41454203-how-do-you-import-the-library-being-tested-when-running-a-nosetest/env

I could not run the env's pip,

nosetest/env$ ./bin/pip
bash: ./bin/pip: /home/harald/git-repos/sandbox/stackoverflow/41454203-how-do-you-import-the-l: bad interpreter: Permission denied

and running just pip gives me the system's pip, with all of its packages.

[yan12125]

Not virtualenv's fault. Shebangs have length limits, see http://www.in-ulm.de/~mascheck/various/shebang/.

This is basically a duplicate of #994 - virtualenv are broken with fragile kernel shebang parsing.

[HaraldNordgren]

Where is the shebang written to file?

And is there maybe some workaround we could employ to make virtual environments more sturdy?

[yan12125]

Where is the shebang written to file?

Do you mean the location of the shebang or the codes that generate a shebang? 1. In the first line of the file bin/pip 2. In distlib

And is there maybe some workaround we could employ to make virtual environments more sturdy?

I don't think there are simple workarounds. A possibility is writing a user space shebang parser just like py.exe on Windows.

[HaraldNordgren]

Using pip as an example, one thing we could do is move the current pip to .pip, remove the shebang line and create a new pip file with the following content:

#!/bin/bash
/path/to/env/bin/python /path/to/env/bin/.pip $@

Basically, wrap each program in a shell script calling the original file, passing on all of its arguments.

That will get around the problem of long shebangs.

And if we quote these paths in Bash we could probably solve the problem with space characters in paths as well.

[yan12125]

OK maybe my definition for "workarounds" is too strict. I thought it shouldn't break existing things. In this case, using /bin/bash usually works but not always. Some systems do not install bash in /bin/ (FreeBSD install it at /usr/local/bin/bash), and some systems do not provide bash at all. (Most Android uses /system/bin/sh)

A similar idea was proposed 3 years ago: https://github.com/pypa/pip/issues/923#issuecomment-19655227. Apparently no PyPA core developers are interested in this approach :(

[HaraldNordgren]

No, you're right. The workaround shouldn't break things. As long as env lives in the same place in every Unix system, this should work just as well:

#!/usr/bin/env sh
"/path/to/env/bin/python" "/path/to/env/bin/.pip" $@

It's nice to see that someone else brought up the same idea. However, @cellularmitosis phrased it in negative terms back then, so it only makes sense that no one took it seriously. It doesn't seem like any of the developers actually commented on it.

Let us give it a positive spin this time around and present it as waterproof solution to long paths and spaces. I'm not exactly sure how I would go about to putting this in the virtualenv source code though.

[yan12125]

Looks better. If it's possible to use a single file, I believe it will be ready for a pull request!

[HaraldNordgren]

@yan12125 I created a pull request.

[vsajip]

I've updated distlib to handle long paths and paths with spaces. I didn't use Harald Nordgren's patch directly (it had some problems - e.g. no tests) but the approach is the same - using '/bin/sh' as the executable. pip / virtualenv maintainers might want to test after locally vendoring the current version of the distlib repository.

[cellularmitosis]

@HaraldNordgren ah, I should have been clearer that my dismay was directed at Apple, not at the virtualenv guys! On Linux, this isn't an issue at all, and I had assumed this was an insurmountable shortcoming of OS X shebang parsing. Glad to see there's a solution!

[Imaskar]

Is it really fixed? I just run into the same issue using latest pip 20.0.2 and virtualenv 20.0.1 and python 3.7.4 . And it still generates the long path.

[gaborbernat]

It seems here we've regressed as we're not using distlib to generate the paths.

[gaborbernat]

Hello, a fix for this issue has been released via virtualenv 20.0.2; see https://pypi.org/project/virtualenv/20.0.2/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-2-2020-02-11) . Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.

[Imaskar]

Thank you for acting quick! I tried 20.0.2 and it worked. It generates full shebang on macOS which supports long paths, and short on Linux.

[MartinKosicky]

Well instead of pip... you can use python -m pip. That helped me

[gaborbernat]

@MartinKosicky you can, but should no longer be needed.

[MartinKosicky]

Ok I am a bit confused. I am using venv probabbly and that is why I can't see this fix. This is virtualenv, which is something else?

[gaborbernat]

Yeah, we're very much different from venv. One advantage of using virtualenv is that we can fix issues faster than venv. There's more detail on differences here https://github.com/pypa/virtualenv/issues/1366#issue-454109948

[MartinKosicky]

very hard to google something about virtualenv because it always gives me venv... even the standard python pages call it virtualenv, but the module is called venv :D

[gaborbernat]

IMHO it should be calling it virtual environments, not virtualenv. If you find virtualenv in venv doc then probably open an issue under http://bugs.python.org/