python pathname of the shebang in the console scripts in '/Scripts'

[return42]

• Pip version: any (tested with 9.0.1)
• Python version: any (tested with 3.6.1)
• Operating system: any (tested on MS-Win)

Question / Improvement?

When using pip under MS-Win, the shebang of the console scripts is always a absolute pathname. So far I can see, for shebang pip uses the value from:

import sys
sys.executable

Since this is always an absolute pathname, those console '.exe' scripts will not work in different (virtual) environments or in portable python setups. In POSIX like environments, it is very common to use a shebang with an env call:

#!/usr/bin/env python

Since there is no 'env' tool on windows, I expect a simple shebang wich does nearly the same, using first python interpreter from PATH:

#!python

My question is, why do we use absolute pathnames in shebang instead of using the first python from the PATH env?

What I've run (my workaround):

I wrote a small hackish pip.bat wrapper around my pip

python -c "import pip, sys;sys.executable='python.exe';pip.main()" %*

which replaces the absolute pathname with sys.executable='python.exe'. So I get portable console scripts with the #!python shebang.

But the question remains, why is it better to use absolute pathnames? I guess this is the same on POSIX systems, where pip also uses absolute pathnames instead of using /usr/bin/env.

Thanks!

Update 10/2018

See https://github.com/pypa/pip/issues/4616#issuecomment-432279328 from @amr66, newer versions (18.1) of pip have moved the main function to pip._internal and you need:

 python -c "from pip._internal import main, sys;sys.executable='python.exe';main()" %*

[pfmoore]

Simple answer, because the scripts are only intended for use in the specific environment the package was installed in. If you install the package in another environment, a separate script for that environment is created. Making the scripts pick the currently active Python means that they will fail if that environment doesn't have the right dependencies installed.

Does that answer your question? If not, can you explain precisely what you are trying to do that is not possible because of the current script behaviour?

[pfmoore]

Simple answer, because the scripts are only intended for use in the specific environment the package was installed in

More precisely, you can use the scripts from anywhere, but they will always use the environment they were created in for their Python interpreter (because that's the environment known to have the correct packages installed).

[return42]

Thanks for your quick reply ..

Does that answer your question?

Yes, I see there are some drawbacks with python taken from PATH.

More precisely, you can use the scripts from anywhere, but they will always use the environment ..

I guess you know more use-cases than I. I thought only about virtualenv, where you have a 'activate/deactivate' script which selects the right environment.

... can you explain precisely what you are trying to do that is not possible because of the current script behaviour?

I setup a portable python with I can go to my customers and run some of my self-written maintenance and analyses tools. On most linux system I do not have problems running my py-scripts (py2/py3 compatible) .. but often on MS-Win a python is not installed and the server is placed in a very restricted array with no access to the internet. In short: no chance to install python. These are my use-case, where I need a portable python (in a ZIP) which I can copy to and run from in my users HOME folder.

To be more general; does it make sense to add an option to pip where the caller can chose if he wants absolute pathname in the shebang (default) or a shebang with a python from the env (PATH)?

[pfmoore]

OK, I see what you're trying to do. Portable python installations on Windows are tricky to get right (I don't think any version of Python before 3.6 gets things completely portable - although the corner cases where there are issues are pretty rare in practice, and likely not things you need to worry about). There are going to be some issues with wrapper scripts having "find Python on PATH" shebang lines, and I doubt it would be workable in general.

For pip, and many other packages that provide a CLI wrapper, you can simply use python -m pip - that's exactly equivalent to using the pip.exe wrapper, and avoids the need for any hacking of the wrapper. For a more general solution, your hack is probably OK. It's not supported, obviously, but I did a quick check of the pip sources and I can't see anything obvious that would cause it to fail.

[return42]

Portable python installations on Windows are tricky to get right

Yes, today it is tricky but for every task there is a starting point ;) For pure python without external C libraries it seems to work fine (I need much more practical experience).

For pip, and many other packages that provide a CLI wrapper ..

Right, that was my first attempt .. but after while more and more packages came into my portable python and I was looking for a more general solution.

For a more general solution, your hack is probably OK. It's not supported, obviously, but I did a quick check of the pip sources and I can't see anything obvious that would cause it to fail.

Thanks for your effort. I also looked at _vendor.distlib.utils.get_executable() and its history gives my the hope that this will not change in the near future ;)

@pfmoore thanks for clarifying .. if you know about other portable pure CPython efforts, please let me know .. I have seen e.g. WinPython, but this is only MS-Win and not pure CPython / not what I was looking for.

[pradyunsg]

@return42 - nice name - is your question resolved?

[return42]

Yes, my question is answered .. but resolved?

I understand why pip use absolute path names and I have a workaround. The cons is: my workaround is hakish and might break in the (near) future ..

The absolute pathname decision of pip does not fit for all scenarios. It would be nice to have a option to set the shebang line. E.g. --shebang=python which inserts the simple shebang #!python.

[DancingQuanta]

I have an issue with this when using Cygwin.

My current setup with Cygwin and Windows python relies on a bash script ~/bin/python, which is included in PATH, that converts the POSIX paths of python scripts to Windows paths and passes that to Windows python to execute the script.

if [[ -z $PYTHON ]]; then
  echo "Cygwin"
  /usr/bin/python $@
else
  ARGS=()
  for var in "$@"; do
    if [[ $var == -* ]]; then
      ARGS+=("$var")
    elif [[ -f $var ]]; then
      file="$(cygpath -w -a $var)"
      echo $file
      ARGS+=("$file")
    else
      ARGS+=("$var")
    fi
    echo "Var $var, Args${ARGS[@]}"
  done
  $PYTHON "${ARGS[@]}"
  echo "Done!"
fi

where $PYTHON is set as path to my Windows python interpreter.

When I install a python script via pip, the script might have a custom shebang that points to the Windows python interpreter. This bypasses my ~/bin/python and so I am unable to execute this script through Cygwin.

Oh and I am using Anaconda which itself is installed in Windows and have no Cygwin equivalence.

So I like the idea of controlling the shebang during installations via pip.

[return42]

@DancingQuanta did you test my hackish pip install replacement:

python -c "import pip, sys;sys.executable='~/bin/python';pip.main()" install ...

does this workaround work for you?

[DancingQuanta]

No dice, the hack does not work for me

13:23:11 andre@LAPTOP-V9S562AR:~
$ ./Anaconda3/python.exe -c "import pip, sys, os; sys.executable='~/bin/python'; pip.main()" install yaml2json            Collecting yaml2json
  Using cached yaml2json-1.0.0.tar.gz
    Error [WinError 2] The system cannot find the file specified while executing command python setup.py egg_info
Exception:
Traceback (most recent call last):
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\commands\install.py", line 335, in run
    wb.build(autobuilding=True)
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_set.py", line 634, in _prepare_file
    abstract_dist.prep_for_dist()
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_set.py", line 129, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_install.py", line 439, in run_egg_info
    command_desc='python setup.py egg_info')
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\utils\__init__.py", line 667, in call_subprocess
    cwd=cwd, env=env)
  File "C:\Users\andre\Anaconda3\lib\subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "C:\Users\andre\Anaconda3\lib\subprocess.py", line 955, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

What file it is trying to find? ~/bin/python? I tried expanding out '~' but this yields different errors

13:24:52 andre@LAPTOP-V9S562AR:~
$ ./Anaconda3/python.exe -c "import pip, sys, os; path = os.path.join(os.path.expanduser('~'), 'bin', 'python'); sys.executable=path; pip.main()" install yaml2json
Collecting yaml2json
  Using cached yaml2json-1.0.0.tar.gz
    Error [WinError 193] %1 is not a valid Win32 application while executing command python setup.py egg_info
Exception:
Traceback (most recent call last):
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\basecommand.py", line 215, in main
    status = self.run(options, args)
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\commands\install.py", line 335, in run
    wb.build(autobuilding=True)
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_set.py", line 634, in _prepare_file
    abstract_dist.prep_for_dist()
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_set.py", line 129, in prep_for_dist
    self.req_to_install.run_egg_info()
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\req\req_install.py", line 439, in run_egg_info
    command_desc='python setup.py egg_info')
  File "C:\Users\andre\Anaconda3\lib\site-packages\pip\utils\__init__.py", line 667, in call_subprocess
    cwd=cwd, env=env)
  File "C:\Users\andre\Anaconda3\lib\subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "C:\Users\andre\Anaconda3\lib\subprocess.py", line 955, in _execute_child
    startupinfo)
OSError: [WinError 193] %1 is not a valid Win32 application

[return42]

@DancingQuanta :

OSError: [WinError 193] %1 is not a valid Win32 application

Your ~/bin/python is a shell script. I do not know if it will work at all.

Ok, now I understand what you try to do: You are on Win and you start a shell script, which starts a suitable Python interpreter as a child process of the shell script. On Win shell scripts are not executable, you need to call the bash .. puh ..

What do you think is the perfect shebang for this (I have no idea)?

[DancingQuanta]

If I execute a python script from a bash environment, I expect the shebang to be usr/bin/env python as that finds my ~/bin/python. This shebang is common in linux. However I do not plan to use python scripts from cmd.

I still get

OSError: [WinError 193] %1 is not a valid Win32 application

if I substitute os.path.join(os.path.expanduser('~'), 'bin', 'python') for /usr/bin/env python.

What processes does pip run that modifies the shebang? If we could somehow substitute the python interpreter's path before it could write that to the shebang.

[pradyunsg]

@vsajip Would the shebang fix in distlib master also resolve this situation?

[vsajip]

No - that fix is for POSIX only, and relates to Linux limitations on shebang parsing. On Windows, the shebang parsing is done by the Python launcher for Windows (if unavailable, shebangs won't work on Windows). IIRC for a shebang using /usr/bin/env python the launcher searches the Windows PATH for a Python executable.

Scripts installed into venvs always use absolute paths to interpreters, allowing them to be used even when a venv isn't activated. This is by design. Users should regard venvs as throwaways - instead of expecting a script installed in one venv to work in another implicitly, users should just install the software into multiple venvs. I realise this doesn't solve OP's problem, which is a niche case - for this, they could (for example) write a small tool which fixed up the shebangs in all the affected scripts to whatever the user's local Python directory is.

[return42]

@vsajip thanks for clarifying.

I realise this doesn't solve OP's problem, which is a niche case - for this, they could (for example) write a small tool which fixed up the shebangs in all the affected scripts to whatever the user's local Python directory is.

Yes, I can write such a script for *nix, but there is no way (except my workaround) for the Win .exe launchers.

So I am really looking forward to set individually shebangs when installing with pip. I mean, I just want a shebang which calls python from the environment .. is this really such a niche case?

[DancingQuanta]

This case is important if you are using Cygwin with Windows python (especially theone that bundles with Anaconda). I like a hack script that modifies the shebang before it get written to the python script during installation. I like to see where in pip it writes the shebang and the hack script will modify that shebang before write.

[pradyunsg]

I like to see where in pip it writes the shebang and the hack script will modify that shebang before write.

If my understanding is correct, that'll be pip.wheel:fix_script. :)

[DancingQuanta]

Thank you, however I am not really good at modifying behaviours of existing python packages.

My fix_script() would be

def fix_script(path):
    """Replace #!python with #!/path/to/python
    Return True if file was changed."""
    # XXX RECORD hashes will need to be updated
    if os.path.isfile(path):
        with open(path, 'rb') as script:
            firstline = script.readline()
            if not firstline.startswith(b'#!python'):
                return False
            firstline = b'#! /usr/bin/env python'
            rest = script.read()
        with open(path, 'wb') as script:
            script.write(firstline)
            script.write(rest)
        return True

Though this hack function would be too long for

python -c "import pip, sys;sys.executable='python.exe';pip.main()" install

I think the above hack function could be in a python script something like

import pip

# modify pip with fix_script() to give hacked_pip

hacked_pip.main()

I need advice on how to modify pip through inheritance, something like

class hacked_pip(pip)

    def fix_script()
        ...

[vsajip]

@return42 I see.

Yes, I can write such a script for *nix, but there is no way (except my workaround) for the Win .exe launchers.

It's possible, but might be only a little bit more work. I'll think some more about this.

So I am really looking forward to set individually shebangs when installing with pip. I mean, I just want a shebang which calls python from the environment .. is this really such a niche case?

Not for scripts you write - but for scripts which are installed into a venv, it is something of a niche case because in general, things could fail if an interpreter other than the venv's interpreter (which uses the venv environment) is used to run a script which is installed in a venv. Just to be clear what I mean, here's a thought experiment:

1. Install foolib into venv A which includes a script fooscript using foolib functionality. Currently this will have a shebang referencing e.g. ~/.virtualenvs/A/bin/python (or the Windows equivalent using Scripts instead of bin).
2. Activate venv B which doesn't have foolib installed, so that its interpreter is on the PATH.
3. Run ~/.virtualenvs/A/bin/fooscript. It will use the venv A interpreter where foolib is available, even though you have venv B activated, because of the absolute path in the shebang.
4. Now imagine if fooscript had a shebang referencing e.g. /usr/bin/env python. If you run ~/.virtualenvs/A/bin/fooscript in this case, with venv B activated, it will fail because the venv B interpreter will be used, and it won't find foolib in its environment.

This is the reason why pip doesn't use e.g. /usr/bin/env python by default.

[return42]

@vsajip: Thanks .. yes, I understand and I know the conflict of different venvs.

This is the reason why pip doesn't use e.g. /usr/bin/env python by default.

May I was unclear. I don't want to see the default changed. I just like to have an option to change that default behavior. IMO its very common to setup (servers) processes by their environment. This is what I found in many enterprise scenarios.

In such scenarios python is mostly a third party tool like other third party tools which are saddled up by the environment. ATM I have to hack around my tool-chain (pip) so that the installation build with, will use the tools (python interpreter) from the active environment. Thats why I vote for an option which lets my decide which shebang is inserted.

[pradyunsg]

@return42: My question is, why do we use absolute pathnames in shebang instead of using the first python from the PATH env?

@vsajip This is the reason why pip doesn't use e.g. /usr/bin/env python by default.

I think the question got answered. :)

I'm tempted to close this issue; I'll resist the urge tho.

---

FWIW, I'm thinking that this is an extremely niche case. It's probably better that instead of changing pip, you change how you'd use tools installed with pip. That said, I'm not gonna be the one taking that call on either parties so I'll bow out of this discussion. :)

[pradyunsg]

As an aside, this is something that can easily be achieved by manipulating pip in unsupported ways to do what you want. I was bored so here's a bodge for that, which might work.

import pip
import pip.wheel

# The following patches pip to use "/usr/bin/env python" in shebangs
def fix_script(path):
    if os.path.isfile(path):
        with open(path, 'rb') as script:
            firstline = script.readline()
            if not firstline.startswith(b'#!python'):
                return False
            # Only the next line is changed
            firstline = b'#!/usr/bin/env python'
            rest = script.read()
        with open(path, 'wb') as script:
            script.write(firstline)
            script.write(rest)
        return True

pip.wheel.fix_script = fix_script

if __name__ == '__main__':
    pip.main()

There's monkey patching, it's using pip in an unsupported manner, it's not well commented and all that.

Don't use it. I'm not even gonna tell you how to use it. So, don't ask/blame me if you use this and something breaks, your computer crashes or your house catches fire because of this. You'll be responsible for that.

It probably does what you want tho.

[return42]

@pradyunsg about https://github.com/pypa/pip/issues/4616#issuecomment-323696941

I think the question got answered. :)

Yes, and it was already answered by @pfmoore in https://github.com/pypa/pip/issues/4616#issuecomment-314378220 Thats, why I asked for an option to pip in https://github.com/pypa/pip/issues/4616#issuecomment-314386777

FWIW, I'm thinking that this is an extremely niche case.

May be from from a narrowed Python & pip POV. But there are guys outside trying to manage real scenarios.

@pradyunsg about https://github.com/pypa/pip/issues/4616#issuecomment-323697538 ...

There's monkey patching, it's using pip in an unsupported manner, it's not well commented and all that.

Thanks for your work on this, but I cant see a big benefit over my

python -c "import pip, sys;sys.executable='#!python.exe';pip.main()" %*

I'm tempted to close this issue; I'll resist the urge tho.

If you think this discussion is nitpicking (since there are now two hackish workarounds), feel free to close the issue. But why not waiting and see what other coming around to this issue will say?

[pradyunsg]

But why not waiting and see what other coming around to this issue will say?

Which is exactly why I didn't close it. :)

[jlapolla-cray]

May I was unclear. I don't want to see the default changed. I just like to have an option to change that default behavior. IMO its very common to setup (servers) processes by their environment. This is what I found in many enterprise scenarios.

In such scenarios python is mostly a third party tool like other third party tools which are saddled up by the environment. ATM I have to hack around my tool-chain (pip) so that the installation build with, will use the tools (python interpreter) from the active environment. Thats why I vote for an option which lets my decide which shebang is inserted.

@return42, I 100% agree with this. I'm presently working around this issue at my workplace. Ideally, I'd like to see a --shebang option available in all of the following places:

• pip install --shebang=/my/shebang ...
• easy_install --shebang=/my/shebang ...
• python setup.py install --shebang=/my/shebang ...

Some of these are outside the scope of the pip Git repo, but I thought I'd throw them out there.

If you are reading this, please comment on this issue. If we get enough comments, then perhaps this is not a niche use case.

[amr66]

I make a windows portable install for a big open-source gis package, called qgis. Portable here means, i can start the software from a usb stick or a network drive. QGIS comes with its own python installation, that is not registered under the OS. The software uses it's own environment under windows and comes with an installer that downloads many packages, several of them are python packages. One baulk of it is, that selfexecutable python scripts have a shebang with an absolute path pointing to a directory usually chosen by the installer. So the admin/user can't install to another directory and my portable install also does not work with these scripts. Inside QGIS the correct python.exe is always in the path, so an option could be using #!python.exe. I'm saying this not beeing a qgis developer.

[return42]

@amr66 instead of using pip, have you tested my workaround calling pip this way:

python -c "import pip, sys;sys.executable='python.exe';pip.main()" ...

see "What I've run" section in the top post.

[amr66]

@return42 I read the whole thread, but didn't try your workaround. Instead i'm using just

python -m pip

It would be nice to provide a general solution with pip to the osgeo4w/gis developer, anyway. Can't think they will accept a workaround at all.

[return42]

@amr66 sorry if I'am slow-witted, I have to ask: does

 python -m pip

fix your shebang?

[amr66]

Oh, no, of course not... after reading here, i thought your command would run pip, not changing the shebang. So i really have to try.

[amr66]

Ah, now i understand, ok, it works fine, thank you.

[amr66]

I used that under python3/pip 18.1, as your command throws an AttributeError: module 'pip' has no attribute 'main'

python -c "from pip._internal import main, sys;sys.executable='python.exe';main()"

[return42]

@amr66 thanks for pointing out. I tested with 18.1 and updated my top post.

[lordyavin]

I recently setup a "portable" Python 2.7 environment that may be used from different installation locations and stumbled across this absolute path issue. I didn't install the scripts into a venv so (as a Windows user and developer) I expected that the python.exe is either looked up in the current working directory or the folders listed in the search path (%PATH%).

If pip has the ability to determine if the installation is going on in an active venv then the abspath should be used. Otherwise use /usr/bin/env python and leave it to the launcher to search the Windows environment to find the executable.

@vsajip wrote:

Not for scripts you write - but for scripts which are installed into a venv, it is something of a niche case because in general, things could fail if an interpreter other than the venv's interpreter (which uses the venv environment) is used to run a script which is installed in a venv. Just to be clear what I mean, here's a thought experiment: ... This is the reason why pip doesn't use e.g. /usr/bin/env python by default.

I agree that a script installed in a venv A depends on it, but if I work in another venv B this script from A shouldn't be available/work. I would be irritated that the script from venv A works. That's why your scenario is inconclusive to me.

[pfmoore]

That's why your scenario is inconclusive to me.

As a counterpoint to your argument, I routinely have scripts from multiple venvs runnable at the same time. Sometimes simply by having more than one Scripts directory on PATH, and often by copying the script wrapper exe into a central location (pipx does this, and hence relies heavily on absolute paths in the shebang).

So changing pip to not generate absolute paths in virtual environments would cause significant breakage.

I've less in the way of motivating use cases for the system Python, but given the existence of things like the embeddable and nuget distributions of Python, which can be used as effectively self-contained environments, it's unclear to me where the line should be drawn - so I'd argue for consistency (absolute paths everywhere) by default.

Note: It wouldn't be hard to write a standalone utility that created a new wrapper with a relative path in the shebang. So even if pip sticks with absolute paths, people who need relative ones can create their own wrappers.

[lordyavin]

So changing pip to not generate absolute paths in virtual environments would cause significant breakage.

I'm sorry you got me wrong. That is not what I proposed. I propose to use absolute paths in virtual environments only. But looks like that won't fit every scenario too.

I agree with you about consistency but IMHO using hard coded absolute paths in applications was and will be bad practice. Maybe I have the wrong point of view here, but nobody is perfect :wink:

Thanks for your reply I will think about using one of the proposed workarounds.

[pfmoore]

I'm sorry you got me wrong.

No, I wasn't clear enough. I knew you were saying that absolute paths for venvs was good, and was intending to agree with you. What I wanted to contrast with was your point that when env B is active, you wouldn't expect to use scripts from env A, whereas I do things like that all the time.

Mostly though, we're in agreement here (except maybe there's areas outside the venv case where you feel that pip's behaviour could be improved, whereas I feel it's probably "good enough" as it is).

[return42]

so I'd argue for consistency (absolute paths everywhere) by default.

That's fine for default, I asked for adding an option to pip command. I can only repeat what I said in 2017: https://github.com/pypa/pip/issues/4616#issuecomment-323689740 and point out this comment from 2018 https://github.com/pypa/pip/issues/4616#issuecomment-408947813

To summarize: As long as we do not have such an pip install --shebang=<my/shebang> option, we can use the workaround:

python -c "from pip._internal import main, sys;sys.executable='<my/shebang>';main()"

By example, always use python from PATH:

  python -c "from pip._internal import main, sys;sys.executable='python';main()"

[jlapolla-cray]

I'm not sure why there's so much resistance to this feature request. It seems like a straightforward feature that solves real use cases without disturbing any existing use cases.

Let's pretend I ask pip to install scipy in the prefix /usr/local (not in a virtual environment). If I later adjust my PYTHONPATH so that another numpy is found first (numpy is a dependency of scipy) instead of the one installed in /usr/local, then scipy and all the scripts installed in /usr/local will automatically use the new numpy that I put on my PYTHONPATH. However, if I adjust my PATH so that another Python interpreter is found first, the scripts installed in /usr/local don't use the new Python that I purposely put on my PATH.

In other words: I can freely adjust which Python packages are found by the scripts in /usr/local, but I cannot freely adjust which Python interpreter is found by the scripts in /usr/local. That seems inconsistent to me.

These kinds of search path adjustments are often needed to work around problems, especially when you are not root on the system you are doing your work on (which is common in enterprise environments). The ability to modify search paths and thus modify the behavior of an existing program is something that every nix user is accustomed to. I'd even say it's a fundamental expectation of nix users. Even binary executables allow changing the dynamic library search path with LD_LIBRARY_PATH, and you can always use a different loader to launch the binary executable, thus changing its behavior in subtle ways.

I understand that there are use cases where you want to completely lock down a Python script and force it to use a fixed Python interpreter and a fixed Python package search path. But there are also use cases where you want a Python script to adapt to its environment. Can we have a --shebang=... option that enables users to more conveniently handle these kinds of adaptive use cases? I don't see why the adaptive use cases should be disallowed or discouraged.

[chrahunt]

@jlapolla-cray, the cost of adding configuration needs to take into account more than just disturbing existing use cases.

Consider user support. This change specifically would make it harder to support users because now we need to always keep its existence in mind. Was this option used, could any config file have set this option, could the person that set up the environment have used this option? Then the question becomes whether the script is set that way because the option was in effect (and maybe the user does not remember) or because there is a bug in pip or an interaction with the system or the result of a platform maintainer making changes to their Python, pip, or any other tool. This relates to and combines with other aspects of the system like whether their Python is system-managed or manually installed, whether the package they're having a problem with was installed globally, as a user, using a target directory, or in a virtual environment, and so on.

[chrahunt]

on MS-Win a python is not installed and the server is placed in a very restricted array with no access to the internet. In short: no chance to install python. These are my use-case, where I need a portable python (in a ZIP) which I can copy to and run from in my users HOME folder.

@return42, is this still your use case? If so, have you considered having your Python zip consist of:

1. "portable" Python itself
2. pip, virtualenv (or use built-in venv for Python 3, you will also not need a separate pip in that case), and all scripts packaged up as source distributions/wheels with console_script entrypoints defined? You can package your own projects up the usual way and for dependencies you can run pip download . and/or pip wheel . for each.
3. a (Powershell/batch) script that extracts the Python, installs pip, then uses pip to install everything else into a virtual environment?

Then working on a customer server would look like:

1. Get Python payload to server.
2. Invoke script
3. Make sure virtual environment Scripts dir is on PATH - then they will resolve to the correct Python by default no matter where you call them from. You could also augment PATH for the current session in the script in 2.

This allows you to maintain all of your scripts and helpers as plain Python packages that can managed, tested, distributed, and re-used in the usual way.

[amr66]

I use a "big" software, called QGIS (a geographic information system) and install that unser Windows by using an installer managed by a group named osgeo4w. The osgeo4w/qgis distribution contains python3, which is not registered in Windows, but exists under a subdirectory of the software. Mainly an environment variable, a custom path and a lot of batch files call all parts of the Software, including python, so that the correct python is always in the path. Osgeo4w also maintains Python packages, which have dependencies with their DLL's, mainly geospatial stuff. Some of the packages have .exe files under pythons "Scripts" directory. All these exe, including pip, contain a shebang pointing to a python executable, with an absolute path. Accessing the whole software from a server (my universities case) or move it on a usb-drive (my case) can't use those "scripts".

If pip could create shebangs without absolute pathes (#!python.exe), this issue would no longer exist. Now, i use a text editor, search for the shebang and change it manually... By the way: same with virtualenv, but this may be another story. I am not sure about all the consequences and side effects of a pip command line switch, but i hope, for my use case and maybe the one of osgeo4w it will be an improvement.

[return42]

@chrahunt: is this still your use case?

This and other similar to comment.

1. "portable" Python itself

Its dead. BTW I like to prefer CPython which by itself is portable . There is only one tool in the python ecosystem that prevents us from using CPython in a portable manner: and this tool is called pip ;)

Then working on a customer server would look like:

• Get Python payload to server.
• Invoke script
• Make sure virtual environment Scripts dir is on PATH

To many steps to get working. Build up my python-ZIP with my workaround enables me to unzip and just start working.

@amr66: I use a "big" software, called QGIS

Same here, I also struggle with "enterprise" software and their f** proprietary solutions.

Now, i use a text editor, search for the shebang and change it manually... By the way: same with virtualenv, but this may be another story.

I wrote a script for my scenarios fix_launcher.py. This yields over entry_points and build launchers from templates. E.g. have a look at the powerscript_ep_template where I have to build up a very special RTE in the launcher of such a "enterprise" solution.

With some changes this script should be work for your scenarios.

[amr66]

@return42: thank you for sharing. QGIS is big but not proprietare, it is even an open source project. So there is a chance that proposed changes will be incorporated. I will point them to you efforts.

[return42]

@chrahunt: Consider user support. This change specifically would make it harder to support users because now we need to always keep its existence in mind.

This reasoning is understandable! .. I know such situations and I always try to get around. You have my fullest sympathy .. At the end, it is a tradeoff. On the one hand to prevent the portable scenarios or on the other hand, to accept further faulty operations.

[jlapolla-cray]

@chrahunt, you have a good point. The more use cases your software serves, the more use cases you have to support. You are wise to scope your software to keep your support costs manageable. If you're already at your limit, then you absolutely should not add more to your workload.

Adding a --shebang=... option adds one more variable to the equation. Why should you increase your support burden for a niche use case like this?

The sparsity of comments on this feature request may indicate that there's not much interest in the --shebang=... option. How many comments do you usually get on a feature request for a popular feature? I would be interested to know.

[pfmoore]

It's not something we particularly track, to be honest. It might be of interest to review previous feature requests (although the pip devs aren't likely to do this, same point about limited resource). I suspect the results would be inconclusive, though - things like how broadly useful a feature seems, how naturally it seems to fit with other features, whether there is an easy workaround, and simply if a core dev is interested, will likely play a big part in any decision.

Personally, I tend to feel that if the use case is relatively niche, and there is a workaround, we should be reluctant to add a feature - but users seem to on average much prefer integrated, "pip does everything" solutions over using multiple tools together like that (and I include myself, when I'm using pip, in the "users" category 🙂) So there's always a bit of tension here.

[chbrown]

I spent some time on this thread trying to figure out how to customize the shebang used in console_scripts installed by pip on macOS, using Python 3.{7,8} from Homebrew. (I realize OP asked about Windows; apologies in advance if this is too far off-topic 😂 — I thought it might be helpful to leave this here for a future wanderer in the same position I was in.)

The sys.executable=... solution worked for me (thanks!) but then I found that Homebrew was doing the same thing; rather than overriding the override, I disarmed it:

export PYTHONEXECUTABLE=/usr/local/bin/python3  # or whatever

(See /usr/local/lib/python3.?/site-packages/sitecustomize.py for the Homebrew-generated file, particularly the assignment to sys.executable at the end.)

[rgoldberg]

Is there currently any way to have a #!/usr/bin/env style shebang used by the various pip* executable scripts?

Would this be possible when installing pip, via some pip script update command, or just by manually editing the pip* scripts?

Would anything bad occur if I manually edit the pip* scripts?

Are there other scripts that I must also modify?

Sorry if this is the wrong place to ask this, or if it's been answered earlier in this issue; I just don't have the time right now to read through the whole thread.

[chrahunt]

@rgoldberg, if you make a separate issue you will get more focused attention. That keeps things neat even if the question is already mentioned somewhere else, and you can edit to improve it as you get more time. :+1: