Missing `py` binary in the windows installer

asmeurer commented 10 years ago

Issue by ionelmc from Monday Jul 21, 2014 at 23:40 GMT Originally opened as https://github.com/conda/conda/issues/819

See http://legacy.python.org/dev/peps/pep-0397/

johnmellor commented 9 years ago

The py binary is the standard way of making Python file associations work on Windows. Specifically, you associate *.py with py.exe, and it parses the shebang line to work out whether it should use python2 or python3. Please fix this!

Documentation:

kitsu commented 9 years ago

It is a little old-fashioned, but ye old script from Ned Batchelder still works for Python versions < 3. I just confirmed this works on Win8 (after making the modification mentioned in the last comment).

""" Change the .py file extension to point to a Python installation."""
import _winreg as reg
import sys

pydir = sys.argv[1]

todo = [
    ('Applications\python.exe\shell\open\command',
                '"PYDIR\\python.exe" "%1" %*'),
    ('Applications\pythonw.exe\shell\open\command',
                '"PYDIR\\pythonw.exe" "%1" %*'),
    ('Python.CompiledFile\DefaultIcon',
                'PYDIR\\pyc.ico'),
    ('Python.CompiledFile\shell\open\command',
                '"PYDIR\\python.exe" "%1" %*'),
    ('Python.File\DefaultIcon',
                'PYDIR\\py.ico'),
    ('Python.File\shell\open\command',
                '"PYDIR\\python.exe" "%1" %*'),
    ('Python.NoConFile\DefaultIcon',
                'PYDIR\\py.ico'),
    ('Python.NoConFile\shell\open\command',
                '"PYDIR\\pythonw.exe" "%1" %*'),
    ]

classes_root = reg.OpenKey(reg.HKEY_CLASSES_ROOT, "")
for path, value in todo:
    key = reg.CreateKeyEx(classes_root, path, 0, reg.KEY_SET_VALUE)
    reg.SetValue(key, '', reg.REG_SZ, value.replace('PYDIR', pydir))

Maybe you guys can include something similar in Windows installers for e.g. Python2.7...

johnmellor commented 7 years ago

I've tested conda with the official Python Launcher for Windows (py.exe and pyw.exe), and it works perfectly, though there are a few things conda could do to make things easier to set up.

I'll first describe the two configurations I tested successfully, then conclude with the suggestions for making things easier.

A) Easy steps to use Python Launcher for Windows with two root conda envs

Install the Python Launcher for Windows (I used launchwin-1.0.1.6.amd64.msi which puts 64-bit py.exe and pyw.exe in C:\Windows, but anywhere on your PATH will do), and let it register itself as the default file association for *.py etc files (security note: don't double click untrusted python scripts after doing this...).
Install both the Python 2 and Python 3 versions of Anaconda or Miniconda side by side, as two independent root environments. In both cases, say yes to "Register Anaconda as the system Python x.x", but say no to "Add Anaconda to the system PATH environment variable" (I went for 64-bit All Users installs in both cases).
At this point none of python/conda/pip should be on your PATH. In order to more easily access them, create two batch files in a folder on your PATH called activate2.bat and activate3.bat with contents like

:: Adjust path to activate.bat as appropriate.
@CALL "C:\ProgramData\Miniconda2\Scripts\activate.bat" %*

:: Override PROMPT for root environment to distinguish Python 2 from Python 3.
@IF NOT "%~1" == "" @IF NOT "%~1" == "root" @GOTO notroot
:: conda's deactivate.bat will reset this when cleaning up the PROMPT.
@SET "PROMPT=(py2) %CONDA_PS1_BACKUP%"
:notroot

Now you can run activate2 or activate3 to activate the corresponding root environments (or e.g. activate2 myenv to directly activate one of their sub-environments), and you'll be able to call python/conda/pip within that environment. Run deactivate as usual to clean up (removes python/conda/pip from the PATH).

Whether or not you currently have an environment activated, you'll also always be able to run both Python 2 and 3 scripts using their respective root environments via the Python Launcher for Windows:

C:\Code>echo import sys; print(sys.version)> no_shebang.py
C:\Code>no_shebang.py
2.7.13 |Continuum Analytics, Inc.| (default, Dec 19 2016, 13:29:36) [MSC v.1500 64 bit (AMD64)]
C:\Code>py no_shebang.py
2.7.13 |Continuum Analytics, Inc.| (default, Dec 19 2016, 13:29:36) [MSC v.1500 64 bit (AMD64)]
C:\Code>py -3 no_shebang.py
3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)]
C:\Code>echo #!/usr/bin/env python3> py3_shebang.py
C:\Code>echo import sys; print(sys.version)>> py3_shebang.py
C:\Code>py3_shebang.py
3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)]
C:\Code>py py3_shebang.py
3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)]
C:\Code>py -3 -m pip --version
pip 9.0.1 from C:\ProgramData\Miniconda3\lib\site-packages (python 3.6)
C:\Code>py -3
Python 3.6.0 |Continuum Analytics, Inc.| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

B) Harder steps to use Python Launcher for Windows with one root conda env

Install the Python Launcher for Windows (I used launchwin-1.0.1.6.amd64.msi which puts 64-bit py.exe and pyw.exe in C:\Windows, but anywhere on your PATH will do), and let it register itself as the default file association for *.py etc files (security note: don't double click untrusted python scripts after doing this...).
Install only the Python 2 version of Anaconda or Miniconda. This will become the root environment. Say yes to both "Register Anaconda as the system Python 2.7" and "Add Anaconda to the system PATH environment variable" (I went for a 64-bit All Users install).
Create a Python 3 environment within this conda install (may require admin rights if you chose All Users earlier): conda create -n py3 python=3 anaconda or if you only want Miniconda: conda create -n py3 python=3
Run regedit and create a subtree under HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\3.6 that matches the one for Python 2.7 that was created by the Anaconda/Miniconda installer, except that it uses the path to your Python 3.6 environment shown by conda info --envs. For example, 3.6\InstallPath might be C:\ProgramData\Anaconda2\envs\py3 and 3.6\PythonPath might be C:\ProgramData\Anaconda2\envs\py3\Lib;C:\ProgramData\Anaconda2\envs\py3\DLLs depending on where you installed Anaconda/Miniconda and how you named your Python 3 environment.

In this configuration, the default python/conda/pip on your PATH will be from your root environment (Python 2.7 in these steps), but you can switch to the Python 3 environment with activate py3, and back with deactivate.

As above, you'll be able to use the Python Launcher for Windows to run both Python 2 and 3 scripts irrespective of your current environment.

(Here I used a Python 2 root environment containing a Python 3 sub environment, but the other way around also works if you prefer Python 2 to be your default python - though a subtle caveat of that is that Python Launcher for Windows would then use Python 3 when it encounters a #!/usr/bin/env python shebang, as it searches the PATH first for /usr/bin/env shebangs, which goes against PEP 394's recommendation that "for the time being, all distributions should ensure that python refers to the same target as python2").

Action items for conda/Anaconda

There are two things that would make this easier:

Since https://conda.io/docs/using/envs.html seems to prefer using sub-environments rather than side-by-side root environments, it'd be good to move forward on https://github.com/conda/conda/issues/107 so users can use non-root environments with Python Launcher for Windows without having to manually edit their registry. This might also help IDEs to detect primary conda environments? Alternatively, add support for side-by-side root environments (using a mechanism like the activate2.bat/activate3.bat above)
Bundle py.exe and pyw.exe from Python Launcher for Windows with all Anaconda and Miniconda installers (it's BSD licensed, and seems intended to be distributed like this), and install them into e.g. C:\Windows if the user chooses "Register Anaconda as the system Python x.x" (whether or not they choose "Add Anaconda to the system PATH environment variable"). As for registering it as the default file association for .py etc files, that probably deserves its own checkbox, since some users will prefer to keep an editor associated with .py files (in which case they'll use py foo.py instead of just foo.py to run a script).

Additional notes

On Windows 10, the first time you double click a *.py file after installing Python Launcher for Windows, it may ask what application you want to use to open it with despite having registered the default file association. Choose "Python Launcher for Windows (Console)". Similarly for *.pyw, but choose "Python Launcher for Windows (GUI)" (pyw.exe).

mingwandroid commented 7 years ago

Thanks for taking the time to work through this. My opinion on the launchers is that we shouldn't add them because:

Any solution that involves adding things to PATH is not something I'd be comfortable with at all for reasons of DLL hell (and your other non-PATH adding solution avoids using conda envs and therefore wastes a lot of space).
conda takes a different approach to 'normal' Windows CPython, emphasising cross-platform consistency which py.exe and pyw.exe work against.
Any package that puts its executables to C:\Windows is misbehaving unless it was written by Microsoft and is a core part of the Operating System. I would be against Anaconda putting executables into that location. There are permissions issues here and issues about stomping over the users files.
The idea of double clicking on Python files from Explorer and running them is to me, quite a scary thing to do. Many Python files aren't intended to be run like that and who knows what they'd do if they were run like that?

mingwandroid commented 7 years ago

Did you investigate this option?:

Customized Commands
    The launcher will support the ability to define "Customized Commands" in a
    Windows .ini file (ie, a file which can be parsed by the Windows function
    GetPrivateProfileString).  A section called '[commands]' can be created 
    with key names defining the virtual command and the value specifying the
    actual command-line to be used for this virtual command.

    For example, if an INI file has the contents:

    [commands]
    vpython=c:\bin\vpython.exe -foo

    Then a shebang line of '#! vpython' in a script named 'doit.py' will 
    result in the launcher using the command-line 'c:\bin\vpython.exe -foo 
    doit.py'

    The precise details about the names, locations and search order of the
    .ini files is in the launcher documentation [4]

johnmellor commented 7 years ago

@mingwandroid wrote:

Any solution that involves adding things to PATH is not something I'd be comfortable with at all for reasons of DLL hell (...).

The py.exe and pyw.exe binaries are carefully designed to be compatible with all versions of python, so it should be fine for installers to overwrite them with more recent versions.

conda takes a different approach to 'normal' Windows CPython, emphasising cross-platform consistency which py.exe and pyw.exe work against.

How so? The only cross-platform way I've seen to indicate whether an executable script requires Python 2 or 3 is to add a shebang, and py.exe and pyw.exe are the only way to parse shebangs on Windows.

(It's also common on other platforms to have binaries called python2, python2.7, python3, python3.6 etc on the PATH; for some reason both conda and CPython create these on Linux but not on Windows.)

Any package that puts its executables to C:\Windows is misbehaving unless it was written by Microsoft and is a core part of the Operating System. (...)

I tend to agree, but that's where the CPython installer puts it by default and there's some benefit in being consistent since the launcher is supposed to be a singleton. The CPython installer does offer the alternative non-All-Users install location %LocalAppData%\Programs\Python\Launcher\ (and adds it to the user PATH rather than system PATH) which avoids elevated privileges.

The idea of double clicking on Python files from Explorer and running them is to me, quite a scary thing to do. (...)

Yes; it seems riskier than on Linux/macOS since Windows filesystems doesn't distinguish executable files. Ideally py.exe would only be the default association when running a script from the command prompt, but not when double-clicking; whilst technically possible, that seems excessively hacky. So the default should probably be not to associate *.py files, and instead have to type py foo.py to run them.

Did you investigate [defining "Customized Commands" in a Windows .ini file]?

That seems equivalent to adding registry entries for the Python 2 and 3 environments (but is less standardized, so e.g. IDEs are less likely to benefit). Perhaps you're thinking that all conda's environments could be listed in the .ini file, then you could add e.g. #!python-myenv as a shebang? Sounds cool, but not very cross-platform, unless conda on other platforms starts adding python-myenv to the PATH.

mingwandroid commented 7 years ago

conda takes a different approach to 'normal' Windows CPython, emphasising cross-platform consistency which py.exe and pyw.exe work against.

How so? The only cross-platform way I've seen to indicate whether an executable script requires Python 2 or 3 is to add a shebang, and py.exe and pyw.exe are the only way to parse shebangs on Windows.

While py.exe and pyw.exe don't exist for the other platforms, the other OSes have their equivalents. macOS has Application Bundles where file associations can be listed ( https://developer.apple.com/library/content/documentation/Carbon/Conceptual/LaunchServicesConcepts/LSCConcepts/LSCConcepts.html#//apple_ref/doc/uid/TP30000999-CH202-TP9 ) and Linux has a mime-types database for the same reason ( https://wiki.archlinux.org/index.php/default_applications ). Conda ignores these and so my argument goes that it is inconsistent to attempt to setup these associations on one OS but not on the others. Redirecting such associations on distributions that have any concept of a 'System Python' could be disastrous too (compatibility, unmet dependencies).

Conda cannot (actually there is a way - via post-install scripts - but in general should try really hard not to) write to any files outside of the installation prefix. Doing that goes against the ideas of isolation and repeatability. The argument made for putting those launchers in %SystemRoot% (that it is on PATH for both Windows and SysWOW64) do not apply to conda since an installation is either all 32-bit or all 64-bit. Writing to %SystemRoot% would require Administrator permissions and, as things stand, electing to do that would cause the entire installation to be written in a way that you need the same permissions to do conda update. The Anaconda Distribution lead is against having any post-install scripts in any of the packages that come bundled as part of Anaconda or Miniconda and due to the permissions issues they can cause I tend to agree.

That seems equivalent to adding registry entries for the Python 2 and 3 environments (but is less standardised, so e.g. IDEs are less likely to benefit).

The thing about the registry is that it's global and ends up being a huge key/value graveyard. I have an objection to putting anything in them that can be put in a file in a folder instead. At least that way, Shift-Delete on the top folder will remove that information. Having said that we do offer to register our Pythons to keep IDEs happy. We do not register any Pythons installed into non-root environments, nor do we fix-up the registration information if the Python version in the root environment is changed so already that registration is a little problematic. IMHO it would be much better for the IDEs to query the first py.exe found on PATH because at least that way the .ini file approach could also be used.

If you can figure a way around all of these issues, personally I would likely have no more objections (but others possibly still would).

Otherwise, would clearly documenting all of what you've found be good enough? Would you be prepared to make a PR to conda-docs if so?

johnmellor commented 7 years ago

it is inconsistent to attempt to setup [file type] associations on one OS [Windows] but not on the others

Yes, it's unfortunate that running in the command prompt and double clicking do the same thing on Windows. So making one consistent makes the other inconsistent :-(

Conda (...) should try really hard not to write to any files outside of the installation prefix

Thanks, it's really helpful to understand the constraints you're aiming for.

In that case it seems the best place to install py.exe and pyw.exe would be a dedicated subdirectory under the conda install root (so if that subdirectory is added to the PATH, activate.bat and deactivate.bat won't touch that PATH entry).

And I tested putting a py.ini in that same subdirectory, with customized commands for both a Python 2 and 3 environment. Unfortunately, whilst that works for #!mycustompython, it turns out that customized commands can't be called python, /usr/bin/python, /usr/bin/env python, etc (ditto with version suffixes), which rather defeats the purpose of shebangs being cross platform.

So it seems the registry is the only option to get py.exe working properly.

we do offer to register our Pythons to keep IDEs happy. We do not register any Pythons installed into non-root environments, nor do we fix-up the registration information if the Python version in the root environment is changed

Are these caveats deliberate to avoid requiring elevated permissions, or have they just not yet been prioritized? For a single user install, the registry keys are in HKCU, so should be easy to update. For an All Users install, it would indeed require a UAC prompt, though conda create and conda update seem to already often require UAC prompts because of All Users Start Menu shortcuts like Jupyter...

mingwandroid commented 7 years ago

And I tested putting a py.ini in that same subdirectory, with customized commands for both a Python 2 and 3 environment. Unfortunately, whilst that works for #!mycustompython, it turns out that customized commands can't be called python, /usr/bin/python, /usr/bin/env python, etc (ditto with version suffixes), which rather defeats the purpose of shebangs being cross platform.

I wonder if the maintainer(s) of py{,w}.exe would consider making that work. I think that would be the ideal solution here.

Are these caveats deliberate to avoid requiring elevated permissions, or have they just not yet been prioritized? For a single user install, the registry keys are in HKCU, so should be easy to update. For an All Users install, it would indeed require a UAC prompt, though conda create and conda update seem to already often require UAC prompts because of All Users Start Menu shortcuts like Jupyter..

To be honest, dealing with permissions, the Windows Registry and UAC are always really messy and full of corner cases, and avoiding it when possible saves us a lot of time and effort, so while conda itself has been making good strides lately to improve multi-user support and locked-down installations, the side of that that interfaces with Windows isn't really our focus here.

petsuter commented 6 years ago

Is there even a way to configure the py.exe launcher to use Anaconda Python? I installed the launcher separately, but sadly it doesn't seem to consider Anaconda Python at all. Is there a way?

mingwandroid commented 6 years ago

No, see https://github.com/ContinuumIO/anaconda-issues/issues/149#issuecomment-295222622

And I tested putting a py.ini in that same subdirectory, with customized commands for both a Python 2 and 3 environment. Unfortunately, whilst that works for #!mycustompython, it turns out that customized commands can't be called python, /usr/bin/python, /usr/bin/env python, etc (ditto with version suffixes), which rather defeats the purpose of shebangs being cross platform.

So it seems the registry is the only option to get py.exe working properly.

.. so this is a bug in py.exe.

petsuter commented 1 year ago

Is this now possible with https://docs.python.org/3/whatsnew/3.11.html#windows-py-exe-launcher-improvements?

petsuter commented 1 year ago

It seems to work out of the box now:

$ py -0
 -V:3.11 *        Python 3.11 (64-bit)
 -V:3.10          Python 3.10 (64-bit)
 -V:3.8           Python 3.8 (64-bit)
 -V:3.7           Python 3.7 (64-bit)
 -V:2.7
 -V:ContinuumAnalytics/Anaconda39-64 Anaconda py39_4.12.0

$ py -3.9 --version
No suitable Python runtime found

$ py -V:Anaconda --version
Python 3.9.12

So py should get bundled now, right?

dzid26 commented 1 year ago

It seems to work out of the box now:

$ py -0
 -V:3.11 *        Python 3.11 (64-bit)
 -V:3.10          Python 3.10 (64-bit)
 -V:3.8           Python 3.8 (64-bit)
 -V:3.7           Python 3.7 (64-bit)
 -V:2.7
 -V:ContinuumAnalytics/Anaconda39-64 Anaconda py39_4.12.0

$ py -3.9 --version
No suitable Python runtime found

$ py -V:Anaconda --version
Python 3.9.12

So py should get bundled now, right?

Does conda need to be in the PATH for that to work?

(base) C:\Users\dzidm>py --list
 -V:3.11 *        Python 3.11 (64-bit)
 -V:3.9           Python 3.9 (64-bit)

Btw, it was super confusing to find out that running .\example.py in conda env is not the same as python example.py.

ContinuumIO / anaconda-issues