Packaging itstool-2.0.7 does not produce a .exe file for itstool

fanc999-1 commented 1 year ago

Description

Hi,

As suggested by one of the Meson build system maintainers1, I was suggested to open a bug here:

With the following setup.py that is derived from configuring itstool on Windows, using Cygwin to call a non-Cygwin installation of Python:

#!/usr/bin/env python

# setup.py.in is converted to setup.py by the configure script. This is not
# intended as the recommended way to build and install itstool. See the INSTALL
# file for details on installing with make. This file is here to make it easier
# to upload itstool to pypi.

from distutils.core import setup

setup(name='itstool',
      version='2.0.7',
      description='XML to PO and back again using W3C ITS rules',
      author='Shaun McCance',
      author_email='shaunm@gnome.org',
      url='http://itstool.org/',
      scripts=['itstool'],
      data_files=[('c:/tools/itstool-2.0.7/share/itstool/its', [
          'its/docbook.its',
          'its/docbook5.its',
          'its/its.its',
          'its/mallard.its',
          'its/ttml.its',
          'its/xhtml.its',
      ])]
)

I was able to build a wheel using python -m pip wheel . (resulting in a itstool-2.0.7-py3-none-any.whl file), but the resulting wheel did not package the itstool script into an .exe file upon installation, but installed it as-is as a Python script, with a shebang line that pointed to the Python installation that was used to install the wheel. As a result, on Windows cmd.exe consoles, as shebang lines are not supported directly, just running itstool fails but will work when one calls python <path-to-itstool-script>.

With blessings, thank you!

Expected behavior

An itstool.exe is produced, that can be run directly if it is in the %PATH% or when called with the full path to it.

pip version

23.3.1

Python version

Python 3.7+

OS

Windows 10 x64

How to Reproduce

Download, unpack and install libxml2 and its Python module, from https://download.gnome.org/sources/libxml2/2.11/libxml2-2.11.5.tar.xz using Visual Studio 2015 or later:
```
$ cd $(sourcedir)\win32
$ cscript configure.js zlib=no iconv=no python=yes # We are using minimal deps for libxml2 here, to simplify things
```

$ nmake & nmake install $ cd ..\python # Change line 17 of setup.py from ROOT = r'.' to ROOT = r'..\win32' $ $(PYTHON) -m pip install . # This will install the libxml2 Python module with the libxml2 DLL.


2. Download the latest itstool from https://itstool.org/download.html, and unpack
3. Replace the contents of `setup.py` with the one in the description (note that only that paths should differ from the copy that shipped with the itstool tarball), and install using `$(PYTHON) -m pip install .`  There will be no `itstool.exe` in the scripts directory upon install.

### Output

_No response_

### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).

jeanas commented 9 months ago

This isn't a bug. The itstool package uses the scripts setup() argument in its setup.py file, which is documented to work like this. It's also deprecated by setuptools, precisely because of this issue, in favor of console_scripts entry points, which are easiest to configure using the [project.scripts] table in pyproject.toml.

eli-schwartz commented 9 months ago

Let me quickly quote myself from the linked issue, when I blamed this on pip and suggested the user should submit a pip bug report.

It looks like itstool uses scripts rather than entry points, so I would call this a bug in pip.

https://packaging.python.org/en/latest/specifications/binary-distribution-format/#recommended-installer-features

It is recommended to generate script wrappers on Windows, or broken things will happen. I wonder why pip isn't making them.

eli-schwartz commented 9 months ago

You say:

It's also deprecated by setuptools, precisely because of this issue,

But I wonder what the logic is in:

pip declining to implement a PEP recommendation
then advising people that due to the lack of implementing a PEP recommendation the functionality is ill-advised and people should not use it
then closing suggestions to implement the PEP recommendation because it's commonly advised to not use the functionality due to lack of implementation

To be clear: I'm not complaining that no one has implemented it, but I do kind of wonder whether the ticket should be kept open as a wishlist item, since it seems likely that scripts do exist, and they aren't actually specific to setuptools in the first place, they're part of the wheel format -- so whether setuptools formally deprecated it isn't actually going to change the likelihood that people will use it.

It would be nice to have some visibility to external contributors reading the bug tracker for discussion about the topic to have an easy way to find references to the idea, and possibly even find time to write up a PR to implement it.

eli-schwartz commented 9 months ago

Also, for context: meson works around this general domain issue (python scripts on Windows where shebang lines are the height of unreliable) by detecting python scripts and attempting to run it with a random python interpreter (actually the same one that meson itself is installed with) in the hope that it will sometimes work assuming you only have one python environment with all tools installed.

This is the right choice for standalone scripts, but not a great choice for scripts installed as part of installing a wheel. It will still work for itstool as long as the user installs both itstool and meson in the same environment. Meson also supports "machine files" which can be used to manually override the array containing the itstool command, including the python executable which should launch it.

(The linked issue includes a changeset to make meson's use of itstool actually respect all this configurability.)

jeanas commented 9 months ago

@eli-schwartz It sounds like you are believing pip does not implement .exe generation, which is not true. It does, when the package uses the proper feature, namely console_scripts entry points. That is what [project.scripts] in pyproject.toml sets, and it's been recommended for a looong time. The scripts keyword argument in setuptools is a legacy feature that is discouraged in favor of console_scripts because console_scripts works on Windows and scripts doesn't.

When you use scripts, this is what happens:

~/tmp/proj $ tree
.
├── pyproject.toml
├── script.py
└── setup.py

1 directory, 3 files

~/tmp/proj $ cat pyproject.toml 
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "proj"
version = "0"

~/tmp/proj $ cat setup.py 
import setuptools

setuptools.setup(scripts=["script.py"])

~/tmp/proj $ cat script.py 
print("Hello, World!")

~/tmp/proj $ python -m build
[snipped]

~/tmp/proj $ cd dist/

~/tmp/proj/dist $ unzip proj-0-py3-none-any.whl 
Archive:  proj-0-py3-none-any.whl
  inflating: script.py               
  inflating: proj-0.data/scripts/script.py  
  inflating: proj-0.dist-info/METADATA  
  inflating: proj-0.dist-info/WHEEL  
  inflating: proj-0.dist-info/top_level.txt  
  inflating: proj-0.dist-info/RECORD

The script ends up in the scripts subdirectory of the .data directory, which basically means it's moved directly under some location on $PATH upon installation. This cannot be changed (short of changing the wheel standard) because it is a feature of the wheel format that can be useful in rare cases. For example, you could put an executable or a shell script under scripts/. You don't want to make exe wrappers for that sort of thing. You could even have both a foo.py and a foo.exe under scripts/ and creating an exe wrapper for foo.py would conflict.

That is also the reason setuptools retains its legacy support for the scripts keyword.

See also https://discuss.python.org/t/whats-the-status-of-scripts-vs-entry-points/18524/4 with multiple pip and setuptools maintainers commenting on the issue.

eli-schwartz commented 9 months ago

@eli-schwartz It sounds like you are believing pip does not implement .exe generation, which is not true. It does, when the package uses the proper feature, namely console_scripts entry points.

Incorrect.

I understand and appreciate that pip implements exe generation for the broader community's favored technology, and believe that pip would be even better if it stuck with the spec's recommendation and didn't discriminate against scripts (a valid wheel feature) just because more people prefer to use entrypoints.

The scripts keyword argument in setuptools is a legacy feature that is discouraged in favor of console_scripts because console_scripts works on Windows and scripts doesn't.

It is not time effective for me to listen to you say this in your initial close message, explain why this logic is circular reasoning, and then have my reply get responded to with...

... not a response, but a reiteration. Of the original circular logic.

Which is another kind of circular logic, but oh well.

You may refer back to my original comment if you wish to know my opinions on whether this should or should not be discouraged, although I'm happy to respond if new information comes up.

The script ends up in the scripts subdirectory of the .data directory, which basically means it's moved directly under some location on $PATH upon installation.

Thanks, I'm aware.

This cannot be changed (short of changing the wheel standard) because it is a feature of the wheel format that can be useful in rare cases. For example, you could put an executable or a shell script under scripts/. You don't want to make exe wrappers for that sort of thing.

Sorry, there seems to be some confusion here. If you read the wheel standard, you'll see that its existing recommendation (which does not need changing) is to make exe wrappers for python scripts.

Not make exe wrappers for exes. Not make exe wrappers for posix sh scripts.

Just make exe wrappers for python files, to accompany the recommendation that wheel installers should rewrite the shebang literal b'#!python' to point to the python installation which is being installed to. Which as I recall, pip already does.

You could even have both a foo.py and a foo.exe under scripts/ and creating an exe wrapper for foo.py would conflict.

If the user has generated their own exe wrapper already I have no objections whatsoever to refusing to generate one and in the process overwrite an existing file.

However I'm very curious whether this criticism equally applies to entrypoints in the event that a foo entrypoint is defined simultaneous to a scripts/foo.exe and pip cannot generate an entrypoint without overwriting an existing file.

pfmoore commented 9 months ago

Let’s reopen this for now. @eli-schwartz has a valid point here, and while I think pip has made an acceptable choice here (not generating wrappers preserves the setuptools behaviour) it’s fair to ask the question.

Note that this is definitely not a bug, though. Pip is acting as designed, and the behaviour is allowed by the spec. Generating wrappers would be a feature request, not a bug.

And to be fair, your tone did not exactly motivate me to click your link and re-read the spec either. This is the second time I've interacted with you on the Internet and the second time you've been abrasive from the start. I doubt this is helpful to anybody.

@jeanas lets keep personal comments out of the discussion. It’s not helpful, and we expect a more welcoming approach. Also, I re-read the comments by @eli-schwartz and I don’t see anything “abrasive” in them. Please be careful in how you interpret comments here.

jeanas commented 9 months ago

I've deleted the comment. Staying fully calm is not going to be possible for me here so I've unsubscribed from the thread as well.

pypa / pip