Deprecate call to `setup.py install` when building a wheel failed for source distributions without pyproject.toml

sbidoul commented 4 years ago

This is one of a series of deprecations meant to collect feedback on the ultimate goal of always installing source distributions by first building a wheel and then installing from it.

This specific issue is about the case when pip calls setup.py bdist_wheel and that fails. In that case, pip currently displays the build error and continues installation by attempting a setup.py install.

In a future version, pip will not attempt setup.py install in that case, and fail the installation right away.

Towards #8102

nhatkhai commented 3 years ago

So can I still make .egg package and delivery it or not? I need a post-install process for devices drivers if not presented, and other crazy windows software installer.... in order to automate and resolve all dependency for my python package.

uranusjr commented 3 years ago

This change only affects source distributions with setup.py but not pyproject.toml. It has nothing to do with eggs.

nhatkhai commented 3 years ago

@uranusjr - It is my case. My packages is stop work with pip 21.x now. What do I do... Is this change because of some security issues?

uranusjr commented 3 years ago

This deprecation has not even been implemented yet. If your package already stopped working, it has nothing to do with this issue.

tomashek commented 3 years ago

Hello. We are considering implementing a package installation using sdist only because we need some internal relative-path symlinks created at installation time and wheels cannot include symlinks. We have not found a way to have the wheel installation process include symlinks, but it was suggested that an sdist could create symlinks in the setup.py. The idea was: create a package with the standard files as a wheel, then create a dependent package sdist only that would create the needed symlinks upon installation.

If such a use-case is (or will soon be) deprecated, we will not pursue it. Any information or alternate suggestions would help.

uranusjr commented 3 years ago

Symbolic links in wheels is a topic that came up a lot of times at discuss.python.org. I think it’s more or less agreed that it’s a valid use case, but the problem is how to specify and implement it correctly since wheel is a cross-platform format and symbolic links are decidedly not cross-platform. I would encourage you to raise this issue on d.p.o again and drive that discussion to the end. It would be tremendous help to drive this deprecation home as well.

tomashek commented 3 years ago

Yes, I understand that the cross-platform challenge with symlinks. That is why a sensible workaround is to implement them at installation time using setup.py. The requirement to create a wheel, though, makes even a source installation on a system need to be "cross-platform", which looks like a needless imposition since there is no implied requirement that the result build be installed on any other platform.

Am I understanding that there is no "pip install" option that does not involve creation of a wheel, and thus no option (even sdist) that would allow "pip install" to include some created symlinks?

uranusjr commented 3 years ago

The wheel format is cross-platform, but a singular wheel does not neeed to necessarily be cross-platform (per-platform support is why wheels exist in the first place). The problem is not that a wheel cannot be used to distribute packages with symlinks, but how it should be designed to do it.

tomashek commented 3 years ago

"The wheel format is cross-platform, but a singular wheel does not neeed to necessarily be cross-platform" -- okay, but in practice, when building a wheel you are limited by the format right now. Unless there is a way to say "allow symlinks" for osx/linux wheel builds (aligns with your comment in 2019 discussion to allow it for *nix platform wheels).

uranusjr commented 3 years ago

There will be a way if someone designs one 🙂

tomashek commented 3 years ago

I don't have the resources to fix the symlink problem. I am commenting here because the limitation imposed by deprecating "pip install" of an sdist running setup.py without creating a wheel creates a problem right now because there is no other accepted symlink solution (in answer to the question above by @sbidoul : "it would be interesting to understand why setup.py bdist_wheel would not work for you, and what would block you from moving to PEP 517")

alex-ber commented 3 years ago

I'm using Docker with Alpline Linux on ARM64 with Anaconda3-2021.05, Python 3.8.8 . I'm using pip==20.3.1. I have following line in my Dockerfile:

pip install 'numba==0.53.1' 'llvmlite==0.36.0' 'sip==4.19.25' 'numpy==1.19.5'

That produces the following warning:

+ > pip install 'numba==0.53.1' 'llvmlite==0.36.0' 'sip==4.19.25' 'numpy==1.19.5'
Requirement already satisfied: llvmlite==0.36.0 in /opt/anaconda3/lib/python3.8/site-packages (0.36.0)
Requirement already satisfied: sip==4.19.25 in /opt/anaconda3/lib/python3.8/site-packages (4.19.25)
Collecting numba==0.53.1
  Downloading numba-0.53.1.tar.gz (2.2 MB)
Requirement already satisfied: llvmlite==0.36.0 in /opt/anaconda3/lib/python3.8/site-packages (0.36.0)
Requirement already satisfied: setuptools in /opt/anaconda3/lib/python3.8/site-packages (from numba==0.53.1) (51.0.0)
Collecting numpy==1.19.5
  Downloading numpy-1.19.5-cp38-cp38-manylinux2014_aarch64.whl (12.4 MB)
Building wheels for collected packages: numba
  Building wheel for numba (setup.py): started
  Building wheel for numba (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /opt/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jwlc7kja/numba_11d72221b9e14cc89b75278cf5dfedc1/setup.py'"'"'; __file__='"'"'/tmp/pip-install-jwlc7kja/numba_11d72221b9e14cc89b75278cf5dfedc1/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-gsv3ina6
       cwd: /tmp/pip-install-jwlc7kja/numba_11d72221b9e14cc89b75278cf5dfedc1/
  Complete output (7 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-jwlc7kja/numba_11d72221b9e14cc89b75278cf5dfedc1/setup.py", line 422, in <module>
      metadata['ext_modules'] = get_ext_modules()
    File "/tmp/pip-install-jwlc7kja/numba_11d72221b9e14cc89b75278cf5dfedc1/setup.py", line 148, in get_ext_modules
      import numpy.distutils.misc_util as np_misc
  ModuleNotFoundError: No module named 'numpy'
  ----------------------------------------
  ERROR: Failed building wheel for numba
  Running setup.py clean for numba
  ERROR: Command errored out with exit status 1:
   command: /opt/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jwlc7kja/numba_11d72221b9e14cc89b75278cf5dfedc1/setup.py'"'"'; __file__='"'"'/tmp/pip-install-jwlc7kja/numba_11d72221b9e14cc89b75278cf5dfedc1/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
       cwd: /tmp/pip-install-jwlc7kja/numba_11d72221b9e14cc89b75278cf5dfedc1
  Complete output (7 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-jwlc7kja/numba_11d72221b9e14cc89b75278cf5dfedc1/setup.py", line 422, in <module>
      metadata['ext_modules'] = get_ext_modules()
    File "/tmp/pip-install-jwlc7kja/numba_11d72221b9e14cc89b75278cf5dfedc1/setup.py", line 148, in get_ext_modules
      import numpy.distutils.misc_util as np_misc
  ModuleNotFoundError: No module named 'numpy'
  ----------------------------------------
  ERROR: Failed cleaning build dir for numba
Failed to build numba
Installing collected packages: numpy, numba
    Running setup.py install for numba: started
    Running setup.py install for numba: still running...
    Running setup.py install for numba: still running...
    Running setup.py install for numba: still running...
    Running setup.py install for numba: still running...
    Running setup.py install for numba: finished with status 'done'
  DEPRECATION: numba was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
Successfully installed numba-0.53.1 numpy-1.19.5

Previously, the following line looks like:

pip install 'numba==0.53.1' 'llvmlite==0.36.0' 'sip==4.19.25' 'numpy==1.16.2'

And it works as expected (numpy was built from source that is exactable behaviour, because this version is pretty old). This version of numpy actually does has wheel for arm64. Numba is installed from source, because it doesn't have the wheel for ARM64, that is ok.

uranusjr commented 3 years ago

What does “previously” mean?

alex-ber commented 3 years ago

I did change in the Docker file. It was pip install 'numba==0.53.1' 'llvmlite==0.36.0' 'sip==4.19.25' 'numpy==1.16.2'

and worked without any warning. With the same pip version.

uranusjr commented 3 years ago

In that case, your issue is not related to this deprecation. You are seeing it only because pip fails to build a version from source; it did not attempt the build because it found a compatible prebuilt binary. This indicates you have a conflict in your dependencies. You can ignore the final warning message about setup.py install.

Please feel free to seek support on more user-oriented forums, such as StackOverflow and the online communities listed in https://www.python.org/community/

pradyunsg commented 3 years ago

prophet

Please file an issue against this project, that they're requiring an install via setup.py install.

For avoiding this message on your end, you can manually install the build-time dependency of this (convertdate) before it similar to how you're installing Cython earlier.

hbksilver commented 3 years ago

So anyone found the solution ?

sbidoul commented 3 years ago

So anyone found the solution ?

@hbksilver The solution to which problem ?

drtoche commented 3 years ago

The message below brought me here, but I must admit I have no idea why I'm here...

WARNING: Built wheel for my_package_name is invalid: Metadata 1.2 mandates PEP 440 version, but 'dev' is not
Failed to build my_package_name
Installing collected packages: my_package_name
  Attempting uninstall: my_package_name
    Found existing installation: my_package_name dev
    Uninstalling my_package_name-dev:
      Successfully uninstalled my_package_name-dev
    Running setup.py install for my_package_name ... done
  DEPRECATION: my_package_name was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. A possible replacement is to fix the wheel build issue reported above. Discussion can be found at https://github.com/pypa/pip/issues/8368
Successfully installed my_package_name-dev

sbidoul commented 3 years ago

@drtoche the important part is WARNING: Built wheel for my_package_name is invalid: Metadata 1.2 mandates PEP 440 version, but 'dev' is not. Your package has an version number (dev) which is invalid, therefore pip rejects the wheel that was built from it, and falls back to the legacy setup.py install method which is less strict

Setting a valid version number in your project should fix both the warning and the deprecation.

drtoche commented 3 years ago

Thanks @sbidoul!

Arpafaucon commented 2 years ago

Hi maintainers !

I was wondering: is there a way to ask pip to be stricter and abort installation in this case ?

Your comment mentions:

In version 21.0, pip will not attempt setup.py install in that case, and fail the installation right away.

But with pip 21.3.1, I still am able to install local packages with invalid versions:

# in a package `mypkg`, with version in setup.py `tag-rc1-342-g3c3542fcb`
$ pip install .
Looking in indexes: <REDACTED>
Processing /src/python/mypkg
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: mypkg
  Building wheel for mypkg (setup.py) ... done
  Created wheel for mypkg: filename=mypkg-tag_rc1_342_g3c3542fcb-py3-none-any.whl size=29290 sha256=026b83a9021a9d2ad51c0586b3f2e68966311a290ec4df8781249f8f3cad58e7
  Stored in directory: /tmp/pip-ephem-wheel-cache-nbejno8l/wheels/64/3e/4e/73312ba594e268dfa32dcdc32f5d399eee17eb8104bbe0c132
  WARNING: Built wheel for mypkg is invalid: Metadata 1.2 mandates PEP 440 version, but 'tag-rc1-342-g3c3542fcb' is not
Failed to build mypkg
Installing collected packages: mypkg
    Running setup.py install for mypkg ... done
  DEPRECATION: mypkg was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. A possible replacement is to fix the wheel build issue reported above. Discussion can be found at https://github.com/pypa/pip/issues/8368
Successfully installed mypkg-tag-rc1-342-g3c3542fcb

I had a look at this issue but may have failed to see the answer I'm looking for (sorry if this is the case).

uranusjr commented 2 years ago

One way to make pip “stricter” is --use-pep517. Stricter in quotes, because this is not exactly the same as disabling setup.py install; the option basically makes pip always pretend there is a pyproject.toml for each package (instead of using the globally installed setuptools), but that should introduce no functional differences for most packages. And since projects with pyproject.toml are never installed with setup.py install, the option kind of achieves what you want.

vsrivastav-pm commented 2 years ago

I am getting a below message on installing a python package with custom version 1.45.0webrc1

  WARNING: Built wheel for pm-selenium is invalid: Metadata 1.2 mandates PEP 440 version, but '1.45.0webrc1' is not
    DEPRECATION: pm-selenium was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. A possible replacement is to fix the wheel build issue reported above. Discussion can be found at https://github.com/pypa/pip/issues/8368

Is there a way to silent these warnings ?
~~Will this warning cause build failures in the future ?~~ is the legacy setup.py going to get discontinued anytime soon ?
Also, a follow-up question, will the above versioning cause build failures if a wheel doesn't get built ?

because i really want to use custom release names 💯

pfmoore commented 2 years ago

is the legacy setup.py going to get discontinued anytime soon

Yes. I'm not sure without checking what the timescale is, but we will be removing this fallback.

Also, a follow-up question, will the above versioning cause build failures if a wheel doesn't get built

I'm not sure what you mean by this, what sort of build are you thinking of apart from building a wheel? All forms of direct build via setup.py have now been deprecated (by setuptools, not by pip) so at some point you'll have to move to a PEP 440 compliant version. And even if you don't get problems with build failures, you'll start hitting issues with other tools which only support PEP 440 style versions.

because i really want to use custom release names

Sorry, the standards don't support them, so you'll increasingly find that standard tools will reject them or fail.

vsrivastav-pm commented 2 years ago

@pfmoore thanks for the quick response confirming that legacy setup.py will removed in the future, will resort to PEP 440 style versions for pre-releases like

X.YrcN  # Release Candidate
1.1.2rc1-1

wushuangpojun commented 2 years ago

Seek guidance, python3.10.2 pyzmq19.0.2 installation failure, where is the problem?

pradyunsg commented 2 years ago

Reach out to pyzmq for help.

wushuangpojun commented 2 years ago

Seek guidance, python3.10.2 pyzmq19.0.2 installation failure, where is the problem?

Building wheel for pyzmq (setup.py) ... error error: subprocess-exited-with-error × python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [248 lines of output] C:\Users\无双破军\AppData\Local\Temp\pip-req-build-5npw69cc\setup.py:1140: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. if V(Cython.version) < V(min_cython_version): C:\Users\无双破军\AppData\Local\Temp\pip-req-build-5npw69cc\setup.py:1148: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. Warning: Couldn't find an acceptable libzmq on the system. Warning: Failed to get cc._vcruntime via private API, not bundling CRT AttributeError: 'MSVCCompiler' object has no attribute '_vcruntime_redist' [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for pyzmq Running setup.py clean for pyzmq Failed to build pyzmq Installing collected packages: pyzmq Running setup.py install for pyzmq ... done DEPRECATION: pyzmq was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. A possible replacement is to fix the wheel build issue reported above.

osmalpkoras commented 2 years ago

I do experience issues on an Apple MacBook Pro (13-inch, M1, 2020) with macOS Monterey 12.3.1. To be particular: when calling pip install deidentify, I get ERROR: Failed building wheel for gensim. I dont get these issues on other machines and I think this fails particular for the M1 architecture. Are wheel issues for this architecture being tracked and if yes, where?

sbidoul commented 2 years ago

@osmalpkoras the build error for gensim was likely printed and you should raise it with that project.

IgorGanapolsky commented 2 years ago

I am unable to install mitreattack-python. It keeps saying

DEPRECATION: cairocffi was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. A possible replacement is to fix the wheel build issue reported above. Discussion can be found at https://github.com/pypa/pip/issues/8368

sbidoul commented 2 years ago

@IgorGanapolsky the build error for cairocffi was likely printed and you should raise it with that project.

KhomZ commented 2 years ago

is it not possible to use tensorflow.examples in colab???

pfmoore commented 2 years ago

is it not possible to use tensorflow.examples in colab???

That's a question for colab support, not here...

KAJURAMBO commented 2 years ago

× python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [23 lines of output] Partial import of sklearn during the build process. Traceback (most recent call last):

Hi,while deploying the application on heroku I am getting following error regarding the wheel.

sbidoul commented 2 years ago

@KAJURAMBO you can probably reproduce the locally by running python -m pip install wheel then python setup.py bdist_wheel and you'll need to find a way to fix that.

smartpierre commented 2 years ago

got this issue while trying to package a docker image using python:3.10-slim-bullseye, switched to python:3.10-slim-buster and everything worked fine

sbidoul commented 2 years ago

In https://github.com/pypa/pip/pull/11456 we are targeting the removal of this setup.py install fallback to pip 23.1 (April 2023).

mattvonrocketstein commented 2 years ago

Moving forward with this will break stuff for me and create a bunch of work. I do like wheel builds, but I think the latest wheel build process is too opinionated, because it breaks with Metadata 1.2 mandates PEP 440 version, but '{git_hash}' is not.

Yes, I really do want git hashes, even thought they don't sort with naive comparison operators. My use-case is that this makes it simple to enforce that python lib versions == github versions == docker tags, and they are all the same value. No, I don't want to change all my processes to ship PEP-440 compatible identifiers to ECR/docker-hub or add a bunch of git tag stuff to existing CI, etc. No, I don't want {semver}-{git_hash} or {cal_ver}-{git_hash}, because those won't work in URLs for github without us mandating additional steps for release-tagging. I don't specify dependencies with inequalities like pkg<=..., so this just isn't a use-case for me.

Due to overreach in the wheel-build process, I am forced to ask that you guys please not change this process. Sorry. I realize that PEP-440 is standards-track, but afaik it leaves open things like enforcement. Pip should only throw a fatal error for a "bad" version string if and only if the user is actually asking for something fancy using <= or ~=, etc. Otherwise we'll soon be in the odd situation where packages can be shipped to pypi successfully, but can't be installed from there.

Please do let me know if there's other places/issues where I need to go to argue this point.

dstufft commented 2 years ago

PyPI mandates PEP440 versions Sent from my iPhone

pradyunsg commented 2 years ago

(whelp, wall of text)

@mattvonrocketstein I understand and empathise that this change will require you to change your workflow and you aren’t willing to do so. We're aware that this will be a disruptive change for users such as yourself -- our goal with these warnings prior to enforcement is to allow for users (like yourself) who rely on setup.py install to come up with approaches to deal with the fact that this change is happening.

As a concrete suggestion, no one will force you to update to the latest pip the moment it is released. You have the option of sticking to an older version of pip -- the main consequence of that is that you won't be able to reach out to pip's maintainers for requesting support; or at least, need to reproduce the issue with the latest pip before asking for support. One way around that is to use Python + pip provided by enterprise Linux distributions -- those will include support and back-ported security fixes from their Python/security teams, while still staying on an older version of pip (at least, as long as their support cycle allows).

Allow me to elaborate on why you might feel like everyone else has already decided to not accommodate for your use case, when advocating against the removal of setup.py install code paths to preserve support for installing packages with arbitrary version strings.

When you found out that your workflow was no longer supported when using wheels, and moved to the setup.py install workflow; you also moved to a significantly more problematic mechanism that has taken the largely-volunteer maintainers a lot more time to untangle + come around to deprecating, and you are now asking for them to not remove the more problematic mechanism — something that has been the direction that broader community efforts have been working toward for... well... many years. That’s not going to fly, no matter how passionately you advocate for it.

The only reason that non-PEP 440 style versions work for you today is because pip/setuptools etc have maintained a non-trivial amount of complexity to deal with them, and because they haven’t yet enforced consistency in that code path. setup.py install has been treated as undesirable for quite a while, among both maintainers and many users due to the complexities that it presents and the broken behaviours it has. Not removing it because it’ll break workflows that rely on the lack of enforcement of basic needs for pip — a package manager needs to be able to understand version strings passed to it, and parse them.

Note that non-PEP 440 versions with setup.py install present a warning from setuptools’ end that they are going to be removed (https://github.com/pypa/setuptools/blob/47cec1cd84def9d64f9251a403bb2305273efc9a/setuptools/dist.py#L544-L549) and from pip's end we have the intention documented as well (https://pip.pypa.io/en/stable/reference/build-system/setup-py/#direct-installation). This issue is about acting on what pip's documentation and setuptools' warnings already state.

You might be wondering what the "complexities that it presents and the broken behaviours it has" are. Here's a non-exhaustive list, that I can come up with in... let's time box this to 5 minutes.

Quirky version comparisions based on whether a version string is a PEP 440 string or not -- all legacy versions are sorted as "lower" than PEP 440 versions.
Increased user confusion due to setup.py install being executed after setup.py bdist_wheel failing.
Warnings from pip about build failures despite a successful installation since setup.py bdist_wheel didn't work but setup.py install does; resulting in user confusion as well as making the underlying code much more complex.
It uses entry point scripts launches that are generated by setuptools, which do not behave like the ones that pip generates.
It does not have an equivalent in a pyproject.toml-based build system, which means either maintaining the legacy codepath forever, doubling the maintainance costs and complexities in pip and setuptools, even though the pyproject.toml-based build system was explicitly designed to be compatible with the setup.py-based build system and the additional costs were deemed acceptable for a multi-year transitory period.
A whole lot of logic and cognitive complexity in pip's and setuptools' codebases to ensure that we don't eagerly parse versions or to otherwise allow allow for non-PEP 440 versions to pass through -- which make paying down technical debt within these codebases more complicated, which makes contributing to these projects more difficult, which makes it less likely that volunteers work on these projects or new contributors start contributing to it.

Ok, time box over. I'll flesh those out into sentences now.

All this is to say, this change is a heavily-loaded one that has a lot more complexities to it than "oh, we don't like arbitrary versions" and disallowing arbitrary versions is a understood (and, depending on who you ask, an appreciated) thing that this will trigger. We're aware of that, and that's why we're being considerate and slow in making this change (first warning for a period, before the eventual removal).

Outside of the coupling with setup.py install, looking only at the version-related usage of yours -- I am not aware of any package management system that allows users to specify arbitrary strings as versions [^1], and I don’t think Python's should block significant improvements to support use cases that rely on that working; especially if they only work within deprecated portions of Python's packaging systems today.

[^1]: This is a bit of a tangent -- if you know of a package management system like that, I’d really like to know for the sake of curiosity, if nothing else. Whether it would be a weak argument in favour of supporting preserving such support, would be a nice case study.

mattvonrocketstein commented 2 years ago

PyPI mandates PEP440 versions

@dstufft i assume you mean public pypi. but the situation has changed since 2014.. privately hosted pypi is a thing now whether we like it or not. so my point stands that moving forward with this means: pip will be unable to install things that are hosted on pypi.

if my private pypi is not enforcing pep-440 compliance and the public one is, i think that raises the basic question: where are the official helm-charts (or similar) for self-hosting pypi? and: is pip intended to work only with public-pypi? do other tools like artifactory enforce pep-440 today, and if so have they done that for long enough that we think it's ok? another basic question.. do you guys actually have telemetry that suggests that most usage of pip is for public pypi, or did you just assume that? what percentage is ok to break? i think you also need to consider that users of PaaS deployments like lambda, MWAA, etc often cannot use public-pypi, and may not have a lot of easy control over packaging details for their dependencies, or things like the version of pip they are using.

so i think this is bigger than just modernizing individual libraries to use the latest obscure combination of setup.cfg/setup.py/pyproject.toml. but on that subject.. where is the actual reference implementation that pip recommends for what modern packaging is supposed to look like? packaging needs are certainly myriad and diverse, but i think what people need are production-ready templates that cover real use-cases. whenever I look for this stuff I find only piecewise or toy examples, and then I'm back to hours of sifting out-dated advice/patterns on blogs and stack-overflow. especially if you're strictly mandating version-strings your way going forward, then i think these project templates need to directly demonstrate practical real-world release processes end-to-end. normally i'd say things like how to use versioneer/scmtools or whatever is up to users, but if you insist on removing user choice that's problematic. versioneer in particular supports pep440 but it is only one option of many. the project boilerplate reference that pip really should be providing needs to pin things, be maintained alongside changing versions of pip, changing versions of versioneer, twine, etc. if pip doesn't want the responsibility of mandating the choice or demonstrating the usage of something like versioneer, then i just think pip needs to stay out of the business of mandating version-string particulars ¯_(ツ)/¯ because otherwise there's just way too much friction in the package-bootstrap process. is there a reference like this that you want to point people at? are there plans to go the poetry/pipenv route where pip itself can at least pave a working project for me with current version of pip?

philosophically, what i see in https://github.com/pypa/pip/pull/1894 is a lot of odd debate/machinations to solve problems that the pep itself is creating. i don't understand all the effort at gaming the standard to optimize for harassing the smallest number of maintainers/users about compatibility, when of course it's very straightforward to just not harass them. i realize that ship has sailed, for better or worse. but maybe we can get clarity on one point: does the PEP require/discuss enforcement or not? afaik, one way to read it is "if you don't comply with new standards, you won't get new features", and in that sense I support the PEP and for some projects would want the features it adds! it is a different matter though to intentionally break legacy projects and create the push-but-not-pull situation with pypi mentioned above. pip seems to be making a choice here that's not explicitly spelled out by the pep, and which is hostile to both users and maintainers.. there's simply no need to break compatibility to get the benefits of pep440 for use-cases that require it. at best it's quite confusing/inefficient to ship advice about standards for almost 10 years and then pivot to enforcement like that was the plan all along. at worst it's disingenuous. but i guess this is urgent now if the competition is more pep440 compliant than pip!

more practically, how long do we have before you force this breakage in new pips? if I pin to older version of pip, how long before that breaks? i think most maintainers/users are simply trying to get by without getting sucked into the great python packaging wars, and we are dealing with constant breakage and churn in the messy packaging/dependency world that we're stuck with. pep440 isn't really fixing that, it's making it worse.. and breakage/backwards incompatibility in packaging is almost as bad as if it were in the language itself. again this has been safe to ignore for almost a decade, if maintainers even knew about it, which of course tends to lend weight to the expectation that standards-compliance is advice that buys us new functionality rather than just the promise of pointless but necessary make-work to avoid breakage. i think there are also significant and common workflows where this will probably be the first time maintainers are hearing about pep440 (i'll add more details about that later in follow up). since pip is always giving advice to pip install --upgrade pip and because that's usually the best way to fix mysterious emergent new problems with breaking changes, i think you can expect that many are following this advice from CI/CD. that's ill-advised, because of course it was always a toss-up whether it would fix things or break things, but it looks like the pendulum is about to swing sharply towards the latter. so my bet is that many will only notice for the first time when their CI breaks or their deployments are unusable.

dstufft commented 2 years ago

i assume you mean public pypi. https://github.com/pypa/pip/pull/1894#issuecomment-47245946.. privately hosted pypi is a thing now whether we like it or not. so my point stands that moving forward with this means: pip will be unable to install things that are hosted on pypi.

I mean PyPI, not a PyPI compatible thing that isn't PyPI.

if my private pypi is not enforcing pep-440 compliance and the public one is, i think that raises the basic question: where are the official helm-charts (or similar) for self-hosting pypi? and: is pip intended to work only with public-pypi? do other tools like artifactory enforce pep-440 today, and if so have they done that for long enough that we think it's ok? another basic question.. do you guys actually have telemetry that suggests that most usage of pip is for public pypi, or did you just assume that? what percentage is ok to break? i think you also need to consider that users of PaaS deployments like lambda, MWAA, etc often cannot use public-pypi, and may not have a lot of easy control over packaging details for their dependencies, or things like the version of pip they are using.

There is only one valid way to specify a version in Python packaging-- that is PEP 440.

Some projects may choose to emit invalid metadata, and other projects may choose to make a best effort at interpreting invalid metadata, but at the end of the day projects that don't emit a PEP 440 version are producing invalid metadata.

There is of course a tension here where there were projects that were producing valid metadata, then a decade ago we declared those invalid after the fact. We attempted to minimize that, but ultimately there were problems solved by the "just support anything as a version number" approach that lead us to standardize PEP 440.

If you disagree with the decision to standardize on PEP 440, that's your right, but that ship has sailed unless you want to write a PEP and gain support for a PEP to roll it back.

one way to read it is "if you don't comply with new standards, you won't get new features", and in that sense I support the PEP and for some projects would want the features it adds!

if I pin to older version of pip, how long before that breaks?

There seems to be some confusion in how pip functions here. If you do not upgrade your version of pip, you are isolated entirely from changes made in newer versions of pip. You can only possibly be affected by this change if someone is trying to use a newer version of pip to install your project that has invalid metadata.

Where this tends to get people is that it's not just the project author's version of pip, but all of their users' as well. With a private PyPI you tend to have more ability to mandate that you have to use a sufficiently old version of pip, with public PyPI you generally have a harder time doing that.

mattvonrocketstein commented 2 years ago

Based on this deprecation warning, it looks like you are planning to lead the PEP-440 enforcement charge from pip install. This is backwards, and makes no sense for the following reasons:

Leaving aside the issue of whether installers should refuse what registries have accepted, builders should not build what installers will refuse. For the project boilerplate I'm looking at, pip wheel . has been broken for a while probably, but that's not even the recommended approach. The newer approach recommended by docs uses python -mbuild, which complains about pep440 quietly at the top of some very noisy output, and then loudly declares victory in green "Success building.." text, and obligingly creates a wheel and a tarball. (There is no error and the warning is easy to miss, especially when run from CI or as just a piece of a larger automation task with tox, etc.) It's unclear why newer tooling like -mbuild should bring back a workflow older tooling like pip wheel is trying to deprecate, but the fact remains that supported/recommended workflows here lead to contradictory behaviour.
Leaving aside the issue of whether installers should refuse what registries have accepted, uploaders should not upload what registries will refuse. Since pypa maintains twine, and twine has an issue for some time about pep-440 enforcement, it seems like there is awareness of this problem. Twine should be the obvious way that PYPA can require something like "server-side" pep-440 enforcement, even when PYPA doesn't control the registry. Again, supported/recommended workflows here would lead to contradictory behaviour.
If you must break something, it's safest to break builds before breaking deployments. Every package legitimately uploaded to a package registry in the last couple of years is potentially a roll-back target for something. Unfortunately pip install has workflows that are related to both development and deployment. If I use pipenv for local development there's no warning and no error for pep440 enforcement. If I then use -mbuild and twine to ship packages, there's no enforcement. If pip install is used from PaaS, then I don't see warnings on headless servers, I just see successful deployments and I don't check logs. So if pip install breaks, I am hearing about changes first from my broken deployment/rollback. The net result here is that end-users or ops is hearing about breakage before developers/maintainers are. And again, contradictory.. PYPA has effectively blessed my package with three different interactions and suddenly choked on the 4th and most important interaction. When that happens, it's unlikely the person seeing that breakage can actually take steps to fix it.

Some projects may choose to emit invalid metadata, and other projects may choose to make a best effort at interpreting invalid metadata, but at the end of the day projects that don't emit a PEP 440 version are producing invalid metadata.

In retrospect, this reads like it might be trying to preemptively dismiss my legitimate confusion/concern with the stuff above, but I hope that's not the case. I don't think it's ok if PYPA's own toolchain is throwing internal consistency out the window.

I mean PyPI, not a PyPI compatible thing that isn't PyPI.

Well there's a lot to unpack here. Still not sure what you're implying since I thought pip was a package installer and not intended to be exactly/only a client for public-pypi. And this raises the questions of whether warehouse is PyPi enough, and whether it enforces pep-440? It seems that folks in this thread know that it does not. (This is confusing since I'm not sure how PyPi has enforcement-features that are missing from warehouse, if PyPi is using warehouse). Not that it matters much, because docs lead me to believe that Warehouse is not actually usable today for a private registry anyway.

So in terms of policy, it seems to me that exactly one the following things must be true:

pip does not acknowledge private-registries as a valid use-case- in which case python effectively only supports OSS development.
pip does acknowledge private-registries as a valid use-case, but only supports warehouse- in which case pip is blocked on requiring new metadata until warehouse is actually usable, and pep-440 aware, and ideally in common usage for some years
pip is agnostic about which registry is being used- in which case pip is blocked on requiring new metadata until PYPA is enforcing metadata standards even across third-party registries using twine, ideally for some years.

There is only one valid way to specify a version in Python packaging-- that is PEP 440.

And one valid package registry also apparently :roll_eyes: - all solutions are very tidy if we're ignoring the real world. But ok. One valid way, so I'll tell my friends, and you can tell yours. You can start with pipenv, -mbuild, twine, and warehouse. Also to the extent you're making a style-guide something that's enforced, it would be great if PYPA would provide a style-normalizer, but after checking into it a bit, hatch (still) isn't really there yet.

Look, it might seem like I'm arguing that the entire PYPA toolchain needs to be tightly coupled, but it's actually the opposite. I don't think changes in pip should be blocked on warehouse deployability or updates to twine. Before PEP-440, these things were loosely coupled. But what PEP-440 does is force tight-coupling on all tooling, or leave us with contradictions like builder-builds-what-installer-refuses. Sorry if that's a hassle, but this is the fall-out from responsibly enforcing your own mandates, and I think you guys asked for this?

IMHO, PEP-440 enforcement on public PyPi all by itself is not even close to hitting due-diligence requirements for the breaking changes you want to require in pip. Warnings that are easy to miss, or even errors which are easy to bypass by reaching for other supported tools in a complex and continuously churning kit also do not help to establish due-diligence. And not that I really expect this, but as a courtesy, it would be nice if the community had as long to deal with this sort of thing as PYPA gave it's own insiders. If the twine issue had been addressed ~4 years ago when public PyPi changed, then that would be ~4 years of fewer broken rollback targets that private registries were hosting.

pfmoore commented 2 years ago

Some background.

PEP 440 has been the standard for version numbers since 2014, and PEP 345 (metadata 1.2) was updated to make PEP 440 versions mandatory in commit 848fcd1b3 (2016). So, for six years packages with non-PEP 440 versions have been invalid, according to accepted standards.

The PyPA is not a unified body, it's a loose community of independent projects, each of which governs itself. There is no central authority, and in particular no-one who can insist that "all PyPA projects" implement agreed standards on any given timeframe. And given that most, if not all, tools in the packaging ecosystem are developed by wholly-volunteer teams, often of one or two people only, any such imposition would be unenforcable anyway.

You are right, different tools are enforcing the standards at different rates, and often incredibly slowly. Some of that is lack of people willing to do the work, and some of it is deliberately trying to avoid breaking projects that haven't yet adapted to the standards. But it's been six years now, and we've probably gone long enough on the basis of letting people know about the new standards and expecting them to change over time. Most people seem to have done so, and any hold-outs probably (in my personal opinion) aren't going to change just because it's the standard.

So we're now at a point where individual projects have to make the hard choice to start enforcing standards and dropping support for projects that don't follow those standards. This is messy, because as you've correctly noted, tools make these decisions independently and there's no co-ordination. So yes, pip will no longer install projects that don't use PEP 440, when setuptools will still build such projects, and twine will allow them to be uploaded to indexes that don't enforce PEP 440. That's messy, and not ideal for the end users.

But short of someone funding a major project to enforce standards consistently across all projects (the exact opposite of what you want!) a messy and piecemeal enforcement is the best we can manage with the resources we have.

The good news, though, is that it's easy to deal with - just conform to the standards. If you don't want to, tools will gradually stop supporting you. But that's the consequence of your choice to ignore the direction Python packaging has been going in (with community support!) for 6+ years now. You can pin your environments to older versions of tools, as they start to enforce PEP 440. No-one is forced to update anything. Public services like PyPI may no longer support you, requiring you to find new hosting where you can choose the policies, but that's just like any other public service.

Frankly, we'd have moved to enforcement much faster if we hadn't been trying extremely hard (and at significant cost to project maintainers) to give people time to move to new standards. It's pretty disappointing and discouraging to find that even though we've made those efforts, people still feel that it's OK to argue that we haven't done sufficient due dilligence, we didn't give the community the courtesy of sufficient lead time (is 6 years really not sufficient?), or that we're ignoring the real world.

I don't think there's anything more to say here. Your situation has been noted, but we're unlikely to change our plans because of it.

I'm not going to lock this conversation, as we do want to hear feedback from other people affected by this change, but it would be appreciated if you could avoid further diverting the discussion onto general questions about standards adoption - if you want to debate those, please have the conversation on a more appropriate forum like the packaging category on https://discuss.python.org/

potiuk commented 2 years ago

Since you asked @pfmoore - I only have one comment to the discussion. I follow this and many other discussions in pip and pypa rather closely over the last few years.

My 3 cents.

I hear over-and-over "we have too little number of people, all volunteers, we cannot deliver what you want. don't expect more from us", I personally know a number of people that would be willing to help and increase the capacity of the pip and pypa efforts (myself including). But they are put off by the attitude of mainters, to put it blantly,

I think good solution to that would be to be open to other people joining the team, opening up with new ideas and ways of thinking. I think there is very little effort from the pypa maintainers to create an open and welcoming environment to nurture and welcome "fresh blood" and increase project's capacity.

Even if there are smart people who have different opinions, making them feeling welcome with their ideas and letting them try things and mentor them in would be a great idea to increase the capacity. You can do it in the way that keeps project in check without impacting overall quality by putting some safeguards in place and making sure the right governance in place so that the new people know how they should approach proposals for changes and how they can engaged in true discussions and when they feel they are heard and when there is a true discussion rather than (similar to the comment above) statments basically saying "we have no time, don't expect more from us" rather than "let's work together and invite others to volunteer to help us to build stronger, sustainable team".

I recently gave a talk about "growing your contributors base" at the Apache Con in New Orleans and one of the points I had there for the maintainers was "know when you need to grow and deliberatly act in the direction to do it - if you do not do it, it will not happen all by itself". Acting in this direction allowed us - in Airlfow to continuously grow not only contiributors base, but also commiter's base without impacting the overal direction of the project (but with many ideas and modifications of the original vision that the "fresh people" brought with themselves. Which required in many cases depriioritising ego of some maintainers in favour of prioritising improvements for the project.

Please don't take it personally, I think you should simply be more empathetic for people who honestly and sincerely want to help, have capabilities and time to do so, by the attitude they see make them think "Meh, it's not worth fighting for it as I am continuously facing with dismissive, harsh and unfair treatment where my voice and feelings are not even seriously considered". This is the current approach I am taking at least after the past experiences. The pip is super important for us and I try to keep up with what's going on in the project, but I dread the idea of making some proposal that goes against current direction (or is not a prioriity for current maintainers) - simply because of attitude of the responses I am expecting.

I understand that you have too little number of maintainers and it is all-volunteered time. I really do. But I think you run yourselves in the corner and you should simply be more open to others who cold help you to grow the team - and deliberately act in this driection.

Those are my 3 cents. Knowing the past discussiosn - I think you might want to hide it as off-topic but before doing it - think a bit about it please. It's very much "on-topic" actually because I think it very much explains the root cause of all similar discussions you have (and there are far too many of those to ignore them IMHO).

pfmoore commented 2 years ago

I understand that you have too little number of maintainers and it is all-volunteered time. I really do. But I think you run yourselves in the corner and you should simply be more open to others who cold help you to grow the team - and deliberately act in this driection.

You may be right. But the problem is that we are not actually constrained on code contributions. We're constrained on people contributing in support of the project direction to discussions like the "enforcing PEP 440" one here, on people doing research to understand the impact of changes we're considering, and on people reviewing existing PRs (in particular, in terms of how well they fit with pip's overall direction - simple "this code does what it says" reviews are also useful, but not the bottleneck).

And outside of pip, we're constrained by the lack of community involvement in standards setting. The pip maintainers have stated, publicly and repeatedly, that we follow standards. So "why is pip no longer allowing non-PEP 440 versions?" shouldn't even be a discussion here. If people don't like the standards, and don't follow them, why not get more involved in the process and ensure that the standards work for you? Even a standard as old as PEP 440 can be changed if there's sufficient justification.

I don't know how to encourage people to contribute in these areas. When we disagree with proposals, that isn't "not being welcoming", it's just as much the case that saying "this should be added" without considering the wider picture, isn't "contributing"[^1].

[^1]: Obligatory Monty Python reference: "This isn't an argument, it's just contradiction." "No it isn't." "Yes it is!"

potiuk commented 2 years ago

When we disagree with proposals, that isn't "not being welcoming", it's just as much the case that saying "this should be added" without considering the wider picture, isn't "contributing"[1

I think there is a big difference in saying "No, This is a bad idea. Go and find all the discussions about it yourself." vs. "Yes, we see your point, and it would be great if you can help to hash it out, happy to help to guide you and help you to understand the reasoning.".

The first one is basically "get lost, I have no time for you", the second is "Yeah we want to help others to grow to becoming contributors and we want to help them in becoming ones".

For example above comment might look like "Yeah. It woud be fantastic if twine has the same checks, we have the same long term vision, and maybe you can help there? Here is how you can get started and we are happy to guiide you".

pradyunsg commented 2 years ago

On the point of "why pip" is that every other piece in the toolchain can basically treat version strings as opaque strings and still do their job. To be overly simplifying, twine is basically a well-crafted POST request. build is basically a glorified subprocess call. setuptools is a tarfile+zip-with-a-different-extension generator. None of those pieces involve trying to parse the version.

Even if we started by adding validation there, those tools erroring out first is gonna result in user pushback that "hey, pip allows it so why don't you!?" responses.

That said, what @potiuk said is appropriate: it's a case of lack of availability of people, and the tools are generally moving in this direction.

"Yes, we see your point, and it would be great if you can help to hash it out, happy to help to guide you and help you to understand the reasoning.".

I literally did this in my first long comment -- minus tell me about your use case -- so I'm feeling a bit ignored right now but I digress. :)

if my private pypi is not enforcing pep-440 compliance and the public one is, i think that raises the basic question: where are the official helm-charts (or similar) for self-hosting pypi? and: is pip intended to work only with public-pypi? do other tools like artifactory enforce pep-440 today, and if so have they done that for long enough that we think it's ok? another basic question.. do you guys actually have telemetry that suggests that most usage of pip is for public pypi, or did you just assume that? what percentage is ok to break?

OK, even if I believe that you're trying to be helpful and make a point, or just gather information, this whole comment is a question soup. That's off-putting since it's using questions to make a point implicitly, without committing to a specific answer, while allowing you to use the fact that you never stated your position to push back or ask further questions to anything someone responds to this with. Besides, there are way too many questions -- answering questions typically takes more energy than asking the question.

Using a primarily-question line in discussions is typically a symptom of bad-faith arguing, since it puts all the onus on the party being questioned, with the person asking questions having the ability to push back to any point being made, by asking another leading question.

Finally, I'd like to not get into an extended discussion on change management here. If you want to have that discussion @potiuk, please file a new issue.

pypa / pip

Deprecate call to `setup.py install` when building a wheel failed for source distributions without pyproject.toml #8368