search

GrahamDumpleton / wrapt

A Python module for decorators, wrappers and monkey patching.
BSD 2-Clause "Simplified" License
2.04k stars 230 forks source link

Missing Python 2.7 windows wheel on version 1.13.1 #189

Closed pythonoi closed 2 years ago

pythonoi commented 2 years ago

I noticed that there are Python 2.7 wheels built for mac and linux for this release, was wondering if it's possible for a Python 2.7 wheel for windows to be added to the PyPI download files for this release.

Thanks!

GrahamDumpleton commented 2 years ago

The build infrastructure used to build Python wheels from GitHub actions doesn't seem to be able to build Python 2.7 binary wheels on Windows. I don't remember the error, but it wasn't from memory anything obvious and something I could do anything about so there was no choice but to disable binary wheels for that combination.

How dependent are you on Python 2.7 on Windows?

Also be aware that Python 2.7 support will likely be dropped in wrapt 1.14.0 as someone has provided a PR to modernise various aspects of Python C API usage and this means finally dropping Python 2.7 support.

ddanilko-sonova commented 2 years ago

We suffered from this as well and could not build a wheel for 1.13.1 due to the following error: error: Microsoft Visual C++ 9.0 is required. Get it from http://aka.ms/vcpython27 (note that the link ends up in a 404 so it's kind'a dead end). Unfortunately, we had to pinpoint an older version of wrapt in our requirements file.

@GrahamDumpleton The biggest problem that I see is that you do not update the major version when doing a backwards-incompatible change but increase the minor version instead. This is causing problems because it's not what most people expect. We are not using wrapt directly but pull it as a dependency of deprecated, which specifies a dependency to wrapt like that:

    install_requires=['wrapt < 2, >= 1.10'],

which is what I would do as well if I were them (== give me the latest compatible version of wrapt). However, we get a backwards-incompatible version 'out of the blue'...

@GrahamDumpleton please consider following semantic versioning, which according to https://semver.org/ means:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes, MINOR version when you add functionality in a backwards compatible manner, and PATCH version when you make backwards compatible bug fixes.

Dropping support for any Python version or operating system is a backwards-incompatible change.

GrahamDumpleton commented 2 years ago

There were no incompatible API changes in the code of the package for 1.13.0. The Windows build issue for binary wheels when using Python 2.7 which is specific to whatever is being used by GitHub actions is only issue I know of. What are you suggesting was an incompatible API change in the package for 1.13.0? For dropping Python 2.7 support, then is fair enough that perhaps should be called 2.0.0, but those changes haven't made it in yet so no decision made on the version yet.

ddanilko-sonova commented 2 years ago

I agree that dropping support for Python 2.7 should be a major release (e.g. 2.0.0).

When it comes to backwards-incompatible changes in 1.13.0, I could spot a few:

Would you agree that those two changes could have been backwards-incompatible for some people?

ddanilko-sonova commented 2 years ago

When it comes to the problem at hand (1.13.0 not being backwards-compatible with Windows+Python2.7), I think that the cleanest solution would be to remove 1.13.0 and 1.13.1 from PyPI and re-release them as 2.0.0. Otherwise, many users of your package would need to adapt their code to not update to 1.13.0 in case they still have systems using Windows+Python2.7.

Do you think that it would be possible to do so?

GrahamDumpleton commented 2 years ago

There is no backwards incompatibility with Windows+Python2.7. The only issue related to that is the specific system used to build wheels in which their environment is broken. The code is tested on other platforms for Python 2.7 and there is no platform specific code in the package so it should still work fine on Windows+Python2.7. So that isn't really going to be an issue. The only issue may be with Python 3.4 where an install may not work.

Anyway, I am going to discuss it with one of the CPython core committers who has been helping with some stuff in wrapt recently. They will have a better idea of what is reasonable given that Python 3.4 is long dead and no longer supported since March 18, 2019.

tiran commented 2 years ago

Graham has reached out to me and asked me to give my opinion on the issue.

Warning: strong opinion ahead

In my opinion it is unreasonable to request build artifacts for Python versions that have passed their end of lifetime. We, the CPython core developer team, have stopped supporting Python 2 not only to reduce our workload, but also spare 3rd party packages like wrapt the burden. If you cannot move away from Python 2, then I suggest that you either build Windows artifacts yourself, pay somebody to build wheels for you, or at least contribute a working solution for Windows builds.

SemVer is not a magic fix for this issue, too. Hynek has summarized the issues with SemVer in his article Semantic Versioning Will Not Save You.

May I suggest a more practical solution?

Users of supported Python versions will get latest version while users of unsupported Python versions like 2.7 and 3.4 will get unsupported wrapt 1.12.1. IMHO that is reasonable and fair. Python 3.6 is the oldest Python version supported by cibuildwheel and also the oldest Python version that can targeted by limited API / abi3 (PR #187) with reasonable effort.

ddanilko-sonova commented 2 years ago

May I suggest a more practical solution?

Yank release 1.13.1 from PyPI. Drop support for all EOLed Python versions (3.5 and earlier) by adding python_requires = >= 3.6. Release 1.14.0 with binary wheels for Python 3.6 to 3.10 as well as PyPy 3.

Lovely ❤️ That's exactly what I would love to see with twoone small adjustments:

EDIT: I just realized that if the support for Python 2.7 is dropped explicitly then pip will not install 1.14.0 in a Python 2.7 environment so after reconsidering, I agree that 1.14.0 is a good number 😉

By no means I wanted to ask for keeping active support for deprecated Python 2.7 - I would simply appreciate not braking existing systems by not mentioning dropping support for them and by mentioning I mean bumping the major release version 😉

tiran commented 2 years ago

I don't see any reason to bump the major version number. wrapt does neither use SemVer nor does 1.14.0 introduce a backwards incompatible API change.

ddanilko-sonova commented 2 years ago

If wrapt does not plan to adhere to semantic versioning now nor in the future, then I guess there is nothing more to do here... Re-releasing 1.13.0 as 1.14.0 does make any sense and does not change anything for its users... 😞

Thanks for your time and hard work in developing open source projects 🍻

pythonoi commented 2 years ago

If it's hard to build for windows-27 and this project is going to be Python 3 only in the near future, I agree with @tiran proposed solution. This will allow the project to move onto Python 3 only 😄 and users who are dependent on py2 will not get updates or installation failures. For now, I'll just pin on py2 as others who have referenced this issue have done and you can feel free to close this issue with whatever approach you decide.

Thanks for your time and responses!

andy-maier commented 2 years ago

Many of our projects that test on Github Actions have also suffered from the error mentioned further up:

error: Microsoft Visual C++ 9.0 is required. Get it from http://aka.ms/vcpython27

The workaround for us was to pin wrapt to <1.13. on Python 2.7 and Python 3.4. It is not pinned on Python 3.5 and higher.

This time, I did not want to hold back with my opinion on dropping Python 2.7 and 3.4 support:

As long as major Linux distributors still support Python 2.7 and 3.5 on their behalf (RedHat is one such example), dropping that support makes life quite difficult for maintainers of Python packages that are included in such Linux distributions.

The only possible solution in such cases seems to be to pin dependent packages, because the maintainers of packages who dropped support for old Python versions often refuse to fix old versions of their packages. So the effort to deal with such decisions is nearly always on the side of the owners of packages that depend on the packages that have dropped the support. In other words, a package that drops support for old Python versions has made their own life easier at the cost of dependent packages that cannot afford to also do that. For example, I am spending my sunday morning today for pinning wrapt to <1.13 in seven Python projects that use Pylint, and in most cases in both the master branch and the latest stable branch of them. And those are just the most important ones, I will have to do that over time in all the others, too.

One consequence of dropping support for older Python versions is that this leads to increasingly not being able to fix functional defects and also security issues, because the dependent packages just stay pinned to the latest version that still supports the Python version in question.

As much as I personally would like to move forward with Python versions and drop old stuff, the reality out there is different and I see Python package maintainers being increasingly disconnected with that part of reality. If we as package maintainers try to force users forward by dropping support for old versions, but Linux distributions are not with us on this path, then there is a significant disconnect that ultimately goes at the expense of our users.

GrahamDumpleton commented 2 years ago

So it seems I may not have been understanding what the issue is here and I have not been provided a suitable explanation of what the actual problem is and how I can address it. Instead I have been getting blasted about not following some ideal around semantic versioning even though the code itself didn't have any changes which would warrant a major version change, and that this is purely an issue with the Python package build system. So the criticism is not helpful in trying to fix the underlying problem, which would still be the preferred option.

The wrapt package code still supports Python 2.7. Support for Python 2.7 has not been dropped.

It was my understanding that the inability to test for Python 2.7 on Windows was very specific to the GitHub actions build infrastructure. No one has said that it is a more general problem with all Python 2.7 installations on Windows.

Thus my expectation has been that people developing on Windows were having no issues with building wrapt on their own systems. What I didn't contemplate was that people were depending on GitHub actions for building their applications as part of their workflow, instead I was only considering that it was effecting just my ability to test it when using GitHub actions.

So this is what I need.

If no one is prepared to help work out what the issue is then nothing is going to be changed because I lack the knowledge to solve any issues with Windows and also have very limited time or desire to work on wrapt these days. It has taken me well other a year to get this last release out and ignoring the obsolete future module on PyPi the wrapt module was the last out of the top 100 packages to get Python wheels, so you can see that wrapt isn't exactly high on my priorities any more.

Although a PR has been submitted by @tiran for dropping Python 2.7 I acknowledge that there are systems out there still using it, so I haven't wanted to drop support for it just yet.

So how about some actual help here please in trying to understand the source of the problem.

tiran commented 2 years ago

You cannot expect that a package maintainer supports Python versions beyond their end of lifetime. Each Python release has upstream support for 5 years, 2.7's life cycle was even extended to 10 years. After EOL you are on your own.

GrahamDumpleton commented 2 years ago

One extra comment I might make is that this problem with GitHub actions could well have existed for a very long time but was hidden. That it was a problem was masked by the fact that if wrapt failed to be built due to a lack of a compiler before 1.13, it would fall back and install a pure Python version. I suspect that the people who helped out update wrapt so that Python wheels could be built may have disabled that fallback and so not having a compiler may now be fatal. So if someone understands the new Python build system that is now used and can work out how to set the environment variable WRAPT_INSTALL_EXTENSIONS=false just for Python 2.7 on Windows that may well solve the problem.

GrahamDumpleton commented 2 years ago

The specific check around whether extensions should be installed is at:

and consists of:

# # --- Detect if extensions should be disabled ------------------------------

wrapt_env = os.environ.get('WRAPT_INSTALL_EXTENSIONS')
if wrapt_env is None:
    wrapt_env = os.environ.get('WRAPT_EXTENSIONS')
if wrapt_env is not None:
    disable_extensions = wrapt_env.lower() == 'false'
    force_extensions = wrapt_env.lower() == 'true'
else:
    disable_extensions = False
    force_extensions = False
if platform.python_implementation() != "CPython":
    disable_extensions = True

So maybe can just hard wire it in there such that if is Windows and Python 2.7, set disable_extensions = True. This of course will have a slight performance impact, but at least it will still build for Python 2.7 on Windows.

tiran commented 2 years ago

Graham, you could also follow my suggestion to release 1.14.0 for >= 3.6 and yank 1.13.1. Users on 2.7 and 3.4 will then get 1.12.1 automatically. The approach is the least amount of work for you. Only download is users of Python 2.7 won't get the bugfixes in 1.13.1, but IMHO that's ok.

GrahamDumpleton commented 2 years ago

If disabling of the compilation of the extensions on Windows when Python 2.7 is used is hardwired, then GitHub actions at:

can be updated to build again for Python 2.7 on Windows.

At the same time, creation of binary wheels will need to be disabled on Windows when it is Python 2.7 and that I don't know how to do with this new system.

GrahamDumpleton commented 2 years ago

@tiran Was trying to avoid that since expect that this will be the last wrapt version released for quite a while again, so would have been nice to have the latest fixes still available for Python 2.7 even if they are minor.

andy-maier commented 2 years ago

On my comment further up: My rant on dropping support for old Python versions was more general and was mainly triggered by the stated intention to do that also for wrapt in the future. Sorry for posting this here, but I saw this so many times now in various of our dependent packages and every time it triggers the need for a "fix" on our end when it saved some time on the package maintainer's end, and I just had to say something about it.

The exact issue I saw with wrapt 1.13.0 and 1.13.1, was only this one:

I relaxed my earlier fix to pin wrapt to <1.13 just for Python 2.7 on WIndows.

tiran commented 2 years ago

CIBW_SKIP: pp* *win32 *win_amd64

GrahamDumpleton commented 2 years ago

Or is it better to be more specific and use:

CIBW_SKIP: pp* cp27-win*

Don't want to block it for all versions on Windows.

GrahamDumpleton commented 2 years ago

@andy-maier Am aware of what is in the logs from the builds, but not why GitHub actions environments fail and complain there is no compiler. I am presuming there is a compiler installed else how would all other Python packages which have extensions be built, or is it a global problem and not just related to wrapt. This is what now one has explained yet.

GrahamDumpleton commented 2 years ago

And @andy-maier, sorry. My frustration isn't targeted specifically at you. You just came along late to the party after all the other discussion. So bad timing. :-)

tiran commented 2 years ago

Python 2.7 needs a specific, ancient version of MSVC. In the past every major MSVC version had its own libc (msvcrt). You could not safely compile a Python extension with a newer version. GitHub may no longer offer the ancient version from 2008.

GrahamDumpleton commented 2 years ago

One last question for anyone, in:

where has:

[gh-actions]
python =
    2.7: py27, py27-without-extensions, py27-install-extensions, py27-disable-extensions
    3.5: py35, py35-without-extensions, py35-install-extensions, py35-disable-extensions
    3.6: py36, py36-without-extensions, py36-install-extensions, py36-disable-extensions
    3.7: py37, py37-without-extensions, py37-install-extensions, py37-disable-extensions
    3.8: py38, py38-without-extensions, py38-install-extensions, py38-disable-extensions
    3.9: py39, py39-without-extensions, py39-install-extensions, py39-disable-extensions
    3.10: py310, py310-without-extensions, py310-install-extensions, py310-disable-extensions
    pypy-2.7: pypy-without-extensions
    pypy-3.6: pypy-without-extensions
    pypy-3.7: pypy-without-extensions

is there a way to qualify this so that if Python 2.7 on Windows only do:

2.7: py27-without-extensions

I am not actually sure whether the pyXY entries are needed anymore since there may not even be a fallback now to use pure Python, which is what those tests checked for.

But then I don't understand:

which is:

extensions = [
    setuptools.Extension(
        "wrapt._wrappers",
        sources=["src/wrapt/_wrappers.c"],
        optional=not force_extensions,
    )
]

In other words, the optional flag. Isn't the point of that to not fail if the extensions fails to compile, or is lack of compiler not included in that, which is a bit silly.

GrahamDumpleton commented 2 years ago

Anyway, will give some thought about this overnight and see in the morning if anyone has got a solution other than hardwiring Python 2.7 on Windows to not use extensions.

andy-maier commented 2 years ago

I did a test run on "windows-latest" (="windows-2019") on GitHub Actions that lists some of the installation directories, and here are the Visual Studio versions it has installed:

10/03/2021  07:14 PM    <DIR>          Microsoft Visual Studio
10/03/2021  08:51 PM    <DIR>          Microsoft Visual Studio 10.0
10/03/2021  08:51 PM    <DIR>          Microsoft Visual Studio 11.0
10/03/2021  08:51 PM    <DIR>          Microsoft Visual Studio 12.0
10/03/2021  08:24 PM    <DIR>          Microsoft Visual Studio 14.0
10/03/2021  08:08 PM    <DIR>          Microsoft Visual Studio Tools for Unitys

Update: Using "windows-2016" on GitHub Actions shows the same MSVS versions available.

That explains why it complains about not finding MSVS 9.0, but it is not clear (to me) why on Windows with Python 2.7, wrapt 1.12.1 installs fine but wrapt 1.13.0 and 1.13.1 do not:

GrahamDumpleton commented 2 years ago

Please test with:

pip install wrapt==1.13.2rc1

because it is a rc you will need to be explicit as will not be picked up if not pinned.

As to the reason why this used to work with <1.13.0, it is because it used to use a non standard way to work out whether the extension could not be compiled.

For Windows this would detect the specific exception type that indicated that the compiler was missing, and not just that the compilation of the extension failed.

When the package installation was moved to setuptools the official Python way of ignoring the fact that the C extension failed to compile and falling back to pure Python was instead used. This newer mechanism in setuptools seems though to regard a lack of a compiler on Windows as a fatal error and doesn't seem to actually regard that as something that means it should fallback to pure Python implementation. Anyway, that is the best I could work out. That is, it is an issue with how setuptools works. This was never an issue with wrapt or any attempt to drop support for Python 2.7 on Windows.

A notice on what was changed can be found in:

In short, compiling the C extension is disabled by default on Windows for Python 2.7. If you have control over the build system and you know you have a working compiler installation for that Python version, you can set the WRAPT_INSTALL_EXTENSIONS=true environment variable and should then hopefully be able to install with the C extension.

GrahamDumpleton commented 2 years ago

Has anyone tried this yet and confirmed it works? Add a comment rather than reaction so can know for sure.

MHendricks commented 2 years ago

1.13.2rc1 installed without issue. The wrapt directory doesn't have _wrappers.pyd, which from above I believe is correct.

C:\Users\mikeh>pip install wrapt==1.13.2rc1 --no-cache-dir
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Looking in indexes: https://pip.blur.com/blur/dev/+simple/
Collecting wrapt==1.13.2rc1
  Downloading https://pip.blur.com/root/pypi/%2Bf/371/e3ce501cf04f1/wrapt-1.13.2rc1.tar.gz (48kB)
     |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU| 51kB 4.7MB/s
Building wheels for collected packages: wrapt
  Building wheel for wrapt (setup.py) ... done
  Created wheel for wrapt: filename=wrapt-1.13.2rc1-cp27-none-any.whl size=20330 sha256=11cbc54667457fa99317097ef87ebd949f3e9241df33b118f89c4a2e0c93d975
  Stored in directory: c:\users\mikeh\appdata\local\temp\pip-ephem-wheel-cache-ba6k7z\wheels\62\be\f4\e3d517e677d13784301206e59cd2b1e0e28577a264e52ce76f
Successfully built wrapt
Installing collected packages: wrapt
Successfully installed wrapt-1.13.2rc1
WARNING: You are using pip version 19.3.1; however, version 20.3.4 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

Setting WRAPT_INSTALL_EXTENSIONS=true causes the pip install to fail as this computer doesn't have compile support.

C:\Users\mikeh>set WRAPT_INSTALL_EXTENSIONS=true

C:\Users\mikeh>pip install wrapt==1.13.2rc1 --no-cache-dir
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Looking in indexes: https://pip.blur.com/blur/dev/+simple/
Collecting wrapt==1.13.2rc1
  Downloading https://pip.blur.com/root/pypi/%2Bf/371/e3ce501cf04f1/wrapt-1.13.2rc1.tar.gz (48kB)
     |UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU| 51kB 3.2MB/s
Building wheels for collected packages: wrapt
  Building wheel for wrapt (setup.py) ... error
  ERROR: Failed building wheel for wrapt
  Running setup.py clean for wrapt
Failed to build wrapt
Installing collected packages: wrapt
    Running setup.py install for wrapt ... error
ERROR: Command errored out with exit status 1: 'c:\python27\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'c:\\users\\mikeh\\appdata\\local\\temp\\pip-install-5ar4qv\\wrapt\\setup.py'"'"'; __file__='"'"'c:\\users\\mikeh\\appdata\\local\\temp\\pip-install-5ar4qv\\wrapt\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'c:\users\mikeh\appdata\local\temp\pip-record-yste3j\install-record.txt' --single-version-externally-managed --compile Check the logs for full command output.
WARNING: You are using pip version 19.3.1; however, version 20.3.4 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

C:\Users\mikeh>python --version
Python 2.7.18

Thank you for fixing this. I'll be glad when I can finally move our studio to python 3 now that Maya and the other software we use finally have support for it. Now I have to get back to updating all of our code 😅.

GrahamDumpleton commented 2 years ago

Version 1.13.2 has been released with these changes. I will leave it as is for now, but will yank 1.13.1 after I see no issues reported with this update.

ddanilko-sonova commented 2 years ago

Instead I have been getting blasted about not following some ideal around semantic versioning even though the code itself didn't have any changes which would warrant a major version change, and that this is purely an issue with the Python package build system. So the criticism is not helpful in trying to fix the underlying problem, which would still be the preferred option.

My frustration isn't targeted specifically at you. You just came along late to the party after all the other discussion.

I am sorry if I was the source of your frustration. I really didn't mean to - I was just looking for a pragmatic, low-effort solution to a problem that I was, as you put it, blasted with in my environment. I am clearly not as experienced as you guys and clearly the solution that I had proposed was not addressing the root of the problem - only the symptoms. I am very keen to see that you figured out the source of the problem and was able to provide a wheel for 1.13.2 version of wrapt for Pytnon2.7 on Windows. Once again

Thanks for your time and hard work in developing open source projects 🍻

I really meant it both then and now 🙂