search

PythonOT / POT

POT : Python Optimal Transport
https://PythonOT.github.io/
MIT License
2.39k stars 497 forks source link

[0.7.0] Provide wheels for Python 3.9+ #344

Closed davidghiurco closed 2 years ago

davidghiurco commented 2 years ago

Describe the bug

Python 3.9.x+ users are unable to install POT==0.7.0 from a wheel because https://pypi.org/project/POT/0.7.0/#files only has wheels built for Python 3.5, 3.6, 3.7, and 3.8.

Thus we have to rely on a source distribution installation.

This, in combination with the fact that version 0.7.0 doesn't have a pyproject.toml (which was added in the 0.8.0 release, see: https://github.com/PythonOT/POT/pull/293) makes source builds extremely annoying on newer versions of Python. Because of the missing pyproject.toml, I have to have a pip install numpy cython in like 8 different places throughout my build scripts if I want to upgrade my Python version to 3.9.x. Not only is this bad practice, but it also introduces sensibilities related to numpy binary compatibility (at least in my case).

Naturally, upgrading to POT==0.8.0 would solve this problem, but the upgrade introduces changes in mathematical algorithms which (at least for several of my projects) break sensible models. Thus we are stuck with POT==0.7.0.

To Reproduce

Steps to reproduce the behavior:

  1. Install Python >= 3.9.0
  2. Run pip install POT==0.7.0

Screenshots

Code sample

Expected behavior

pip install POT==0.7.0 should work on Python 3.9.x+ without extra commands being necessary apriori.

Environment (please complete the following information):

Output of the following code snippet:

>>> import platform; print(platform.platform())
macOS-10.16-x86_64-i386-64bit
>>> import sys; print("Python", sys.version)
Python 3.9.7 (default, Sep 16 2021, 08:50:36) 
[Clang 10.0.0 ]
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.21.4
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 1.7.3
>>> import ot; print("POT", ot.__version__)
POT 0.7.0

Additional context

This problem can be solved in one of 2 ways (or both):

1) Upload POT==0.7.0 wheels for Python 3.9+ to PyPI

2) Backport https://github.com/PythonOT/POT/pull/293 to 0.7.0

rflamary commented 2 years ago

POT 0.7 was released in May 2020 several month before Python 3.9 was released and several month before PEP 621 (for pyproject.toml) was created. So while I am very happy 0.7.0 still works on modern python and acknowledge that the installation was/is a bit of a pain we are unable to guarantee backward compatibility of older versions of POT with future versions of Python.

However we aim at avoiding regressions in the newer versions, and you mention potential ones: ‘the upgrade introduces changes in mathematical algorithms which (at least for several of my projects) break sensible models.‘. Can you please elaborate on these problems by opening issues with reproducible examples? We really wish to provide easy and seamless updates for new versions of POT.

What you propose is actually very difficult and might have sides effects on other users :

  1. wheels have a fixed ABI decided by numpy that has changed in 1.20 so there is no way to ensure that those wheels will work on the dependency (depending on the version of numpy) of 0.7.0 (this was fixed in 0.8.1 forcing a more modern version of numpy)
  2. We do not have the bandwidth for maintaining and updating several stable versions and ensuring that it build and works well on newer python. We cannot change the source distribution of pypi for 0.7.0 since it can break other users configuration.
davidghiurco commented 2 years ago

We cannot change the source distribution of pypi for 0.7.0 since it can break other users configuration.

You have a lot more experience developing and maintaining packages than I do, but how would adding a pyproject.toml to the 0.7.0 source distribution break anyone or anything? The source distribution is compiled at installation time, against whatever numpy version the user has installed on their system. It probably isn't the same numpy version that POT==0.7.0 .whl was built against anyway.

So you wouldn't be forcing any numpy ABI breaking updates by changing the source distribution, and a pyproject.toml is fairly innocuous to the rest of the build. I'd be curious to hear what your objections are. I am aware that POT==0.8.x+ bumped the numpy version due to ABI changes, but the numpy version in pyproject.toml of POT=0.7.0 doesn't have to be the same as in the 0.8.x release(s).

Can you please elaborate on these problems by opening issues with reproducible examples? We really wish to provide easy and seamless updates for new versions of POT.

I have to do a lot of regression testing, so if I find conclusive evidence that POT==0.7.0 -> POT==0.8.x upgrade, all other things held equal, indeed is the culprit, I will do my best to inform you. I will open a separate ticket if that's the case.

rflamary commented 2 years ago

Hello we cannot change the source distribution for POT 0.7.0 since it will have some side effects.

On the other hand, I am ok with releasing on pipy a new (old) version 0.7.1 or 0.7.0post1 with the only change being the version number and the pyproject.toml file. Note that there will be no testing and this version is considered deprecated and won't be updated. Is that OK with you?

davidghiurco commented 2 years ago

That would be extremely helpful. I'm already using POT 0.7.0 on Python 3.9 which isn't tested and supported, but is working quite well for me.

I am personally a fan of the .post1 suffix since it denotes that it contains the same source code as the original 0.7.0 release, but with a packaging fix.

rflamary commented 2 years ago

Done,

https://pypi.org/project/POT/0.7.0.post1/

Please check that it works for you and close the Issue.

davidghiurco commented 2 years ago

Thanks, I can confirm the source installation of 0.7.0.post1 now correctly pulls in build-time dependencies numpy and cython, thus eliminating the annoying workarounds of needing to have those dependencies already installed before-hand.

However, this source installation introduces a new problem, that of numpy's binary forward compatibility but lack of binary backward compatibility. Meaning, compiling against numpy>=1.16 will actually compile against the newest possible numpy (due to the open-ended restraint), thus breaking binary compatibility when used on a platform with an older numpy version.

I've opened a separate issue for this: https://github.com/PythonOT/POT/issues/346