search

matplotlib / basemap

Plot on map projections (with coastlines and political boundaries) using matplotlib
MIT License
772 stars 390 forks source link

New style namespace packages #576

Open ksunden opened 1 year ago

ksunden commented 1 year ago

This is a follow up to matplotlib/matplotlib#25381.

Newer setuptools is going to completely deprecate old-style namespace packages, it seems. The newer style is to simply not have a __init__.py and has a few small changes to the setup.py.

This conforms with PEP 420

What I have not done yet, but am willing to do (at least for the packaging/ci side of things) is up the minimum supported python version. (partly wanted to see which CI would actually fail, if any) Since PEP 420 is py 3.3+, that is probably the lowest version that should be continued, however all of python <=3.6 is fully end of life (and even 3.7 is end of life at the end of this month) SPEC 0, which is adopted by much of the scientific python ecosystem, including numpy and matplotlib, would put the python version at 3.9 currently.

With this change, the following files are removed from the wheels:

basemap-1.3.7+dev-cp311-cp311-linux_x86_64.whl

basemap_data-1.3.2-py2.py3-none-any.whl

basemap_data_hires-1.3.2-py2.py3-none-any.whl

Note that setuptools was already removing the ns package __init__.py files from the wheels.

And the following are removed from the sdist:

basemap-1.3.7+dev.tar.gz

basemap_data-1.3.2.tar.gz

basemap_data_hires-1.3.2.tar.gz

None of these changes are super surprising, and is just a consequence of PEP 420 that these are not needed, and namespace packages are implicit.

molinav commented 1 year ago

Hi @ksunden! Thanks for the feedback from the current matplotlib development. Yes, at some point basemap needs to align with the rest of matplotlib, and it is really time to start saying goodbye to some of the EOL Python versions.

Since this is something really breaking, I would like to add this to a new major version of basemap (say 2.0.0). I still have a minor version in mind (1.4.0) with small updates that I still would like to keep in EOL versions (i.e. infamous 2.7).

Is it correct that the namespace convention is targeted to matplotlib 3.8.0? I can make basemap align with this matplotlib release (currently basemap pins to matplotlib < 3.8, so basemap is still linking to a matplotlib with the old namespace convention).

Whenever the time of setting a higher python_requires comes, I would be in favour to start at least with Python 3.5. Keeping support for Python 3.3 and Python 3.4 makes no sense to me, because on Windows I would need Visual Studio 10, which is basically very difficult to get (it is actually easier to compile Python 2.7 extensions for Windows). Starting with Python 3.5, extensions are built on Windows with Visual Studio 2015, which is easy to get. Anyway, as of 2023 users should be using at least Python 3.8, so setting python_requires to >=3.8 or >=3.9 is totally reasonable.

I would like to mention that CentOS 7 EOL is planned for 2024-06-30, and there the default system Python versions are 2.7 and (through EPEL) 3.6.

PS: some of the failed jobs in the PR are not related to your changes, but to some update in the GitHub runner image around May 2023, which now does not allow to fetch linux/386 Docker images from a linux/amd64 host. I am currently taking a look to it.

ksunden commented 1 year ago

As far as I can tell, there actually is no incompatibility with mixing older versions of mpl and newer versions of basemap (or vice versa) introduced here (outside of incompatibilities with python <3.3, which would be quite an old version of matplotlib at this point).

Setuptools has for quite a while already been removing the __init__.py files from wheels/installations that go via wheels, which includes sdist installs on python versions which support it. So the remaining change is not actually affecting runtime really at all, and is just about updating the metadata for the implicit namespace package (and this is the bit which precludes python <3.3, actually).

So yes, the change takes effect in mpl 3.8, but it is more of a "make the packaging line up with what is largely already happening" than anything, so it really shouldn't be a user facing change at all. When I was investigating the change on mpl, I installed basemap (modified only so that the mpl<3.8 pin was not applied) without the change from this PR worked with the modified mpl, and it did. (and while I did not actually test the inverse, there is no actual directionality to namespace packages, so fairly confident it would work as expected)

Looking at https://pypistats.org/packages/basemap:

python3 <3.4 are not included in the stats at all, probably because there are no wheels for them

python 3.4 is almost literally 0 downloads (a handful of days have a single download, but those are quite few and far between, and represent still ~0% of downloads)

python 3.5 participated in a fleury of activity that 2.7 also saw in January/february, but both before and after that was quite similar to 3.4 in that there are effectively zero downloads at all.

python 3.6 has a bit of a baseline of downloads at ~10/day (overall is ~1000/day to put it in prespective) so that is still ~1-2 %

python 3.7 had a giant spike in early may that is visible in the overall download numbers, where it represented 80% of downloads for a week or so, but outside of that is usually ~10%

python 3.8 doesn't have the spike, but is otherwise commensurate to 3.7

(I don't think anybody is suggesting getting rid of 3.9+ quite yet, so not going to analyze those, but suffice to say 3.10 is the most at ~40%, and 3.9 is second at ~30%, 3.11 is still quite low at ~5%)

Python 2.7 had a fleury of activity in january to february (aligning perfectly with the 3.5 activity), but outside of that is still ~1-2%, quite similar to 3.6)

Ultimately, I will go with whatever cutoff/timeline/release you choose, @molinav, but my recommendation based on these numbers would be to either require 3.7+ or follow SPEC0 and do 3.9 plus (the latter is what most of the core scientific python packages, including matplotlib, do)

And of course users on older python are not completely unable to use basemap, just not the newest versions, which at a certain point they will have to upgrade their stack to get things to work and be updated.

As for the RHEL/CentOS defaults, I think the overlap of people who:

Is quite a small subset, one that I would assume is zero people until proven otherwise, and thus not weigh it highly in making maintenance decisions. (Of course if people speak up for that, that should be weighed)

lima-limon-inc commented 1 month ago

Hi there! I've been trying to package basemap for Gentoo and I've been having problems regarding the namespaces. It would seem like the fix provided by @ksunden would fix the issue, however it is not up to date with the latest commit.

Are there any plans to update to the newer PEP standard?

Thankd in advance!

PS: I am by no means a Python/Gentoo expert. I apologize in advance for any mistakes and/or misconceptions.

ksunden commented 1 month ago

I have rebased and added changes to CI to not test on python < 3.5.

I have not changed the packaging or adjusted the version number or anything, as those are decisions I leave to you, @molinav.

lima-limon-inc commented 1 month ago

I have rebased and added changes to CI to not test on python < 3.5.

So these patches should work for the 1.4.1 release, correct?

Thanks in advance

lima-limon-inc commented 1 month ago

@ksunden Thank you very much for the patches! The Gentoo ebuild seems to be working correctly!

Have a good week