search

GenericMappingTools / pygmt

A Python interface for the Generic Mapping Tools.
https://www.pygmt.org
BSD 3-Clause "New" or "Revised" License
758 stars 220 forks source link

Adopt NEP29 - drop support for Python 3.6 and NumPy 1.16 #690

Closed weiji14 closed 3 years ago

weiji14 commented 3 years ago

Originally posted at https://github.com/GenericMappingTools/pygmt/issues/644#issue-716952700, but updated slightly

Just bringing up the topic of whether we should follow NEP 29 — Recommend Python and Numpy version support as a community policy standard as noted before in #340 (comment). This would mean dropping support for Python 3.6.

In practice, we'll need to follow in the footsteps of our dependent packages:

Some options we can take are:

Does it mean that users cannot install pygmt via pip install pygmt or conda install pygmt if they're still using Python 3.6? Or they can still install it, but there is no guarantee that all PyGMT functionalities work well with Python 3.6?

It will be quite hard to support four minor versions of Python (3.6, 3.7, 3.8, 3.9), especially for the conda packages where there is a separate build for each one (unless we can work out how to get a 'noarch' build). We could still allow pip install pygmt Python 3.6 (with no support guarantees, i.e. no CI checks), but it will become a maintenance burden when our dependency packages (numpy, pandas, xarray) move on.

There's no rush to do all this, since Python 3.9 is just released, and there are some libraries still to catch up. Even matplotlib (who has more maintainers) is dropping Python 3.6 for their v3.4 (see https://github.com/matplotlib/matplotlib/pull/17662) so it's worth thinking about following NEP 29 for PyGMT v0.3.0.

Originally posted by @weiji14 in https://github.com/GenericMappingTools/pygmt/issues/644#issuecomment-705376657

Are you willing to help implement and maintain this feature? Yes

seisman commented 3 years ago

As per the comment in https://github.com/GenericMappingTools/pygmt/issues/665#issuecomment-728748849, we should strictly follow semantic versioning. As we definitely will add new features in the next few weeks/months, the next release will be v0.3.0, and we can drop the Python 3.6 support now.

In v0.4.0, we will bump the minimum required GMT version to 6.2.0.

leouieda commented 3 years ago

@seisman that depends also on how much work it is to maintain 3.6 support. For 3.5, it was a bit annoying since a lot of pathlib didn't work properly and there were no f-strings. But for 3.6 there isn't much that's like that. So if it's just the extra CI, it might not be too bad. That said, I have no problem with being more aggressive 😡

seisman commented 3 years ago

So if it's just the extra CI, it might not be too bad

We recently had a sudden failure for Windows + Python 3.6 (see https://github.com/GenericMappingTools/pygmt/issues/693 and some debugging efforts in #694). I guess something wrong with the conda-forge's Windows builds, but it only happens to Python 3.6. It would be difficult to track down the cause. That's one of the reasons I'd like to drop Python 3.6 support soon.

weiji14 commented 3 years ago

A bit of a tangent, but is there a chance we could setup PyGMT to be no-arch (https://www.anaconda.com/blog/condas-new-noarch-packages) like xarray at https://anaconda.org/conda-forge/xarray/files? That would mean that:

  1. We don't have to care about what OS or Python version people are using (could be 3.6, 3.9, even the unreleased 3.10 perhaps)
  2. The act of adopting NEP29 becomes pretty irrelevant (from PyGMT's point of view), as people would just be forced to update to a newer Python/NumPy version when the time comes.
seisman commented 3 years ago

is there a chance we could setup PyGMT to be no-arch

It looks promising.

The act of adopting NEP29 becomes pretty irrelevant (from PyGMT's point of view), as people would just be forced to update to a newer Python/NumPy version when the time comes.

Dropping Python 3.6 (or any other old Python versions) means that we don't have to run CI jobs for old Python versions, and we can use some new Python features which are only available in new Python versions. So we still need to adopt NEP29 or less aggressive.

weiji14 commented 3 years ago

is there a chance we could setup PyGMT to be no-arch

It looks promising.

Ok, PR for noarch Python conda packages is at https://github.com/conda-forge/pygmt-feedstock/pull/13 (generated by conda-forge bot), it looks pretty slick!

leouieda commented 3 years ago

We recently had a sudden failure for Windows + Python 3.6

That's reason enough for me 🙂

And +1 to the noarch thing. Makes the conda-forge builds a lot faster.

weiji14 commented 3 years ago

Ok, the noarch conda-forge PyGMT v0.2.1 package is up, and this should work on Python 3.6-3.9 (theoretically on 3.10 too but I haven't tested).

I think we can drop Python 3.6 from our CI (the less aggressive option) now, and decide whether or not we should adopt NEP29 closer to PyGMT v0.3.0's release in a month or two (the aggressive option, i.e. Python 3.6 would just not work at all).

seisman commented 3 years ago

I think we can drop Python 3.6 from our CI (the less aggressive option) now,

I agree!

the aggressive option, i.e. Python 3.6 would just not work at all

How does it work? If we bump the minimum required Python version to 3.7 in setup.py and the conda-forge recipe. What will happen if a Python 3.6 user installs pygmt?

seisman commented 3 years ago

How does it work? If we bump the minimum required Python version to 3.7 in setup.py and the conda-forge recipe. What will happen if a Python 3.6 user installs pygmt?

I tried to install pygmt under Python 3.5:

conda create --name pygmt35 python=3.5
conda activate pygmt35
pip install pygmt

The command installs pygmt v0.2.1 successfully (with some warnings/errors with netcdf4). But when I import pygmt, it gives the following errors, as expected:

>>> import pygmt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/seisman/.anaconda/envs/pygmtold/lib/python3.5/site-packages/pygmt/__init__.py", line 123
    print(f"  version: {__version__}")
                                    ^
SyntaxError: invalid syntax

As for conda install pygmt, I see the following errors:

Specifications:

  - pygmt -> python[version='>=3.6|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.9,<3.10.0a0|>=3.8,<3.9.0a0']

Your python: python=3.5
weiji14 commented 3 years ago

The conda case is easy to handle, just need to change recipe/meta.yaml, specifically the runtime requirements at https://github.com/conda-forge/pygmt-feedstock/blob/c3783e707743286ba63d9a32d5fc5f86f87207cc/recipe/meta.yaml#L27

For the pip case, we could strictly enforce Python 3.6+ (or 3.7+ if we adopt NEP29) via a python_requires setting (see https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requires) in our setup.py file (or pyproject.toml in the future). To be less aggressive (also less work), we could just leave things as it is and folks would just get SyntaxErrors for any non-compatible Python 3.7 code they run (which would be very few at this point).

seisman commented 3 years ago

For the pip case, we could strictly enforce Python 3.6+ (or 3.7+ if we adopt NEP29) via a python_requires setting

That makes sense. I once thought these lines define the minimum supported Python version.

https://github.com/GenericMappingTools/pygmt/blob/7e7da78c773c280d2d9c6724c79ce8b41f6532b0/setup.py#L31-L34

Then I'm in favor of the less aggressive option. We only test Python 3.7-3.9, but allow users run pygmt in old Python versions, although there is no guarantee that pygmt will work as expected.

seisman commented 3 years ago

Can we close the issue?

weiji14 commented 3 years ago

Should have mentioned this earlier. NEP29 implementation also requires that we set a minimum numpy version (currently numpy=1.15+, will be numpy=1.16+ in Jan 13, 2021).

There's a new PyPI package called nep29 at https://pypi.org/project/nep29/ that might help, see also https://github.com/numpy/numpy/issues/15173. Perhaps we could set up a monthly CI to check for this?

seisman commented 3 years ago

NEP29 implementation also requires that we set a minimum numpy version (currently numpy=1.15+, will be numpy=1.16+ in Jan 13, 2021).

But we're not testing old numpy versions, so how can we know if PyGMT works with numpy 1.15?

weiji14 commented 3 years ago

First question is: Do we want to adopt NEP29? If so, then there's a few things we'll need to do.

But we're not testing old numpy versions, so how can we know if PyGMT works with numpy 1.15?

One way is to split our CI into two:

This is inspired by xarray's setup at https://github.com/pydata/xarray/tree/v0.16.1/ci/requirements where they have different conda environment.yml files for each setup. There probably aren't many problems with running PyGMT code on NumPy 1.15, but checking a slightly older NumPy version does sound like a good idea to me.

Optionally, we could drop the middle (Python 3.8 + NumPy 1.17) check to save on resources, and have a 'bleeding' edge CI check (Python 3.10+ NumPy 1.20 + GMT Latest) so we're aware of any upstream changes. Again, we don't have to adopt NEP29 straightaway, but it's worth having a think about it.

seisman commented 3 years ago

Is it too much burden for us?

weiji14 commented 3 years ago

Ok, some time has passed, and we should now be supporting Python 3.7+ and Numpy 1.17+ only as per NEP29.

Optionally, we could drop the middle (Python 3.8 + NumPy 1.17) check to save on resources,

I think we should do this. It should be relatively straightforward to set the desired numpy version to install after #1033. Currently we have 9 builds in total (3 Python versions x 3 Operating Systems). If we drop the middle Python version, that would save even more CI resources (always feel bad about using up so many CPU cycles!). Specifically we will only need to test (as of March 2021):

I'll work on this NEP29 adoption in the next few hours, should get this sorted for the next PyGMT v0.4.0 release