Python2 support

[fafhrd91]

I’d like to drop python2 support

[pganssle]

I don't have a huge problem with this, but I will say that as a library developer who is intending to support Python 2.7 for some time longer, it will make it harder to write a Rust backend for my libraries that support both 2 and 3.

It might be a good idea to get a stable API to enable polyglot Rust for those still supporting Python 2.7 for some time - e.g. if you want to write a Rust backend for your Python 2/3 library, pinning the 2.7 version against an old PyO3 build will still work with minimal code changes, even if you can only use a subset of all pyo3 features.

[jlgerber]

I happen to work in an industry (vfx for film/tv) in which python is ubiquitous (its embedded in a ton of commercial apps), but which has not updated to python3.x yet (vendors are targeting 2020). This would rule out Pyo3 for us (which is a shame). I suppose that it is good to know now.

[pganssle]

@jlgerber Well, luckily, support for Python 2 in the general ecosystem is waning every day. It's already past the point where you should be able to go to your management and say, "Tick tock, the writing is on the wall". The more things like PyO3 drop support for Python 2, the easier it will be to make the case that you need to be moving to Python 3 now.

[jlgerber]

@pganssle - As i said, our timetable is driven by commercial vendor integration of python3, which is slated for 2020 based on the vfx open platform. Until applications like Maya, Houdini, and Nuke all support python3, its moot.

[pganssle]

Well, point is that this sort of thing isn't going to get any easier. It's ridiculous that none of those applications read the writing on the wall years ago and added Python 3 support. Even "we're targeting 2020" is basically saying, "We are going to wait until the last possible minute". Making it clear that the official EOL is not the only thing to worry about will help people prioritize this.

That said, if you're in an environment where you can't even use Python 3, it's shocking to me that you would be allowed to use Rust. You probably don't want to get your Rust infrastructure tied up with all your legacy Python code anyway.

[konstin]

Python 2 support will be removed in 2019, as both python 2.7 will have it's end of life at 2020-01-01 and a significant share ecosystem will finally drop python 2 support.

[ijl]

python2 is 16% of usage on https://www.jetbrains.com/research/python-developers-survey-2018/.

How about removing it now? I can do a pull request if people are ok.

[konstin]

@ijl I think after 0.6 would be a good time to remove python 2 support. But note that it's not as trivial as deleting a bunch of files. You need unify num-common.rs and num3.rs, remove the Py_3 cfg flag, change the build script, etc.

[dw]

Sorely disappointed to see this. Python 2.x is not going away any time soon, regardless of how loud people may shout and scream that it is and will magically stop existing on some particular day, despite the reality of the free licenses covering the millions of lines of self-supported code in the 2.x ecosystem. This is a weird and deeply embarassing meme in the contemporary Python community that does not match whatsoever the behaviour of any other software in the history of computing.

There will be Python 2.x machines around in 10 years, and there will always be customers with applications so large and under-budgeted they could never have possibly moved. I hope the maintainability gains from dropping 2.x were profound and worth breaking compatibility for, as it severely limits the dependability of Rust for use in a commercial context: not just technically, but culturally. C++ is good enough, and that ecosystem always understood compatibility as an integral part of craftsmanship.

[pganssle]

@dw I'm not particularly happy about it since it will make it much harder for me to add a Rust backend to my libraries that still support Python 2, but it's a bit unfair to imagine that this is part of some conspiracy to wish away Python 2. I think most of the maintainers of this library just don't see Python 2 as a sufficiently important platform to support at the moment that it justifies the maintenance burden.

If you look at the library before this was merged, it was not actually written in a way that makes maintaining Python 2 support particularly easy, and it would be a decently large effort to actually unify the Python 2 and Python 3 codebases, which is a lot of burden, particularly when the kind of people who can use Rust (I think specifically nightly Rust, or if we're ready for stable then a very recent version) are not generally the kind of people who cannot use Python 3.

If you think it's worth the effort, forking this project to maintain Python 2 support is definitely an option. PyO3 is itself a fork of rust-cpython. If there's commercial interest in this, then you may even be able to get someone to pay to do it!

[dw]

The size of the removal diff is definitely quite surprising. From previous experience porting from 2.x to 3.x, the C API barely changed, and only a tiny handful of (C) macros were generally necessary to paper over the differences. So perhaps the previous approach taken in this project was more verbose than necessary and was a genuine source of maintainability problems.

Would there be interest in returning 2.x support if it could be done in a smaller amount of code? Really outside of module initialization and Unicode the 2.x/3.x APIs are highly compatible..

[konstin]

I can recommend reading https://python3statement.org/ which explains the reasons for removing python 2 support much better than I could

Would there be interest in returning 2.x support if it could be done in a smaller amount of code? Really outside of module initialization and Unicode the 2.x/3.x APIs are highly compatible..

No. Also for pyo3 there's a much bigger difference than for other library since the goal is to support basically all of the c api, we can't use c macros in rust and need to do our own linking logic. However as @pganssle said you're free to fork pyo3.

[dw]

the goal is to support basically all of the c api

This is easily possible without breaking backwards compatibility. I hope you realize the C API has continued to vary quite rapidly within the 3.x series. Perhaps the goal is to only fully support the daily CPython master branch?

rust-cpython serves the desired purpose without forcing any artificial tradeoffs, so forking won't be necessary.

[kngwyu]

Is there any statistic about how many people use Python2?

I'm basically a Rust programmer and really not familiar with the nature of the Python community. So if one say 'There will be Python 2.x machines around in 10 years...', I can't judge it's true or not from my knowledge.

[fafhrd91]

python2.7 EOL is January 1st 2020

[birkenfeld]

So if one say 'There will be Python 2.x machines around in 10 years...', I can't judge it's true or not from my knowledge.

It's certainly true, in the same way that 'There will be Windows XP machines around in 10 years' is true: outdated, unmaintained legacy versions keep being used (and will be exploitable) for various reasons. But in the interest of putting volunteer effort into what matters most, it is not worth supporting IMO.

Especially since I assume the cross section of legacy Python and Rust is pretty small, Rust 1.0 having been released 6.5 years after Python 3.0.

[dw]

According to PyPI BigQuery, 53% of all downloads yesterday that included an identifiable runtime version were downloaded by some version of pip or setuptools running on Python 2.x. That is 54.9 million packages downloaded by software that allegedly will cease to exist in 185 days time. In practical terms the EOL date is meaningless, as it bears no relationship with the longevity of Python 2.x in the real world.

The "real" EOL date is much further away -- and at least not before the last long term support releases expire for Linux distributions presently shipping it. For example, Ubuntu 18.04 which includes support for Python 2 does not reach EOL until April 2028.

[fafhrd91]

supporting old code if huge burden for maintainers, what is the problem with the fork?

[birkenfeld]

software that allegedly will cease to exist in 185 days time.

I'm not sure why you keep pushing that strawman: nobody says that Python 2 and its packages will be purged from existence upon EOL. Of course they won't. Likewise we're not talking about purging older versions of PyO3 that did support Python 2 from existence. But it is very reasonable, and common, for packages to stop supporting Python 2 at some point.

Ubuntu 18.04 which includes support for Python 2 does not reach EOL until April 2028.

The relevant date, i.e. end of publicly available support, is 2023.

[dw]

@fafhrd91 thanks for your feedback. I'll consider a "PyO23" if rust-cpython is eventually no longer sufficient.

[fafhrd91]

so you dont even use pyo3?

[dw]

@fafhrd91 I found this ticket in the process of figuring out whether I wanted to use PyO3 :) It is obviously more popular and the design looks nicer, but broken compatibility is a showstopper. Thanks for your work