search

StackStorm / community

Async conversation about ideas, planning, roadmap, issues, RFCs, etc around StackStorm
https://stackstorm.com/
Apache License 2.0
8 stars 3 forks source link

Python 2 Deprecation #40

Closed arm4b closed 2 years ago

arm4b commented 4 years ago

There were some sparks and scattered talks around this topic that slip through the cracks. Let's start an async discussion about Python 2 Deprecation to think more how can we do this.

The Problem

Python 2 is old and reached its EOL. It served us a long time, but today per @blag valid point, Python 2 pulls the project down because of CI issues, our pip dependencies are dropping support for py2 and so it makes maintenance hard. Not to forget about the modern Python 3 features we can't use to keep the codebase py2-compatible.

Removing Python 2 support and relying on Python 3 only is the required thing to do. The only questions are When? and How?.

The Plan

1) Communication

We can't just go and remove python 2 in nearest release, breaking users installations. Don't be that software! This should be communicated beforehand. Make the blog Announcement, describe the transition plan, expectations and migration path so everyone is on the same page. That's why it's important for @StackStorm/tsc to come up with a conclusion about the desired plan, versions affected, roadmap, what needs to be done, when, who's in game and so on.

2) Deprecation Warnings

See https://github.com/StackStorm/st2/issues/4938 for more info. We'll need a Deprecation Warning in st2 to indicate that Python 2 will be fully removed in st2 v3.X (to be decided) version. As you can imagine, there are many users with old python2 legacy packs they wrote before.

So there are 2 aspects we want to cover:

As faster we can decide on this and implement, - earlier we could remove py2 later.

3) Real Work: EL7 and U18 packages on Python 3

After v3.3.0 CentOS 7 / RHEL 7 and Ubuntu Xenial will be the only platforms with py2 st2. The biggest amount and careful work is here, https://github.com/stackstorm/st2-packages packaging and making sure we prepare python3 virtualenv st2 packages. This task needs a technical research and thinking how to handle it better:

4) Follow-ups

st2 repository

Remove Python 2 from the CI build matrix

Puppet, Ansible, ~Docker, K8s~, Vagrant

There might be some simple follow-up tasks to update the affected deployment methods for python 3. I made sure that Docker/K8s is already covered.

Announce that Python 2 was finally Deprecated

As usual, we can announce it via Release Blog post when the py2 breaking change comes into effect. If someone interested to write more dedicated article about Python 2 lifetime and Python 3 plans, - that would be awesome to say "Good bye Python 2" and celebrate that.

Questions

1) Considering the py2->py3 transition path and deprecation warnings, which release version we want to target for the full py2 removal?

2) A couple of months ago idea about Python 2 Deprecation Working Group was mentioned. This needs to materialize. Who's involved, who wants to drive, help, contribute, work on these tasks? This is a perfect task to get involved and Contribute.

Please share your ideas, opinions, technical insight or voice for participation. Removing things should be fun!

Python 2 Deprecation project to be populated with the tasks: https://github.com/orgs/StackStorm/projects/15

namachieli commented 4 years ago

I think heavy communication is the way to go. Start having warnings in the install that call this out.

blag commented 4 years ago

I think it would be wise to split up the Python 2 discussion into two separate but related parts: the ST2 codebase itself, and packs on Exchange.

ST2 codebase itself

I don't know that we need to have deprecation warnings for Python 2 in our codebase. Everybody who codes in Python 2.7 has been warned about this for a few years now every time they fire up python, so it's not going to be a surprise for anybody.

We also don't have any official corporate sponsors anymore, so I don't feel like we owe it to anybody to continue to support Python 2.7 in future ST2 relases. Python 2.7 is deader than a "just resting" parrot! 😆

Exchange Packs

Exchange packs are a different story. Currently, most packs support both Python 2.7 and Python 3, including tests under both environments. That means that people who are still using ST2 on Python 2.7 might be installing packs using Python 2.7.

Ideally, we should support that until those versions of ST2 are EOL. While we do have a way for packs to advertise support for which Python version they support, I think it's fair to continue to support both Python versions in Exchange packs as long as there are ST2 versions that support Python 2.7.

This means that removing Python 2 code and tests from Exchange packs will be dependent on how quickly we can move to Python 3-only ST2 releases. Once the last version of ST2 that supports Python 2.7 is EOL, then we can start removing Python 2.7 support from Exchange packs and their CI/deploy configs.

The timeline for Exchange packs is therefore much longer than for the ST2 codebase. All the more reason to drop Python 2.7 sooner in ST2 itself, IMO.

arm4b commented 4 years ago

Interesting opinion about the python 2 in Exchange Packs.

I agree that as fast we remove py2 support would be better for the project.

I however disagree we can do it in one release or without a warning because everybody knows py2 is old. This is not the reality. I believe that communicating the deprecation, including deprecation warnings in st2 is the way to serve the community well. It's based on the the feedback we got previously from community and still getting similar signals.

Hence as earlier we can do this: https://github.com/StackStorm/st2/issues/4938 and start communicating py2 deprecation faster we can really drop it. Hence I believe actionable item here is to add this warning https://github.com/StackStorm/st2/issues/4938. BTW this was initially discussed in May.

amanda11 commented 4 years ago

I also have concerns that we are seeing errors being raised where packs work on python 2.7, and not python 3. There was an IMAP one in the last few weeks, before that I noticed ghost2logger only worked on python 2(though that was fixed with ST2 monkey_patch). In last week another raised on mysql. How many packs are heavily being used in python 3? Can we find out from community, or do code analysis (that iter_items problems seems a common one). Getting an "in your face" deprecation warning to me sounds a good idea.

guzzijones commented 4 years ago

We will have to ensure the OS we target support a package with a python3 named binary. Centos 7, 8. Ubuntu 18, 20. I think. That way we can change all the cmd scripts to use that one. Then we add this package as a dependency. Users need to be warned IMO.

arm4b commented 4 years ago

During the testing of Add python warning if install a pack that only supports python 2 a PR by @amanda11 a few things I noticed:

@amanda11 @winem @Sheshagiri @blag WDYT?

amanda11 commented 4 years ago

I was wondering whether for packs there was going to need to be a use-python2 flag, to replace the use-puthon3 when we went further with the py2 removal. But then my contrary argument to it was then does that mean people won't move and we still have to CI packs on py2. Though it would make things easier for users.

We could also put in a warning if python versions is not defined.

With the st2ctl start what warning are we giving the users? It isn't their fault that stackstorm on that platform is using py2. So would the message be annoying and unhelpful. Putting in logs would be ok and along the lines of warning them we will move it to py3. But I am not sure what the aim of a warning to them on startup is - it's not like the packs where they can do something about it. We may then get the questions about how to run st2 on Centos 7 with python 3, but as we don't CI it then it wouldn't be a supported variant. Therefore what was the aim of the warning?

amanda11 commented 4 years ago

More thoughts. On st2ctl I guess if the warning was along the lines of "Stackstorm will be deprecating support for python 2 in a near future release. Please make sure all your packs support python 3." might be an idea. So I might have changed my mind!

arm4b commented 4 years ago

Right, the first message to deliver is about any possible py2 packs. Second is the platform itself running under the python 2 environment which will likely be affected by the breaking py2 -> py3 change in the future releases. With that warning we could also encourage users migrating U16->U18 or EL7->EL8 platforms respectively. That sounds like a good "FYI" message so people are aware about the future Python 2 breaking changes and act accordingly.

amanda11 commented 4 years ago

Thanks, @armab That makes sense, and clarifies it. So a few things to get across on the py 2 startup then. On those Py2 systems are we telling them that we will support py3 on their current platform (if its EL7/U16) in the future, or not yet known? Just trying to think of phrasing.

arm4b commented 4 years ago

Not sure how the technical implementation would go, but yes, the future intent is to try to build the st2 packages with the python 3 dependency for both EL7 and U16 platforms.

arm4b commented 4 years ago

So Python 2 deprecation warnings (https://github.com/StackStorm/st2/issues/4938) were implemented by @amanda11 and will be included in the platform since v3.3 release. That includes warnings in: 1) st2 pack install 2) st2ctl 3) st2 services logs on startup

As a @StackStorm/maintainers @StackStorm/contributors team we'll also need to gather and discuss the plans and technical side about Python 2 deprecation. We'll need someone from @StackStorm/maintainers to lead this effort, write a blog post about Python 2 future plans to keep our community up to date with the Python 2 deprecation and give everyone enough time for migration.

amanda11 commented 3 years ago

After "volunteering" to lead this effort, I wanted to update with my understanding of the current status:

I have a blog post of "where we are", but will wait to publish until I can update to include what our future plans are.

As mentioned above we need to decide what the next steps are and the order. See @blag point above for differences between ST2 and Pack support.

How best to now proceed? Do we want to schedule a meeting to discuss in more detail? Who is interested in attending? Similar time of day to TSC?

Would be good to get some input on agenda for meeting, some suggestions below:

nmaludy commented 3 years ago

Added my thoughts below:

amanda11 commented 3 years ago

Thanks @nmaludy. Some queries:

nzlosh commented 3 years ago

I drew up a comparison between officially supported Python versions and their availability on officially supported OS'. (plus Debian because I'd like to see it included in the list of tested OS'). The colouring is based on how many days before the product hits EOL.

eol_python_vs_os

As @blag had already stated elsewhere, Python3.8 is the optimal version to support going forward. If we have Python3.8 support, it will give at least 2 years of official support of Python across Ubuntu, CentOS and Debian.

CentOS 6 and Ubuntu 16.04 support should be dropped now as there's no way forward with these OS'.

amanda11 commented 3 years ago

Thanks @nzlosh that's a great table.

One already gone :-) CentOS 6 has already been dropped in the StackStorm 3.3 release.

I'm seeing consensus so far on dropping Ubuntu 16.04 and adding Ubuntu 20.04/Python 3.8 support instead.

Will be interested to see if anyone, has a contrary view?

arm4b commented 3 years ago

That's a good breakdown! Looks like you people covered here all the technical corner cases and dependencies in one topic, eg. what's blocked on what. Great work! :+1:

Looking from the community support side, I don't think it's possible to drop the Ubuntu 16.04 support right now, until it reaches its EOL. Turns out however it's the only thing that would block py2 deprecation. I'm not sure if we want to wait with py2 until April 2021 though, py 3.6 from PPA might be an option? I guess timing really depends on team progress around py2 deprecation.

Ubuntu 20.04 and Python 3.8 is great. We can include it anytime as one of the supported platforms as soon as: 1.) Py 3.8 is added to the st2 build matrix, passed, stable, works 2.) New U20 packages are produced and well tested. This work could be done anytime and included in any st2 release as soon as it's ready. @nzlosh want to volunteer starting that work? With the new 3 months st2 release cadence things become easier.

arm4b commented 3 years ago

@StackStorm/maintainers @StackStorm/contributors Let's indeed organize a Python 2 Deprecation Meeting. We'll announce it so everyone from Community who wanted would be able to join and also discuss.

Same time as TSC meetings. Next week (13 Oct) or Week after (20 Oct) Tuesday? WDYT?

punkrokk commented 3 years ago

Considering no one is paying us to support Ubuntu 16, can we announce we will deprecate it soon rather than have to port python3 there? I know it’s not EOL, but we are a python project mostly and that’s a hard requirement.

JP Bourget / @punkrokk

On Oct 9, 2020, at 10:18 AM, Eugen C. notifications@github.com wrote:

 That's a good breakdown! Looks like you people covered here all the technical corner cases and dependencies in one topic, eg. what's blocked on what. Great work! 👍

Looking from the community support side, I don't think it's possible to drop the Ubuntu 16.04 support right now, until it reaches its EOL. Turns out however it's the only thing that would block py2 deprecation. I'm not sure if we want to wait until April 2021 though, 3.6 from PPA might be an option? I guess timing really depends on team progress around py2 deprecation really.

Ubuntu 20.04 and Python 3.8 is great. We can include it anytime as one of the supported platforms as soon as: 1.) Py 3.8 is added to the st2 build matrix, passed, stable, works 2.) New U20 packages are produced and well tested. This work could be done anytime and included in any st2 release. @nzlosh want to volunteer doing that work?

— You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

cognifloyd commented 3 years ago

We should announce that 3.2 (or 3.3) is the last release that will support Ubuntu 16. If any bug fixes are required for Ubuntu 16, we will gladly cut a new 3.2.x (or 3.3.x) release to resolve those issues for the next year while people work out how to transition to something better. New releases should not be saddled with the python2 baggage.

amanda11 commented 3 years ago

Having just spent a few hours fighting a python 2 pack - that actually doesn't work as several of its dependencies are now only python3 (and versions aren't pinned). The sooner py2 goes the better! I wonder how many packs work on a fresh ST2 py2 install.

arm4b commented 3 years ago

Considering no one is paying us to support Ubuntu 16

Payment factor is not a reason to treat the project community worse and scare away users.

Ideal scenario is to avoid breaking changes like suddenly removing platform before its EOL or dropping Python 2 in one release. We'd like to maintain the established project standards. At least team needs to seek the ways to solve the problem first and research what's possible. Sudden breaking changes are always easiest, but is a last resort if we talk about core st2.

amanda11 commented 3 years ago

To me it seems to me the centos7 upgrade to py3 is an easy win. We need to also add in ubuntu 20 and py 3.8 support. Then after that is all achieved we need to see what to do with Ubuntu 16. But happy to discuss further at py 2 meeting, but I don't see we can do anything on U16 until above is achieved.

arm4b commented 3 years ago

Team gravitated towards running the Python 2 Deprecation meeting on Tuesday, October 20 as this week is more v3.3 release-focused.

cognifloyd commented 3 years ago

Our StackStorm-Exchange CI infrastructure no longer supports python2.7 because:

  1. 3.4/master don't support python2. Fixing this requires an update (pin 2.7 tests to st2 3.3) in .circleci/config.yml in every pack that tests on python2.7.
  2. The packaging docker images no longer support python2.7, so the platforms (xenial and centos7) that used 2.7 with st2 3.3 can't build anymore: https://github.com/StackStorm/st2packaging-dockerfiles/tree/master/packagingbuild
  3. The 3.3 dependencies are currently not installable on python2.7: https://github.com/StackStorm/st2/pull/5131
  4. There may be other issues where we've fully migrated master to python3, but that is as far down the rabbit hole as I went.

I already started to prepare the PRs for 1, but then I ran into 2 and 3, and I expect 4 to come knocking any day now.

So, we need to decide if (a) its worth re-introducing python2.7 across our infrastructure to support exchange pack testsing on python2.7, or (b) drop python2.7 tests on the exchange.

If we drop the python2.7 tests on the exchange, then we could continue supporting 2.7 but only on an as-reported basis. If someone submits a PR for 2.7 and it doesn't break with python 3, great. If a pack is upgraded so that it can only be used with python 3, then people can downgrade to an earlier version that supported python 2.

I am willing to prepare/push PRs for all the exchange packs to drop python2 testing if we come to that decision.

Thoughts? Can we drop python2.7 testing on the exchange?

guzzijones commented 3 years ago

Drop 2.7 IMO.

On Fri, Jan 29, 2021, 2:08 PM Jacob Floyd notifications@github.com wrote:

Our StackStorm-Exchange CI infrastructure no longer supports python2.7 because:

  1. 3.4/master don't support python2. Requires an update (pin 2.7 tests to st2 3.3) in .circleci/config.yml in every pack that tests on python2.7.
  2. The packaging docker images no longer support python2.7, so the platforms (xenial and centos7) that used 2.7 with st2 3.3 can't build anymore: https://github.com/StackStorm/st2packaging-dockerfiles/tree/master/packagingbuild
  3. There may be other issues where we've fully migrated master to python3, but that is as far down the rabbit hole as I went.

So, we need to decide if (a) its worth re-introducing python2.7 across our infrastructure to support exchange pack testsing on python2.7, or (b) drop python2.7 tests on the exchange.

If we drop the python2.7 tests on the exchange, then we could continue supporting 2.7 but only on an as-reported basis. If someone submits a PR for 2.7 and it doesn't break with python 3, great. If a pack is upgraded so that it can only be used with python 3, then people can downgrade to an earlier version that supported python 2.

I am willing to prepare/push PRs for all the exchange packs to drop python2 testing if we come to that decision.

Thoughts? Can we drop python2.7 testing on the exchange?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/StackStorm/discussions/issues/40#issuecomment-769992629, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACZ5TIMEGE655U25DDOOPL3S4MBTPANCNFSM4OOHL7YQ .

blag commented 3 years ago

Agreed - drop Python 2.7. It's been dead-dead for over a year at this point. Anybody who hasn't upgraded off of it knows that they are playing with fire, and we are still generous enough to afford them the workaround of installing previous versions of a pack.

arm4b commented 3 years ago

+1. Worth mentioning in the upcoming v3.4.0 Release Announcement.

BTW dropping the Python2 Tests in Exchange de-facto means deprecation of the Python 2 with the respective pack metadata consequences.

cognifloyd commented 3 years ago

This is an example PR for what I'd like to do to all exchange packs after StackStorm-Exchange/ci#101 is merged: StackStorm-Exchange/stackstorm-acos#18

Feedback welcome before I push PRs to all the rest of the packs. I have a branch with these updates prepared for all packs.

Interestingly, once we drop python2.7 tests, the circleci config across packs becomes very consistent for all packs except for the weekly job schedule (Sunday vs Saturday) and some extra circleci jobs in zabbix pack.

arm4b commented 3 years ago

That went a bit excessive.. including Email, Slack, Github notifications. Python_2_Deprecation_·_Issue_40_·_StackStorm_discussions_-_2021-01-30_12 51 59

@cognifloyd To avoid future Github spam in StackStorm-Exchange, instead of opening a bunch of PRs, can you please work with @blag or @nmaludy so they’ll push the changes based on automation script you have directly to master for every pack?

This is pretty much an administrative task (we have st2 bot account for the purpose) and I appreciate your involvement to help the Exchange maintainers.

cognifloyd commented 3 years ago

Yes. Sorry for the spam. I haven't pushed PRs to all repos yet; I've only committed the changes to my forks. My mistake was including urls like this discussion's in the commit description. Again, I'm really sorry for spam.

arm4b commented 3 years ago

@cognifloyd NP and thanks for helping there. Let's try to rely on mechanisms we have to minimize the noise while doing the same impact. I'm thinking that the StackStrom-Exchange story might need a bit more prep.

arm4b commented 3 years ago

@StackStorm/tsc If we drop the Python 2 tests in Exchange, I think it's a bit more involved. Removing Python 2 Tests means de-facto dropping the py2 support. No tests = no support and there is no guarantee or indicator anymore from that point if pack code remains py2-compatible or not as well as no release setting the border line: py2 vs py3.

So removing tests is only start of the story.

We were serious about the approach taken with Python 2 Deprecation for StackStorm, including communication, thinking about user impact, dedicating a release, deprecation warnings and even some API/CLI changes postponed up to 3.5.

Let's be serious about the StackStorm Exchange as well. We could do something close to: 1) Drop Python 2 Tests for all the Exchange Packs (CircleCI) (as discovered by @cognifloyd and @blag). 2) Stop Python 2 support guarantee by removing python_versions: 2 from pack meta. 3) Add the Changelog record for every pack stating Deprecation of the Py2 support. 4) Bump "major" pack versions. Meaning if pack was 1.2.3 we'll release the new 2.0.0 setting the clear border line. 5) Do it in one go via st2 bot account against the master branches for every pack to minimize the noise. 6) Include the update of Dropping py2 in Exchange in the upcoming v3.4.0 Release Announcement (by @blag and @amanda11). That'll be perfectly aligned with the entire st2 py2 removal story.

This way we'll be super clear about what's going on and better balanced with respect to user expectations.

blag commented 3 years ago

No tests = no support

I strongly disagree with this assertion. There's plenty of code in StackStorm, and in StackStorm packs that has no test coverage, but I think we all consider such code still supported.

Ensuring that all supported code is tested is absolutely the ideal, but it is not a rule, and I don't even think it's an expectation.

@armab I like that checklist, but I don't think step 1 has to happen at the same time as step 2. And, if we drop python_versions: 2 at the same time as we release ST2 v3.5, then I don't even think we need to bump the pack version up a major version number.

Most churn in Exchange packs is due to dependencies suddenly causing failures because the dependencies have removed support for Python 2. So in the past we've pinned a lot of the dependencies that are failing. If we remove the Python 2 tests now, and then remove official support for them in May with ST2 v3.5, that's only going to be about 3 months where supported (by us) Python 2 code is exposed but not tested. And users who are still running on Python 2.7 are already aware that they are playing with fire by running on a runtime that has reached EOL, so I don't think the impact is going to be severe. And for whatever impact we do have, honestly, I'm out of empathy for those users at this point. Python 3 has been out for 13 years now. The writing has been on the wall for 13 years. Three more months of us still supporting that code (eg: by not merging in Python 3-only code) but not testing that code is not going to change the risk calculations for anybody still running on Python 2.7.

Edited to add:

Whatever we decide to do, we should consider the cost/benefit of it. We should not go through heroics to keep the Python 2.7 tests running, since they're only going to be useful for the next three months. And keeping support but not tests for Python 2.7 through ST2 v3.5 benefits our users without significantly costing us anything.

I'm don't consider support without tests to be an ideal solution, and I do not advocate that approach in general, but I think in this specific situation it's an acceptable, temporary compromise between keeping our users happy and amount of effort required on our parts.

cognifloyd commented 3 years ago

To be very clear: Python 2.7 pack tests on StackStorm-Exchange are broken and can't be fixed without significant time and effort.

I already spent several days on it, trying to get it working, and discovering several of the broken pieces. I don't think fixing the python 2.7 tests is worth the effort.