TSC Meeting (10 Jan 2023) - Python3 + OS versions support burden

January 2023 @StackStorm/tsc 1 hour planned meeting:

Tuesday, 10 January 2022, 09:30 AM US Pacific.
See https://github.com/StackStorm/community/issues/33 for the schedule.
TLDR; Zoom link: https://us02web.zoom.us/j/81082101702?pwd=N1V4TWdYRVQ4SXBsaFh1TFYvVDA0UT09

Meeting Agenda

Scheduling of TSC meetings

should we move +1 week to avoid DST differences between US/Europe?
- 1st Tuesday of the month -> 2nd Tuesday of the month
  Python3 version support matrix burden
  
  The python 3 version matrix and how it relates to OS flavors we support when every OS ships with its own py3 version dictates where the st2 could be installed and tested (via https://github.com/StackStorm/st2ci/tree/master/rules). A big part of every release is that we need to plan for the upcoming OS/python3 matrix changes and it became a nightmare and a big support burden for the @StackStorm/tsc team.
  - currently we support U18 (py3.6), U20 (py3.8), EL7(py3.8?), RockyLinux-EL8(py3.8)
  - https://github.com/StackStorm/community/issues/103
  - identified python 3.9 and python 3.10 support as key candidates for release. Especially given Ubuntu 18.04 EOL early 2023, and Ubuntu 22.04 requiring python 3.10. RHEL also dropping support for python 3.8 on EL8, in favour of python 3.9.
  - with dropping U18 (EOL April 2023) we can drop python3.6 (which is already EOL)
  - Q: how we release 3.9 e.g. O/S packages, python packages, docker etc.
  - Q: how does the new pants packaging can help with the python version burden?
  - Note: supporting multiple python versions also affects how we manage StackStorm-Exchange CI, as every new py version adds overhead for the pack install (in)compatibility:
  - see: https://github.com/StackStorm-Exchange/stackstorm-jira/pull/49#issuecomment-1356391522
  - Options:
  - should we continue many versions python/OS support?
  - multiple python versions for each OS flavor
  - no maintainer resources
  - not fun, a nightmare
  - should we drop deb/rpm packages and ship docker only?
  - here is an image, use it on any OS
  - modern approach, many new projects do it
  - simplifies release automation and maintenance
  - we don't manage anymore how the stackstorm could be installed
  - free up resources, focus on the st2 core functionality
  - should we package our own python bundle together with st2?
  - removes python3 OS-level dependency
  - adds security maintenance burden
  - allows us to unify python3 support via single version
  - st2 core
  - exchange
  - still need to maintain/test deb/rpm for many OS flavors

Attendees

@cognifloyd, @amanda11, @nzlosh, @rush-skills, @winem, @armab, @nzlosh, @dalesmith, @mamercad, @arms11

Meeting Minutes

StackStorm at DigitalOcean

@mamercad highlighted the usage of StackStorm:
- automated fleet remediation
- ticket management
- not in prod yet, but soon
- Important:
- @DigitalOcean can potentially help with computing resources for @StackStorm build/release/CI infrastructure to help the project
- https://github.com/StackStorm/st2/pull/5836
- https://github.com/StackStorm/st2/pull/5852
- Planning to write a blog post at some point
- more TBD
ST2 Pain points so far:
- managing multiple st2 instances (one per region)
- pack installation and long-running workflows
- and more @mamercad to share in #tsc
StackStorm at CERN:
- Ankur @rush-skills is doing something similar with a large fleet and StackStorm Remediation at @CERN
- Presentation: https://indico.cern.ch/event/1200682/contributions/5090609/
Good to see a larger adoption
We encourage folks to list their production use case at StackStorm ADOPTERS.md
- This helps @StackStorm as a project!

Scheduling of TSC meetings

Every year we have at least 2+ points of confusion with the meeting schedule being a 1st Tuesday of the Month. Issues:

Europe time zone change vs US time zone
Bank holidays, New Year
@StackStorm/tsc proposed to move the meetings to the 2nd Tuesday of the month
- DECIDED!
- TODO: Change https://github.com/StackStorm/community/issues/33
- TODO: Change Google Calendar

Python3 version support matrix burden

During the past releases the python version vs OS version matrix was a burden for the team when different OS ship specific versions of python we need to support and package a dedicated st2 deb/rpm for it.

U18 will be deprecated soon with py3.6
U22 adds py3.10
EL8 will be dropping py3.8 and replacing with py3.9
Do we package our own py3 version as a deb/rpm package?
- and keep one py supported version for many years?
- pros: unified python version across the fleet, doesn't matter the OS
- downside: security patches/maintenance
@cognifloyd: try to support different python versions (instead of one)
- otherwise it just pushes the pain of python to a later point
Once we'll have pants working
- users can just install it via pypi
- irrespective of OS
- as long as they have supported py version
How Pants can help:
- building python wheels
  - we can distribute via pypi
  - as an official pre-bundled st2 package
  - with pip dependencies installed/pinned
- Opportunities:
  - simpler builds, real dependency pinning
  - untie st2 installation from OS
  - potentially removing the deb/rpm packaging
  - the artifact building responsibility is transitioning from deb/rpm -> py wheel
  - less moving pieces/testing/release machinery
  - if not removing deb/rpm, then slightly simplifying the build
  - arm builds in the future
  - paints WIP on adding the arm support
  - realistic for st2 to build the arm wheel
- Progress by @cognifloyd
  - 70 PRs so far
  - WIP since 2021
  - Target (a preview?) for the next 3.9 release
  - More reviews a welcome for the pants PRs!
  - https://github.com/StackStorm/st2/labels/pantsbuild

Conclusion

Many interesting technical discussions, DO ST2 adoption and usage, TSC Meeting schedule decision (move +1 week), Python3 support burden to simplify with pants wheels, tentatively target in st2 v3.9 as a preview (April 2023).

StackStorm / community