Move minimal supported Python version to 3.11

[ela-kotulska-frequenz]

What's needed?

It would be very nice to be able to use new improvements in Python 3.10/3.11, which introduced some big features and some major improvements to typing:

• dataclass_transform
• Self
• Variadic generics
• Pattern matching (match statement)
• Exception groups

Currently we need to use a few hacks to be able to do simple things, like forward declarations (using from __future__ import annoations) or include extra backported modules like typing_extensions or exceptiongroups.

Proposed solution

Move to python 3.11 as the minimum required version.

Blockers

In README there is sentence: For arm64 only Python 3.8 is supported (due to some dependencies that only support 3.8).

These dependencies was an xgboost package that I've build locally, since we wanted to have gpu support for our nvidia jetson xaviar nx edge controllers Actually these dependencies were never used anywhere. It is not build and needed now.

• [x] xgboost compiled for controllers GPU: not being used right now.
• [x] Ubuntu shipping 3.11: We can use the official Python docker images (based on Debian/alpine)
• [x] mypy doesn't support all 3.11 features yet (not a hard blocker though): https://github.com/python/mypy/issues/12840 (is not all done but only very minor and obscure things are missing, so it should not be a blocker)
• [x] Tensorflow only supports 3.10 for now, but 3.11 is coming: https://github.com/tensorflow/tensorflow/issues/58032
• [x] Ask @frequenz-floss/datasci-team if there are other required ML libraries needed
• [ ] Disable tests for older python versions in branch protection

Check if a dependency is Python 3.11-compatible

• https://observablehq.com/d/eb336fc983fcdf52

[christianparpart]

Actually, locally, on all my Linux systems, I am having Python 3.11 installed. Despite a few issues today I had in getting this repo built, I managed to succeed, using Python 3.11. Maybe we can run the tests against 3.10 as well as 3.11 and support both?

[shsms]

Oh, I thought we were already supporting python 3.11. I guess we need to add that to the CI first, then.

[christianparpart]

Oh, I thought we were already supporting python 3.11. I guess we need to add that to the CI first, then.

NB: It's on CI now.

[leandro-lucarella-frequenz]

I renamed this issue because I think it was not about supporting Python 3.10 (which was already supported) but to make it the minimum supported Python version (so we can use more modern features in the SDK itself). I also updated it to Python 3.11, as eventually Tensorflow should support it.

Still, I would be very careful about doing this move, as once it is done there is no (easy) way back. Things I would have in mind before doing this:

• Check we'll really never, ever need to use a compiled package that it is only supported in Python 3.8 (either because we won't use the package or because it can be compiled with Python 3.10 / 3.11 too).

• Check the current supported version in the latest Ubuntu LTS (and probably other popular distros). This is a nice to have, but I think making it work out of the box in a Ubuntu LTS is a nice feature. If we decide against this, we have to be conscious people might need to compile their own Python to use the SDK.

I also don't think we need to do this in v0.18.0, so we might consider moving it to a future milestone.

[leandro-lucarella-frequenz]

I updated this issue again, adding current blockers to move to Python 3.11. I removed moving to Python 3.10 as an option because I think is too much effort to move to 3.10 first and shortly after to 3.11 again (I assume the blockers will be lifted sooner than later, but we can reconsider if that doesn't end up happening).

[leandro-lucarella-frequenz]

The current tensorflow is compatible with Python3.11, at least it is installable in Python 3.11. It seems like some issues remain with supporting CUDA.

Can we do some testing to see if it is enough for what we need? It would be awesome to make the jump to 3.11, also for all the background task stuff having task groups and exception groups would make everything much easier.

[leandro-lucarella-frequenz]

Now the issue in tensorflow is closed, there is one comment saying cuda is supported. Someone complains there are no binary wheels for mac M1, but I guess that shouldn't be a stopper for us.

[leandro-lucarella-frequenz]

I added a link a web that shows major libraries compatibility with 3.11. It's not perfect because then a project specifies an open ended python version support (like we do), it's not 100% sure it really works, but in general it should: https://observablehq.com/d/eb336fc983fcdf52