Python 3.12 Support - Githubissues

rtibbles commented 1 year ago

Overview

Add support for Python 3.12 to Kolibri

### Tasks
- [x] Update the setup.py to include Python 3.12 as a supported version
- [x] Update our Python test matrix in Github Actions, and tox to include Python 3.12
- [ ] Ensure all Python automated tests still pass on Python 3.12
- [x] Cross check with Python 3.12 [what's new](https://docs.python.org/3.12/whatsnew/3.12.html) to look for any other possible incompatibilities
- [x] Update our use of importlib.metadata entry_points, as its dict interface has been removed, and instead `select` should be used to return all of the `kolibri.plugins` namespaced plugins - completed in #11417

This is not a blocker for 0.16.0, but would be best to add to the soonest next release, as Python 3.12 will be released on October 2nd 2023.

im-NL commented 11 months ago

Hello! I tried working on this and noticed that the package le-utils is not available in python 3.12,

ERROR: Could not find a version that satisfies the requirement le-utils==0.2.1 (from versions: 0.0.2, 0.0.3, 0.0.4, 0.0.7, 0.0.8, 0.0.9a0, 0.0.9b0, 0.0.9rc0, 0.0.9rc1, 0.0.9rc2, 0.0.9rc3, 0.0.9rc4, 0.0.9rc5, 0.0.9rc6, 0.0.9rc7, 0.0.9rc8, 0.0.9rc9, 0.0.9rc10, 0.0.9rc11, 0.0.9rc12, 0.0.9rc13, 0.0.9rc14, 0.0.9rc15, 0.0.9rc16, 0.0.9rc17, 0.0.9rc18, 0.0.9rc19, 0.0.9rc20, 0.0.9rc21, 0.0.9rc22, 0.0.9rc23, 0.0.9rc24, 0.0.10, 0.0.11, 0.0.12, 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.1.10, 0.1.11, 0.1.12, 0.1.13, 0.1.14, 0.1.15, 0.1.16, 0.1.17, 0.1.18, 0.1.19, 0.1.20, 0.1.21, 0.1.22, 0.1.23, 0.1.24, 0.1.25, 0.1.26, 0.1.27, 0.1.28a1, 0.1.28, 0.1.29, 0.1.30, 0.1.31, 0.1.32, 0.1.33, 0.1.34, 0.1.35, 0.1.36, 0.1.37, 0.1.38, 0.1.39, 0.1.40, 0.1.41, 0.1.42)
ERROR: No matching distribution found for le-utils==0.2.1

got this ^ on running pip install -r requirements.txt w/ Python 3.12

im-NL commented 11 months ago

pytest requires a module called "imp" which has been dropped completeley in python 3.12.

I believe this can be fixed if we switch to a pytest version > 5.0.0. Need guidance on how to proceed from here. (including the le-utils problem, should I raise an issue in learningequality/le-utils ?)

edit: would actually have to switch to pytest 6.2.5

MisRob commented 11 months ago

@im-NL I can see there is some conversation in https://github.com/learningequality/kolibri/pull/11420. I a m not sure if there's some support needed on this issue?

im-NL commented 10 months ago

Hello, first off I apologise for replying so late I have been busy with college. I have done the fifth task in this issue, but was facing a lot of problems while solving the other tasks as there is not enough compatibility of libraries with python 3.12 at the moment for us to add support for it, (some dependencies don't have support for py 3.12 yet). In #11420 it is just stated that we can add python 3.12 as a supported version only when all other tasks are finished (which they are not). I have mentioned the problems I was facing in some previous comments and did not understand how to continue.

MisRob commented 10 months ago

Thank you for the update @im-NL. I'm sorry to hear it's troublesome. The colleague who opened this issue and has more context about your previous work in this area is out for a couple of weeks, so I think it'd be best to wait with this particular task and return to this after he's back.

rtibbles commented 10 months ago

Raising an issue on le-utils would be helpful, as it seems adding support there is a needed prerequisite.

If there's no way to move forward on this without upgrading pytest, then this issue will have to wait on our dropping support for Python 2.7 (which is the reason we're pinned to such an old version of pytest) - that is another rabbit hole that I would not wish to send you down.

It may be better at this point to raise the issue on le-utils - which you would be welcome to solve, and I can give some guidance there - and then we'll unassign you from this issue, as it seems it is blocked for now.

rtibbles commented 8 months ago

Unassigning this for now - le-utils has been upgraded to add Python 3.12 support also.

holta commented 5 months ago

Python 3.12 support would be extremely useful 15 days from today — when Ubuntu 24.04 LTS is officially released!

As discussed within:

rtibbles commented 5 months ago

We will probably not make that deadline - although to avoid issues for end users, our PPA will not be updated to support 24.04 until we have resolved this!

holta commented 5 months ago

our PPA will not be updated to support 24.04 until we have resolved this!

Are you saying there might be an actual PPA for Kolibri 0.17 pre-releases?

(Rightly or wrongly I was under the assumption there would not be — and that 0.17 pre-releases would instead be made available using redirect URL https://learningequality.org/r/kolibri-deb-next — as outlined at https://github.com/learningequality/kolibri/issues/11892#issuecomment-1950323027 ?)

rtibbles commented 5 months ago

I had originally suggested that here: https://github.com/learningequality/kolibri/issues/11892#issuecomment-1950245601

But you had said that was not so helpful, and a fixed URL would be of most interest.

holta commented 5 months ago

I had originally suggested that here: #11892 (comment)

But you had said that was not so helpful, and a fixed URL would be of most interest.

True that a PPA helping implementers stay up-to-date with the absolute very latest "bleeding edge" Kolibri could also be incredibly useful!

(I guess I was just trying to make things easier for you, when I suggested a "dead simple" redirect URL similar to https://learningequality.org/r/kolibri-deb-latest, allowing for rapid "one off" bureaucracy-free download-and-install.)

RECAP: Both approaches / Either approach extremely useful, each in their own way, so whatever you decide! 💯

rtibbles commented 5 months ago

Righto!

holta commented 5 months ago

Righto!

@rtibbles and @marcellamaki and @RadinaMatic if they are interested:

What is Quality[*] ?

Integration Testing ("the real IT") en route — e.g. community CI/CD or similar, during actual product development — instead of "kicking the can down the road" to untangle serious bugs later (e.g. after 0.17 is released, or years later...) 🔭
In that sense, @rtibbles was right on to push for something like a regularly-updated PPA back in February (if this is attainable!!) 🙏
KEY: Things like pre-release PPAs and snap risk-level channels {stable, candidate, beta, edge a.k.a. alpha} fundamentally represent an ongoing stream (trust relationship) empowering {QA people, all staff who touch Kolibri no matter how occasionally, hidden demographics, wider global community} to tighten up the most purposeful feedback loops — consciously or unconsciously — building a Quality Community Product. 💯

[*] Or, for another opinion, read https://en.wikipedia.org/wiki/Zen_and_the_Art_of_Motorcycle_Maintenance !

(While trying to answer the Quality/Community question below as conscientiously as we can...)

11892

PS My personal opinion is that Snap's 4 risk-level channels (release streams) are way too many — simplifying with just 2 release channels/streams is almost always best — e.g. I like to name these "conservative" vs. "progressive" but I realize others dislike those politically overloaded terms (-:

holta commented 5 months ago

Kolibri 0.17 presumably will also need work with Python 3.13, as Debian 13 "Trixie" converges around that later this year?

Anyway something to possibly think about.

With Python 3.13.0 Beta 1 expected next Tuesday (2024-05-07) with "No new features beyond this point" e.g. beyond those roughly outlined here:

https://docs.python.org/3.13/whatsnew/3.13.html

(Python 3.13 final release is 5 months away, on 2024-10-01:)

https://peps.python.org/pep-0719/

marcellamaki commented 5 months ago

Hi @holta - since we have been working hard over the last year to transition towards a more iterative release process and schedule (in line with many of the same ideas you laid out in your comment above!) with more frequent minor releases of Kolibri, we plan to release 0.18 this fall. This would be in line with the Python 3.13 final release timeline you lay out here, and we will track 3.13 support as part of that project. We will just be focusing on Python 3.12 for 0.17.

holta commented 3 months ago

Hi @holta - since we have been working hard over the last year to transition towards a more iterative release process and schedule (in line with many of the same ideas you laid out in your comment above!) with more frequent minor releases of Kolibri, we plan to release 0.18 this fall. This would be in line with the Python 3.13 final release timeline you lay out here, and we will track 3.13 support as part of that project. We will just be focusing on Python 3.12 for 0.17.

@marcellamaki @rtibbles

Huge thanks! 💯

Any chance a Kolibri 0.17 "alpha" might be available to test with Python 3.12 (e.g. Ubuntu 24.04 LTS) later this month?

rtibbles commented 3 months ago

We should do, yes - although one of our QA team did test 0.16.2 on Ubuntu 24.04 and reported it working - have you seen different behaviour?

holta commented 3 months ago

@rtibbles

Just FYI Kolibri 0.16.2 still fails on Ubuntu 24.04.

Specifically, the initial stage works (apt install kolibri), but then kolibri manage provisiondevice --facility "Kolibri-in-a-Box" --superusername "Admin" --superuserpassword "changeme" --preset "formal" --language_id "en" fails:

TASK [kolibri : Provision Kolibri, while setting: facility name, admin acnt / password, preset type, and language] ** fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": "\"/usr/bin/kolibri\" manage provisiondevice --facility \"Kolibri-in-a-Box\" --superusername \"Admin\" --superuserpassword \"changeme\" --preset \"formal\" --language_id \"en\" #--preset \"formal\" --language_id \"en\" --verbosity 0 --noinput\n", "delta": "0:00:00.866802", "end": "2024-06-13 13:13:55.547582", "msg": "non-zero return code", "rc": 1, "start": "2024-06-13 13:13:54.680780", "stderr": "Traceback (most recent call last):\n File \"/usr/bin/kolibri\", line 33, in \n sys.exit(load_entry_point('kolibri==0.16.2', 'console_scripts', 'kolibri')())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/bin/kolibri\", line 25, in importlib_load_entry_point\n return next(matches).load()\n ^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/importlib/metadata/init.py\", line 205, in load\n module = import_module(match.group('module'))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib/python3.12/importlib/init.py\", line 90, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"\", line 1387, in _gcd_import\n File \"\", line 1360, in _find_and_load\n File \"\", line 1331, in _find_and_load_unlocked\n File \"\", line 935, in _load_unlocked\n File \"\", line 995, in exec_module\n File \"\", line 488, in _call_with_frames_removed\n File \"/usr/lib/python3/dist-packages/kolibri/utils/cli.py\", line 12, in \n from django.core.management import execute_from_command_line\n File \"/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/init.py\", line 10, in \n from django.apps import apps\n File \"/usr/lib/python3/dist-packages/kolibri/dist/django/apps/init.py\", line 1, in \n from .config import AppConfig\n File \"/usr/lib/python3/dist-packages/kolibri/dist/django/apps/config.py\", line 4, in \n from django.core.exceptions import ImproperlyConfigured\n File \"/usr/lib/python3/dist-packages/kolibri/dist/django/core/exceptions.py\", line 5, in \n from django.utils.encoding import force_text\n File \"/usr/lib/python3/dist-packages/kolibri/dist/django/utils/encoding.py\", line 11, in \n from django.utils.six.moves.urllib.parse import quote, unquote\nModuleNotFoundError: No module named 'django.utils.six.moves'", "stderr_lines": ["Traceback (most recent call last):", " File \"/usr/bin/kolibri\", line 33, in ", " sys.exit(load_entry_point('kolibri==0.16.2', 'console_scripts', 'kolibri')())", " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", " File \"/usr/bin/kolibri\", line 25, in importlib_load_entry_point", " return next(matches).load()", " ^^^^^^^^^^^^^^^^^^^^", " File \"/usr/lib/python3.12/importlib/metadata/init.py\", line 205, in load", " module = import_module(match.group('module'))", " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", " File \"/usr/lib/python3.12/importlib/init.py\", line 90, in import_module", " return _bootstrap._gcd_import(name[level:], package, level)", " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", " File \"\", line 1387, in _gcd_import", " File \"\", line 1360, in _find_and_load", " File \"\", line 1331, in _find_and_load_unlocked", " File \"\", line 935, in _load_unlocked", " File \"\", line 995, in exec_module", " File \"\", line 488, in _call_with_frames_removed", " File \"/usr/lib/python3/dist-packages/kolibri/utils/cli.py\", line 12, in ", " from django.core.management import execute_from_command_line", " File \"/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/init.py\", line 10, in ", " from django.apps import apps", " File \"/usr/lib/python3/dist-packages/kolibri/dist/django/apps/init.py\", line 1, in ", " from .config import AppConfig", " File \"/usr/lib/python3/dist-packages/kolibri/dist/django/apps/config.py\", line 4, in ", " from django.core.exceptions import ImproperlyConfigured", " File \"/usr/lib/python3/dist-packages/kolibri/dist/django/core/exceptions.py\", line 5, in ", " from django.utils.encoding import force_text", " File \"/usr/lib/python3/dist-packages/kolibri/dist/django/utils/encoding.py\", line 11, in ", " from django.utils.six.moves.urllib.parse import quote, unquote", "ModuleNotFoundError: No module named 'django.utils.six.moves'"], "stdout": "", "stdout_lines": []}

Error(s) pasted in below appear... very much the same as 2024-02-29 in the link here:

https://github.com/learningequality/kolibri/issues/11895#issuecomment-1971758266

root@u24-kolibri:~# journalctl -u kolibri
Jun 13 13:13:50 box systemd[1]: Starting kolibri.service - LSB: kolibri daemon, an offline education platform...
Jun 13 13:13:50 box runuser[26048]: pam_unix(runuser:session): session opened for user kolibri(uid=997) by (uid=0)
Jun 13 13:13:50 box runuser[26048]: pam_unix(runuser:session): session closed for user kolibri
Jun 13 13:13:50 box systemd[1]: kolibri.service: Control process exited, code=exited, status=1/FAILURE
Jun 13 13:13:50 box systemd[1]: kolibri.service: Failed with result 'exit-code'.
Jun 13 13:13:50 box systemd[1]: Failed to start kolibri.service - LSB: kolibri daemon, an offline education platform.
Jun 13 13:13:51 box systemd[1]: Starting kolibri.service - LSB: kolibri daemon, an offline education platform...
Jun 13 13:13:51 box runuser[26313]: pam_unix(runuser:session): session opened for user kolibri(uid=997) by (uid=0)
Jun 13 13:13:51 box runuser[26313]: pam_unix(runuser:session): session closed for user kolibri
Jun 13 13:13:51 box systemd[1]: kolibri.service: Control process exited, code=exited, status=1/FAILURE
Jun 13 13:13:51 box systemd[1]: kolibri.service: Failed with result 'exit-code'.
Jun 13 13:13:51 box systemd[1]: Failed to start kolibri.service - LSB: kolibri daemon, an offline education platform.
Jun 13 13:13:56 box systemd[1]: Starting kolibri.service - Kolibri...
Jun 13 13:13:57 box kolibri[26728]: Traceback (most recent call last):
Jun 13 13:13:57 box kolibri[26728]:   File "/usr/bin/kolibri", line 33, in <module>
Jun 13 13:13:57 box kolibri[26728]:     sys.exit(load_entry_point('kolibri==0.16.2', 'console_scripts', 'kolibri')())
Jun 13 13:13:57 box kolibri[26728]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 13 13:13:57 box kolibri[26728]:   File "/usr/bin/kolibri", line 25, in importlib_load_entry_point
Jun 13 13:13:57 box kolibri[26728]:     return next(matches).load()
Jun 13 13:13:57 box kolibri[26728]:            ^^^^^^^^^^^^^^^^^^^^
Jun 13 13:13:57 box kolibri[26728]:   File "/usr/lib/python3.12/importlib/metadata/__init__.py", line 205, in load
Jun 13 13:13:57 box kolibri[26728]:     module = import_module(match.group('module'))
Jun 13 13:13:57 box kolibri[26728]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 13 13:13:57 box kolibri[26728]:   File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
Jun 13 13:13:57 box kolibri[26728]:     return _bootstrap._gcd_import(name[level:], package, level)
Jun 13 13:13:57 box kolibri[26728]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jun 13 13:13:57 box kolibri[26728]:   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
Jun 13 13:13:57 box kolibri[26728]:   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
Jun 13 13:13:57 box kolibri[26728]:   File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
Jun 13 13:13:57 box kolibri[26728]:   File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
Jun 13 13:13:57 box kolibri[26728]:   File "<frozen importlib._bootstrap_external>", line 995, in exec_module
Jun 13 13:13:57 box kolibri[26728]:   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
Jun 13 13:13:57 box kolibri[26728]:   File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 12, in <module>
Jun 13 13:13:57 box kolibri[26728]:     from django.core.management import execute_from_command_line
Jun 13 13:13:57 box kolibri[26728]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 10, in <module>
Jun 13 13:13:57 box kolibri[26728]:     from django.apps import apps
Jun 13 13:13:57 box kolibri[26728]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/apps/__init__.py", line 1, in <module>
Jun 13 13:13:57 box kolibri[26728]:     from .config import AppConfig
Jun 13 13:13:57 box kolibri[26728]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/apps/config.py", line 4, in <module>
Jun 13 13:13:57 box kolibri[26728]:     from django.core.exceptions import ImproperlyConfigured
Jun 13 13:13:57 box kolibri[26728]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/exceptions.py", line 5, in <module>
Jun 13 13:13:57 box kolibri[26728]:     from django.utils.encoding import force_text
Jun 13 13:13:57 box kolibri[26728]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/utils/encoding.py", line 11, in <module>
Jun 13 13:13:57 box kolibri[26728]:     from django.utils.six.moves.urllib.parse import quote, unquote
Jun 13 13:13:57 box kolibri[26728]: ModuleNotFoundError: No module named 'django.utils.six.moves'
Jun 13 13:13:57 box systemd[1]: kolibri.service: Control process exited, code=exited, status=1/FAILURE
Jun 13 13:13:57 box systemd[1]: kolibri.service: Failed with result 'exit-code'.
Jun 13 13:13:57 box systemd[1]: Failed to start kolibri.service - Kolibri.

rtibbles commented 3 months ago

OK, thanks @holta!

holta commented 3 months ago

Any chance a Kolibri 0.17 "alpha" might be available to test with Python 3.12 (e.g. Ubuntu 24.04 LTS) later this month?

We should do, yes

Thanks @marcellamaki, @rtibbles & All for 0.17.0 Alpha 0 !

Are either of February's #11892 automated/streamlined install paths now becoming more viable?
Do go ahead and offer both PPA "streams" if practical:
1. Risk-Averse PPA ≈ latest release, e.g. 0.16.2 (equivalent to https://learningequality.org/r/kolibri-deb-latest !)
2. Risk-Loving PPA ≈ latest PRE-release, e.g. 0.17.0 Alpha 0 (equivalent https://learningequality.org/r/kolibri-deb-next ?)

PS As @jredrejo knows, any school system wanting to switch from "bleeding-edge / progressive PPA" to "stable / conservative PPA" could do this almost effortlessly using...

add-apt-repository --remove ppa:whatever/ppa

...or...

ppa-purge ppa:whatever/ppa

holta commented 3 months ago

Clarif that Kolibri 0.17 Alpha 0 basically works on Ubuntu 24.04 LTS — but take note of...

1) kolibri --version output isn't very clean, as you probably know:

   root@box:~# kolibri --version
   /usr/lib/python3/dist-packages/kolibri/dist/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.18) or chardet (5.2.0)/charset_normalizer (2.0.12) doesn't match a supported version!
     warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
   kolibri, version 0.17.0a0

2) Initial visit to page http://box/kolibri got stuck, showing...

(But forcing a reload of the page worked.)

iiab-diagnostics showing installation details: https://dpaste.com/GM755NQMA

rtibbles commented 3 months ago

Do go ahead and offer both PPA "streams" if practical:

We do currently have kolibri-proposed and kolibri PPAs for this purpose, although the main issue at the moment is the lack of CD to our kolibri-proposed PPA. The main limitation here is just our internal bandwidth to implement this - it's on the radar, but other things are taking priority at the moment.

rtibbles commented 3 months ago

The page 'getting stuck' feels like a caching issue, but I can't say for sure.

holta commented 3 months ago

The page 'getting stuck' feels like a caching issue, but I can't say for sure.

Kolibri 0.17 Alpha 0 has the exact same "first visit to page http://box/kolibri gets stuck" bug on Ubuntu 24.10

Just like on Ubuntu 24.04

(Very minor bug, but FYI.)

rtibbles commented 3 months ago

The issue you are seeing is the same as this issue, it looks like: https://github.com/learningequality/kolibri/issues/12305

We have a fix for it, which should go into the next pre-release of 0.17.0.

holta commented 3 months ago

We have a fix for it, which should go into the next pre-release of 0.17.0

Fix confirmed!

Just FYI 0.17.0 Beta 0 still has ugly output at the command-line here:

root@box:~# kolibri --version
INFO: Initialized plugins.json
/usr/lib/python3/dist-packages/kolibri/dist/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.19) or chardet (5.2.0)/charset_normalizer (2.0.12) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
/usr/lib/python3/dist-packages/kolibri/dist/magicbus/process.py:9: SyntaxWarning: invalid escape sequence '\ '
  """A Bus subclass for managing the state of a process.
/usr/lib/python3/dist-packages/kolibri/dist/magicbus/plugins/servers.py:1: SyntaxWarning: invalid escape sequence '\ '
  """
kolibri, version 0.17.0b0

root@box:~# python3 --version
Python 3.12.4

marcellamaki commented 2 months ago

Closed with #12468

learningequality / kolibri

Python 3.12 Support #11316

Overview

11892