search

wrye-bash / wrye-bash

A swiss army knife for modding Bethesda games.
https://wrye-bash.github.io
GNU General Public License v3.0
455 stars 79 forks source link

Python 3.11 Upgrade #644

Closed Infernio closed 1 year ago

Infernio commented 1 year ago

Let's start this to keep track of the eventual upgrade.

Blockers:

Things to do once blockers are cleared:

Followup work in the future:

Infernio commented 1 year ago

I tried building a wxPython wheel for 3.11 and succeeded(!), but importing wx fails due to this bug: https://github.com/wxWidgets/Phoenix/issues/2297

Edit: upstream bug is closed as fixed, but no release has been made using the fix yet :(

Utumno commented 1 year ago

Good job - seems there is some activity in that one let's see if solved soon I am deep inside MobObjects here :)

On Fri, Oct 28, 2022 at 3:56 PM Infernio @.***> wrote:

I tried building a wxPython wheel for 3.11 and succeeded(!), but importing wx fails due to this bug: wxWidgets/Phoenix#2297 https://github.com/wxWidgets/Phoenix/issues/2297

— Reply to this email directly, view it on GitHub https://github.com/wrye-bash/wrye-bash/issues/644#issuecomment-1295031957, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKNIV7UCYKKVTVPEC37DSTWFPLP5ANCNFSM57ZLVUAA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

Infernio commented 1 year ago

I managed to use the workaround mentioned in that issue (which is a no-go for anything beyond testing, obviously, since the built wheel is still broken) to do some testing.

First impression is that WB with 3.11 feels much faster/more responsive than WB with 3.11. A quick benchmark of the BP confirms that impression.

py3.10: 0:00:15.016
py3.11: 0:00:09.656

No idea why the time is off (the real time was something like 39s for 3.10 and 34s for 3.11).

Edit: time is no longer off since d885f401b2ee47c4da8ad45d8a5886975f64146e.

Edit 2: another run, with the fixed timer:

py3.10: 0:00:37.091
py3.11: 0:00:34.129

Definitely faster, though might not be that much - then again, we also haven't optimized yet (i.e. getting rid of our "inlines", which are slower now IIRC).

Infernio commented 1 year ago 
$ py -3.11 -m timeit -s "a = 'foo'; a_isalpha = a.isalpha" "a.isalpha()"
20000000 loops, best of 5: 16.6 nsec per loop
$ py -3.11 -m timeit -s "a = 'foo'; a_isalpha = a.isalpha" "a_isalpha()"
10000000 loops, best of 5: 20 nsec per loop
$ py -3.10 -m timeit -s "a = 'foo'; a_isalpha = a.isalpha" "a.isalpha()"
20000000 loops, best of 5: 18.6 nsec per loop
$ py -3.10 -m timeit -s "a = 'foo'; a_isalpha = a.isalpha" "a_isalpha()"
20000000 loops, best of 5: 14.6 nsec per loop

Yeah, looks like we should get rid of "inlines" once we're on py3.11. Added it to the checklist.

Infernio commented 1 year ago

Some more testing: we should only go after the simple method "inlines". More complex ones, like this one from APreserver, are still useful:

In [1]: from dataclasses import dataclass

In [2]: @dataclass
   ...: class PatchFile:
   ...:     loadSet: frozenset[str]
   ...:

In [3]: @dataclass
   ...: class APreserver:
   ...:     patchFile: PatchFile
   ...:
   ...:     def initData(self):
   ...:         loaded_mods = self.patchFile.loadSet
   ...:         for x in range(100):
   ...:             if 'foo' in loaded_mods:
   ...:                 pass
   ...:

In [4]: import string

In [5]: pres = APreserver(PatchFile(set(string.ascii_letters)))

In [6]: %timeit pres.initData()
1.34 µs ± 7.4 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [7]: @dataclass
    ...: class BPreserver: # lol
    ...:     patchFile: PatchFile
    ...:
    ...:     def initData(self):
    ...:         for x in range(100):
    ...:             if 'foo' in self.patchFile.loadSet:
    ...:                 pass
    ...:

In [8]: pres2 = BPreserver(PatchFile(set(string.ascii_letters)))

In [9]: %timeit pres2.initData()
1.69 µs ± 6.54 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)
Infernio commented 1 year ago

Did some more testing. Zero-cost exceptions are going to be huge for us. I was able to save >1 million calls on startup by caching Path.__hash__ and FName.__hash__, which benefit massively from the zero-cost try for their cache.

lojack5 commented 1 year ago

Yeah, this is a good read-through especially. Lots of the in-lining should be a lot less necessary (I always hated that the patterns looked ugly).

Utumno commented 1 year ago

New wx issue for 3.11 https://github.com/wxWidgets/Phoenix/issues/2296

wheels https://alldunn.visualstudio.com/wxPython-CI/_build/results?buildId=1061&view=logs&j=1673e16c-f486-5fbe-9e06-af58c9e250b7

EwoutH commented 1 year ago

Python 3.11 wheels are now available with the release of lxml 4.9.2!

Infernio commented 1 year ago

We've got more locale nonsense in our future:

================================================== warnings summary ===================================================
Mopy/bash/tests/test_localize.py::TestLocalize::test_setlocale_pl_PL
Mopy/bash/tests/test_localize.py::TestLocalize::test_setlocale_nocli
Mopy/bash/tests/test_localize.py::TestLocalize::test_setlocale_pl_dash_PL
Mopy/bash/tests/test_localize.py::TestLocalize::test_setlocale_de_DE
Mopy/bash/tests/test_localize.py::TestLocalize::test_setlocale_sv_SE
Mopy/bash/tests/test_localize.py::TestLocalize::test_setlocale_fr_CM
  E:\Infernio\Desktop\Programming\wrye-bash\Mopy\bash\tests\test_localize.py:44: DeprecationWarning: Use setlocale(), getencoding() and getlocale() instead
    print(f'getdefaultlocale: {locale.getdefaultlocale()}')

Mopy/bash/tests/test_localize.py::TestLocalize::test_setlocale_nocli
Mopy/bash/tests/test_localize.py::TestLocalize::test_setlocale_pl_dash_PL
  E:\Infernio\Desktop\Programming\wrye-bash\Mopy\bash\localize.py:73: DeprecationWarning: Use setlocale(), getencoding() and getlocale() instead
    language_code, enc = locale.getdefaultlocale()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================================== 186 passed, 8 warnings in 4.17s ===========================================

getdefaultlocale() is deprecated and will be removed in Python 3.13. Thankfully we do have wxPython 4.2.0+ and Python 3.11 now, so maybe we can finally fix the locale issues for real.

Utumno commented 1 year ago

Re: mac:

(wrye-bash-3.11) ubik-mac13@ wrye-bash ((no branch, rebasing utumno-647)) $ python -m pip install attrdict3
Collecting attrdict3
  Downloading attrdict3-2.0.2-py2.py3-none-any.whl (10 kB)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, attrdict3
Successfully installed attrdict3-2.0.2 six-1.16.0
(wrye-bash-3.11) ubik-mac13@ wrye-bash ((no branch, rebasing utumno-647)) $ python -m pip install wxPython
Collecting wxPython
  Using cached wxPython-4.2.0.tar.gz (71.0 MB)
  Preparing metadata (setup.py) ... done
Collecting pillow
  Downloading Pillow-9.4.0-2-cp311-cp311-macosx_10_10_x86_64.whl (3.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 815.5 kB/s eta 0:00:00
Requirement already satisfied: six in /Users/ubik-mac13/venv/wrye-bash-3.11/lib/python3.11/site-packages (from wxPython) (1.16.0)
Collecting numpy
  Downloading numpy-1.24.1-cp311-cp311-macosx_10_9_x86_64.whl (19.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.8/19.8 MB 428.7 kB/s eta 0:00:00
Building wheels for collected packages: wxPython
  Building wheel for wxPython (setup.py) ... done
  Created wheel for wxPython: filename=wxPython-4.2.0-cp311-cp311-macosx_10_10_universal2.whl size=24773559 sha256=ac8d23ddb802d29608e2c7f00fcff1620f09b13f69f7fe28b19e7f4e3cafc2f6
  Stored in directory: /Users/ubik-mac13/Library/Caches/pip/wheels/11/0e/8e/c35fe8a9527b1f76dd8dc25defb111abc514c1fffbfd618071
Successfully built wxPython
Installing collected packages: pillow, numpy, wxPython
Successfully installed numpy-1.24.1 pillow-9.4.0 wxPython-4.2.0

Now let's see if it launches -> YES

Infernio commented 1 year ago

You may have to fix the extensions of the Python extension modules for wxPython like in https://github.com/wxWidgets/Phoenix/issues/2297, unless that was a Windows-only issue.