Python 3 Upgrade

[Infernio]

Goold old Python 2 is dead in about 3 months as of the time of writing.

What follows is a list of roadblocks:

• [x] chardet. We bundle version 1.0.1, from 2008. May just happen to work on py3, but I wouldn't hold my breath - done in 8e201c49bbc809da89b1bda1d269f4cb7619dfc0.
• [x] wxPython. Neither 2.8 nor 3.0 have Python 3 versions available. However, the work put into the wx3 upgrade was not wasted; it will make upgrading to wxPython 4.0, which does have Python 3 support, much easier - see 050391ca22d7c8451390cbff6fd150ab0b9bcabd, done in 22de7ff9e804b2bcaa8a819922f4b5100b86d80f.
  • [x] loot-api-python. Python 3 versions are available starting from version 4.0, but see #431 for the obvious problem there - done in 934f3527e59ac8b0f923c2f6dc6908ad902a1c46.
  • [x] #480 and #312 are blockers, for the most part - will be much easier to proceed to Python 3 once they've progressed further
  • [x] Sub-issues:
  • [x] #470
  • [x] #597
  • [x] #491
  • [x] Well, the entire codebase. A decent chunk can probably be handled by automated tools, but we need to consider the output carefully - plus automated tools won't be nearly enough to catch everything. In particular:
  • [x] Unicode strings and byte strings
    • [x] ur'' is gone - done in 660ecbb81f49d478bdc8e4a8905e328d1daf9dca and d43ad244170e2110a6daca7d5febed4020550247
    • [x] We should go through all our unprefixed string usages and add either b'' or u'' - done in way too many commits to list
    • [x] StringIO needs to be changed to BytesIO where applicable - done in 7c95d05c629a9a368f1ad5e6b4b3243de504249a
    • [x] str needs to be replaced with bytes or unicode depending on how it's used - done in 04835ecce0f82d59f02faba38099ebaa74633922
    • [x] basestring is problematic - it's gone in py3, but 2to3 will change it to str. Most usages should be replaced with unicode or bytes, but a couple really do need to handle both - done in 42b879faa847df37bd75d5cc8d4d1d38a7e7dfc4
    • [x] Revisit all open uses, check if opening in bytes mode is needed - done in c120e18a43c2f5abb31d579c99f211e7aee6b322
  • [x] We need to get rid of getattr calls that pass in bytes, most notably due to ModFile.__getattr__ - #312/#480, (hopefully) done in 0848872b18c6fb6554cc757a3372d7ed86e613c9
  • [x] Tuple parameter unpacking is gone in py3. The only place we use this is in lambdas, which is handled by 2to3 - done in 7a6b7c2a591013c05f6f5d6fb827acbf81d65662
  • [x] zip/map/filter/range need to get replaced with either the itertools/xrange variant or an equivalent comprehension - plus add list() calls if we modify the iterables in the loop - worked on in 709e146b3d968af527bf1d055fa86e1ca4eaaa3d and e8c4d4388d1c3747c48473975c4293c2cfe3f314, done in a75dbc638a349e7dc893aabd7823cb25d2dfb2ea
  • [x] All keys and iterkeys calls can go -> done in 567f1539ed2e6708554cc47ef4656febd13d92b3 and 09e1633a617aeec6ebb5139e6099dc848b825a25
  • [x] items and values: Change to iteritems/itervalues, then run 2to3's dict fixer. If it changes them to iter(...), change them to viewitems/viewvalues instead - done in d22469bc235ab67b14e14dd45e4e3ce268bd936d
  • [x] Old-style classes need to be switched to new-style classes ahead of time to catch any issues that arise - see the commit message in fa74a1b7a61d9b3150f0d2b171145e171f2d27e5, for example - worked on in cae844b9dde8af014b09a1cb24af2348d5620058, done in 6db5b8b59e28bc46a9d42e966d31007e113c59e6
  • [x] True division updating - for each usage, we need to consider if we want floats or integers - done in 505d26f2edef37d9356f5f2fb9e24e50c55df705
  • [x] Relative imports - are we using the explicit ones everywhere already? Should mostly be caught by automated tools, but problems can arise here - done in c6926621258353a7cff14a58bc7f468ddaf6b804
  • [x] Longs are gone, replaced with regular integers - a bunch of unnecessary usages in records code, which will be dropped in the course of #480, any other usages should be investigated - worked on in d8d03ca39e1e9f85250fd014cabcc2a65945e5e7, done in 197b6a2de78acd723f9d747fd6751fd2c68cf944
  • [x] ints no longer have a size limit, so sys.maxint is gone - depending on the use case, could be replaced by sys.maxsize - done in 659e5b696be5083b9bef0d39356acc30ab46b5a4
  • [x] cmp and __cmp__ are gone; usage is rare - five in ScriptParser and belt and one each in cint, balt and mods_metadata - done in 5a98eb4c025651f4e9366db2a7d488ec2068f1fc'

Things that are not roadblocks:

• [x] comtypes - dropped during wxPython 3.0.2.0 upgrade
• [x] gitpython - dropped in build script rework, see pygit2 below
• [x] py2exe - readily supports Python 3 - we should however consider our options once we get there, see #491
• [x] pyfiglet - readily supports Python 3
• [x] pygit2 - readily supports Python 3
• [x] pymupdf - readily supports Python 3
• [x] pytest - readily supports Python 3
• [x] python-lz4 - readily supports Python 3
• [x] pywin32 - readily supports Python 3
• [x] pyyaml - readily supports Python 3
• [x] scandir - built into Python 3, we'll just drop this once we upgrade
• [x] toml - readily supports Python 3

Useful tracking regexes (make sure to enable *.py mask):

• ur prefix: ur('|")[^,)]
• Old-style classes: class [^:(]+:
• Old-style super initializer calls: (?<!self\))\.__init__
• WIP regex for all old-style super calls, has tons of false positives: [A-Z]\w+\.\w+\(self(,|\))
• Absurd regex for tracking non-library imports that need updating: ^ *import (?!os|sys|re|errno|time|subprocess|io|threading|wx|wx\.adv|cPickle|argparse|atexit|codecs|ctypes|platform|shutil|traceback|codecs|chardet|lz4|win32api|win32com|yaml|tempfile|Tkinter|collections|copy|csv|copy|datetime|stat|string|struct|textwrap|scandir|inspect|pkgutil|math|gettext|locale|msgfmt|pygettext|operator|webbrowser|array|zlib|binascii|_winreg|win32gui|importlib|random|toml|pytest|glob|logging|zipfile|pygit2|py2exe)
• open usages without false positives: (?<!webbrowser\.)(?<!io\.)(?<!def )\bopen\b

Some resources:

• Supporting Python 3: An in-depth guide - and by 'in-depth' this does mean in-depth
• Porting Python 2 Code to Python 3 - a general overview from python.org
• 2to3 - automated code porting tool

Added to 308, may be addressed sooner - there's lots of factors at play here. There is also no way that I caught every issue we'll encounter on the road to py3. Supersedes: #194 and #404 Follow-ups: #55

[Utumno]

It would at the moment unfortunately due to getattr war and 480-pt3 branches - I intend to use from future import unicode_litterals there to reduce the diff

On Tue, Jan 12, 2021, 1:26 AM Lojack notifications@github.com wrote:

Quick question: how disruptive would it be to go ahead and prefix all the strings in brec and games? Dunno how much conflict resolution that would cause.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/wrye-bash/wrye-bash/issues/460#issuecomment-758288248, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKNIV5A7YTHURFDWDZJU4LSZOCIXANCNFSM4IVAKWCA .

[lojack5]

Ok, will just wait then. Itching for the py3 upgrade.

[Infernio]

@wrye-bash/bashers I did another test run of wrye-bash/2to3 based on the latest nightly. The results are on xxx-inf-py3.

Notes from this run:

• ws_comma is a pure PEP8 fixer, excluded it: 02bf777c06ca35f2edc0bf45e9a3d6747226771c
• We recently started making use of auto-tuple unpacking: 1f4c8336855dc2aa65c98cb516f4d3d598e1a767. This was removed in PEP 3113 and hence leads to the unbelievably ugly code from that fixer. Apparently the best solution is to use local methods and unpack in there instead: https://stackoverflow.com/questions/21892989/what-is-the-good-python3-equivalent-for-auto-tuple-unpacking-in-lambda
• We need to rename that next variable in belt: 0fb969667ee2e500fc7688d89a91e92c449e2fe5
• Looks like there's only one usage of filter left: 0fea640caa96d1a5be534f447b9ab422c5bdbc8f
• Good checklist of places where we should use iteritems/viewitems/etc.: 3167dc80fb162655258b5138b79d11e726bad95f
• The idioms fixer is fine to run on py3 upgrade now, but we should not move its changes back because they'd break the CIStr fixes we just made: cfa8770df0be3bfef8bbe97068c5410e50af097d
• There's one place where we use isinstance(foo, int) or isinstance(foo, long) instead of isinstance(foo, (int, long)) - this makes the isinstance fixer not follow up on the long fixer in that spot: 780b7e49c82df620abd78c70b2a3c9e46d638a9d and 7c923ad9c2f3f2bb0c0dd9734932d0c787e77ea7
• mod_files.py needs a from __future__ import print_function: bab7cb8606d1902df52e8bffa7a962a058fc12ea
• Good checklist for places where we should use izip: e3882fe298328398c222acdb56b8e24faca301cc
• Not sure about this one: be476e6a686ec9d96e31f473cd75db1e2fc2f503 - seems like we should maybe just drop those try statements, the fixer seems to handle it fine

[Infernio]

I'll try to tackle a few of the lower-hanging ones in there today, unless someone is faster than me :P

[Utumno]

Haha won't be faster than you - in the parsers if you can extract a method like I said here 23e53e66c114f67817f27d954816180ccf96bafe

[Infernio]

First round of changes/fixups done, 2to3 result is now much cleaner (also at <800 lines worth of changes).

[Utumno]

Thanks Inf - forget about _key_sort will do (conflicts with as yet unmerged code)

[Infernio]

Full log of 2to3 output is here: https://gist.github.com/Infernio/6cc3517ed869baf1615c36d3e61776ff Only interesting bit is this:

RefactoringTool: Warnings/messages while refactoring:
RefactoringTool: ### In file /mnt/e/Infernio/Desktop/Programming/wrye-bash/Mopy/bash/bosh/_saves.py ###
RefactoringTool: Line 98: You should use 'operator.mul(('=Ib', 5), num)' here.
RefactoringTool: Line 108: You should use 'operator.mul(('=Bf', 5), num)' here.

Lines in question:

self.factions = list(starmap(_unpack, repeat(('=Ib', 5), num)))
# ...
self.modifiers = list(starmap(_unpack, repeat(('=Bf', 5), num)))

No idea what that's about, I've never even heard of starmap 😅 The name is very cool though :P

[Infernio]

Another surprise: 2to3 does not replace __unicode__ with __str__. I'll add it to our 2to3 fork.

Edit: on a happy note, though:

localize.py   83 setup_locale: No translation file for language 'en-GB', falling back to English
localize.py   86 setup_locale: Set wx locale to 'en' (en_GB)
localize.py   95 setup_locale: Set Wrye Bash locale to 'English'
pdfviewer using PyMuPDF (GPL)
bash.py  231 dump_environment: Using Wrye Bash Version 308
bash.py  231 dump_environment: OS info: Windows-10-10.0.19041-SP0, running on Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
bash.py  231 dump_environment: Python version: 3.9.1 (tags/v3.9.1:1e5d33e, Dec  7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)]
bash.py  231 dump_environment: wxPython version: 4.1.1 msw (phoenix) wxWidgets 3.1.5
bash.py  231 dump_environment: python-lz4 version: 3.1.3; bundled LZ4 version: 1.9.3
bash.py  231 dump_environment: pyyaml version: 5.4
bash.py  231 dump_environment: Input encoding: utf-8; output encoding: utf-8
bash.py  231 dump_environment: Filesystem encoding: utf-8

I've gotten to the point where the game selection shows properly :)

[Infernio]

As of d4aa8bf4ca763b8dc6a4e2bbb2d605fd700322b6, tests work on py3.

Some important notes from this work:

• StringIO. We're using it sometimes with bytes, sometimes with unicode. It's meant for unicode only, py2 just doesn't enforce that. Which leads to it blowing up on py3 of course. So we need to change StringIO usages that actually need BytesIO to that - thankfully BytesIO is available on py2.
• bytes on py2 - 2to3 leaves str alone on upgrade, which leads to bolt.encode immediately breaking because it will still check for str instead of bytes. Thankfully the bytes alias is available on py2 and 2to3 respects it, so we should change all str usages to unicode or bytes. Prefixing isn't enough ;)
• There are a bunch of deprecation warnings issued for invalid escape sequences, e.g. in docstrings. We can easily fix these with backwards-compatible commits.

[Infernio]

@Utumno I'm currently going through the StringIO usages, will probably return to work on more 2to3 fixers after that.

[Infernio]

Nice, the StringIO from io actually has the same behavior as the py3 one:

PS C:\Users\Infernio> py -2
Python 2.7.18 (v2.7.18:8d21aa21f2, Apr 20 2020, 13:25:05) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import StringIO
>>> import io
>>> a = StringIO.StringIO()
>>> b = io.StringIO()
>>> a.write(u'foo')
>>> b.write(u'foo')
3L
>>> a.write(b'foo')
>>> b.write(b'foo')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unicode argument expected, got 'str'

That should really help catch wrong usages.

[Infernio]

It's running 🎉 Granted, all my settings don't work and the tabs are in the wrong order :P

[Utumno]

WOW

Yep settings backwards and forward and up and down compat...

Edit:

RefactoringTool: Line 98: You should use 'operator.mul(('=Ib', 5), num)' here.

False positive https://stackoverflow.com/a/43630024/281545

[Infernio]

I just built the first py3 Bashed Patch 🎉

It was an empty BP in FO3, but still :P

[Utumno]

rebasing

[Utumno]

Took all day - had to redo 27e65c64535a9b16d62208771c3863b6c55fe720 and al - I rebased your streak of insta-classic commits up till here d8c9caa629a5d772cfce03dbee7d4f2f6fb65d67 - dev ready not even much squashing needed (addressed some of the issues - diff with old nightly head). Will have to review rest of nightly

[Infernio]

Thanks :) Still can't believe we're this close to getting it to run 😅

Edit: I went to pull your changes to dev in, and git gc'd in the background - took ~5 minutes. Turns out my .git folder had grown to ~300MB :P

[lojack5]

Tuple parameter unpacking we only use in lambdas, and 2to3 handles that so I marked it as complete (unless we don't for some reason plan on running that fixer?)

[Infernio]

I thought we'd gotten rid of it completely as of one of @Utumno's commits?

Edit: the reason the point was there is that the code 2to3 generates for it is incredibly ugly. See e.g. 1f4c8336855dc2aa65c98cb516f4d3d598e1a767.

[Utumno]

Oh yes there should be none left - hey is the Ctrl-c shortcut working for you (copy selected infos paths to clipboard)? for me returns 3 as the code: 26379dc7e93af903feae141d56f00ed842e4d5f6

[Infernio]

Just tested. With that commit it does not work for me, so I reverted it locally. After doing that, it works for me. I get code 67 (which is ord(u'C')).

[Utumno]

Thanks Inf so that's something in my env (some autohotkey script probably) - great so will omit on next iteration

[lojack5]

Gone? I must have been looking at a stale branch then.

[Infernio]

Easiest way to test is to run the fixer from 2to3 and check git diff afterwards.

[Infernio]

I reran 2to3 on xxx-inf-2to3 and started rebasing xxx-inf-py3. Nearly everything conflicts :/

Edit: done, but haven't tested it yet.

[Utumno]

Great job - will try and finish parsers the next few days and then 460

I don't even want to think of the rebase of 480/543 :P

[Infernio]

Do we need another stable py2 release, or can we do the VDATA3 upgrade (i.e. decoding all bytestring settings) on py3 in the same release?

[Utumno]

I'd have to think a bit about that - will do when I visit the 460 branches

On Tue, Mar 30, 2021 at 1:39 PM Infernio @.***> wrote:

Do we need another stable py2 release, or can we do the VDATA3 upgrade (i.e. decoding all bytestring settings) on py3 in the same release?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wrye-bash/wrye-bash/issues/460#issuecomment-810115109, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKNIV2DWNCIAU3M7DTATFTTGGS5VANCNFSM4IVAKWCA .

[Infernio]

Rebased my py3 branches.

[Infernio]

b0b85345537e287a4add2e6647c23692b749d3c3 is done, this is the last thing I wanted to do to make 2to3 less noisy/more helpful.

[Infernio]

Archives handling needs work:

bain.py 1356 _refreshSource: Unable to read archive 'G:\steam\steamapps\common\Fallout3 Mods\Bash Installers\FOOK2 v1-2 Open Beta Main Files Hotfixed 20140427-4448-1-2.7z'.
Traceback (most recent call last):
  File "E:\Infernio\Desktop\Programming\wrye-bash\Mopy\bash\bosh\bain.py", line 1352, in _refreshSource
    list_archive(tempArch, _parse_archive_line)
  File "E:\Infernio\Desktop\Programming\wrye-bash\Mopy\bash\archives.py", line 174, in list_archive
    maList = __reList.match(line)
TypeError: cannot use a string pattern on a bytes-like object

Pretty sure we're calling 7z via subprocess without decoding the output...

[Infernio]

BAIN works now:

...and the first real Python 3 BP:

[Infernio]

Incredibly enough, there don't seem to be any regressions in that BP. And py3 seems to be a decent bit faster at building the BP. 54s vs 71s for that FO3 BP up there.

[Infernio]

@Utumno I think we should do the VDATA3 upgrade on one last stable py2 release.

Pros:

• Doing it this way definitely works and is much lower risk than trying to do both at the same time.
• If we do it this way, we can reject anything that isn't in VDATA3 format once we're on py3 - which means we can drop every single backwards compatibility hack right then along with all py2 weirdness. Otherwise we'd need a VDATA4 for dropping backwards compatibility hacks.
• 309 obsoletes valda's FO3/FNV versions for good, which may be an important enough reason to release a stable.

Cons:

• Outside of the FO3/FNV change mentioned, 309 is pretty small.

If we do decide to do it now, we should take this opportunity to drop the __reduce__ overrides in bain.py and converters.py. They're there to make versions prior to 307 work with newer settings formats, but that's obviously pointless if we change the format version to VDATA3 anyways. That also means we'll be able to drop the hacky imports in bosh/__init__.py once we are on py3 since we'll require VDATA3 at that point.

[Utumno]

Let me have a closer look at VDATA3 - 309 being small is not really an issue - and we did shuffle a lot the code so yeah

On Sat, May 1, 2021, 1:57 PM Infernio @.***> wrote:

@Utumno https://github.com/Utumno I think we should do the VDATA3 upgrade on one last stable py2 release.

Pros:

• Doing it this way definitely works and is much lower risk than trying to do both at the same time.
• If we do it this way, we can reject anything that isn't in VDATA3 format once we're on py3 - which means we can drop every single backwards compatibility hack right then along with all py2 weirdness. Otherwise we'd need a VDATA4 for dropping backwards compatibility hacks.
• 309 obsoletes valda's FO3/FNV versions for good, which may be an important enough reason to release a stable.

Cons:

• Outside of the FO3/FNV change mentioned, 309 is pretty small.

If we do decide to do it now, we should take this opportunity to drop the reduce overrides in bain.py and converters.py. They're there to make versions prior to 307 work with newer settings formats, but that's obviously pointless if we change the format version to VDATA3 anyways. That also means we'll be able to drop the hacky imports in bosh/init.py once we are on py3 since we'll require VDATA3 at that point.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wrye-bash/wrye-bash/issues/460#issuecomment-830612666, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKNIV54WWUBPFXOHWIIQNLTLPNBTANCNFSM4IVAKWCA .

[Utumno]

Otherwise we'd need a VDATA4

I am afraid we need a VDATA4 for #543 - see also __drop_paths in 6f0a5bce24999de1855fb7178af0685e4516993c (broken generally). But I don't mind a VDATA4 tbh as 543 is difficult to finalize now and getting rid of some of the cruft is a good idea anyways. Then a 309 beta with the settings changes to catch regressions and then py3 on 310 first thing?

In a7f68b5e900daeca8770a3c8751e9f134b24ea47 do not save the settings however - this should be taken care of in normal exit.

[Infernio]

IIRC I made it save settings because otherwise it won't convert a couple settings files because it checks if they've changed. I don't recall which ones those were, but I want to say converters and the like?

[Infernio]

Also, I just realized that the VDATA3 commit will break defaultdicts because it doesn't pass the factory into the constructor. Do we pickle any defaultdicts?

[Utumno]

Hmmm - good question - a "solution" would be to copy the dict then clear it and update from the (converted) copy

Unrelated but... do not drop ordered dicts if you thought about that - py3 ones are ordered but if order matters to us we should name accordingly - ordered_xxx - then drop (so after 460)

I discovered some regressions in parsers (from Rounder mostly) that I added on nightly. Takes a lot of cruft for the ride :)

[Infernio]

I'll have to do some testing. And yeah, doing anything with relying on dict insertion order needs thought wrt. naming, typing etc.

[Utumno]

New fixups in parsers - I am confident I caught most bugs in there now plus quite a few code gone. Intend to merge next round - seems maintenance is ongoing on your front too :P

Re: VDATA3 let's go but in a way VDATA4 (CIStr keys) is as easy as possible - needs some thought

[Infernio]

seems maintenance is ongoing on your front too :P

All of those were born from trying to update the mods in my SSE load order to their latest versions. Added like 10 future nice-to-haves to my TODO list too :P

[Infernio]

I've rebased my py3 branch. Left 68c3921c3 out because it conflicted hard and I don't really know what the code around it is doing anymore :P

[Infernio]

I've also done a bit of testing on VDATA3 now:

• We don't seem to pickle any defaultdicts, so that's good.
• The conversion went smoothly for all the games I tried (Enderal SE, FO3, FO4), but I don't have a very complicated setup for any of those games. I want to test it with my large SSE setup as well.
• My Enderal SE setup was brand new, but still had bytestrings in it. Mostly due to using self.__class__.__name__ in gui and writing bytestrings into BP configs. Should be fixed as of 8371b6a31c76232fbfb8cba7dcd912d794dc1176.

[Infernio]

Got a lot of py3 testing done today. Pretty much everything seems to work now 🎉 One thing to watch out for is 7c5114632d906d29eb30115b8a0cdb5e9a47c6c9 - there may be a couple more of these bound method problems hiding in the codebase.

[Utumno]

I've rebased my py3 branch. Left 68c3921 out because it conflicted hard and I don't really know what the code around it is doing anymore :P

No worries will get to that

I've also done a bit of testing on VDATA3 now:

Great job!

[Infernio]

nightly-vdata3 has the rebased VDATA3 commit.

[Utumno]

I'd say we bit the bullet with VDATA3 - that is have 309 out. However we shouldn't drop compatibility with VDATA2 unless well into py3 - there will be people upgrading from 308 to 310 - let's drop VDATA2 in 311?

Meanwhile parsers were broken in py3 - csv.reader is unicode now (duh) - see 1e85d6e44975d9543173aa5c46ce36b180a72dac - let me rebase a bit :)

[Infernio]

I don't know if doing the upgrade on py3 is possible. Maybe with encoding='bytes' and running conv_obj on it, but I'd have to test it first.

On 21. May 2021, at 11:23, Utumno @.***> wrote:

 I'd say we bit the bullet with VDATA3 - that is have 309 out. However we shouldn't drop compatibility with VDATA2 unless well into py3 - there will be people upgrading from 308 to 310 - let's drop VDATA2 in 311?

Meanwhile parsers were broken in py3 - csv.reader is unicode now (duh) - see 1e85d6e - let me rebase a bit :)

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe.