python3 porting based on v1.1.8

[madeddy]

• works since ~April 2021 on apps with RenPy engine up to version 7.4.11; later also as base for v8.0.x
• only issues are known and unrelated to the port like the "at transform" bug or warnings about "unknown user-statements" which are logical
• changes on the readme should be ignored/discarded (they're just for info about the branches)
• un.rpyc submodule does not work (no changes on it)

[madeddy]

Admittedly i did beforehand never test the un.rpyc submodule, because it was never my focus. Any file edits in there are from 2to3.py or manual obvious changes for py3 i spotted.

Two of the tests failed because i didn't bother to change the shebang and from a undiscovered unicode-str conversion bug. Both did in local tests never error.

[OdinVex]

Compilation attempts result in exec block issues because of Python's demand for including parenthesis.

Traceback (most recent call last):
  File "../Tools/Ren'Py/unrpyc-py3_port-v1.1.8/un.rpyc/compile.py", line 157, in <module>
    decompiler_rpyc = p.ExecTranspile(base + """
  File "../Tools/Ren'Py/unrpyc-py3_port-v1.1.8/picklemagic/pickleast.py", line 689, in ExecTranspile
    node = ast.parse(string, mode="exec")
  File "/usr/lib/python3.10/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 14
    exec '''
    ^^^^^^^
SyntaxError: Missing parentheses in call to 'exec'. Did you mean exec(...)?

Using python compile.py -d -o -p 1 -r -m ../picklemagic/. (It hates minimization at the moment.)

[madeddy]

I explained in both of my posts that the un.rpyc submodule does not work.

[OdinVex]

I explained in both of my posts that the un.rpyc submodule does not work.

I wasn't using submodules in this case, manual download and move. I get what you meant, then.

[zmook]

I would sincerely love to get my hands on these changes, since most games fail decompilation now with the existing master branch. I've tried cloning madeddy/unrpyc, but I cannot quite figure out how to merge the py3_port@v1.1.8 and the Renpy_8_7.5_compat_fix@v1.1.8 branches without getting conflicts. And I think we need both, yes? If CensoredUsername is not around to process these pull requests, could you either provide a merged branch in the madeddy fork, or notes on what I need to do myself?

[madeddy]

I would sincerely love to get my hands on these changes... ... but I cannot quite figure out how to merge the py3_port@v1.1.8 and the Renpy_8_7.5_compat_fix@v1.1.8 branches without getting conflicts. And I think we need both, yes?

You're correct, you need both parts for RenPy v8.0.x based apps. However for v7.5.x you need just the compat fix, because there is RenPy still on Python 2! Not sure about the new RenPy 7.6.x(Py2) and v8.1.x(Py3), there could be additional changes needed.(We will have to wait and see...)

As for your request, i did already complete branches of this code for RenPy 8/Py3: https://github.com/madeddy/unrpyc/tree/py3_v1.1.8 and the v1.2.0 also but has some extra (non necessary luxus) changes.

You should look at the readme's of the branches. Good luck! Greets

[eirnym]

I should say, that most pre-compiled renpy 7.x games works well on RenPy 8.2.x nighlty without any recompilation, Some Python modules obviously won't be working as they have Python2 bytecode. There's not many of them though

[eirnym]

@madeddy The decompiller for Python 3.x works well (1.2.0_dev at least) on Python 3.11 and RenPy 7.x and 8.x

I suggest following fixes (could be done in separate smaller PRs)

1. For Python3 (with or without if) to use dictinstead of OrderedDict as dict guarantees the order since Python 3.7
   • it's useful to minimise CPU and memory overhead of the OrderedDict if not needed.
2. Don't use processes if single file supplied.
   • it's often some resources has not been freed, and there's no point of the process pool anyway.
   • I'd just add an if statement just after file list
3. Don't write a file if decompilation/de-ast has been failed by default.
   • For a user it's better to have an original rpyc if processing failed and repeat this process if needed.
   • To achieve this it's possible to pass io.BytesIO instead of file and write when no errors or requested
   • Interactive mode and/or additional parameters could be an option
4. Add an option to skip decompilation if a decompiled file exists. Normal mode, without a warning.
   • proposed argument names: no-clobber or -C
5. Pass arguments by name if too many of them
   • example: call of decompile_rpyc in worker function
6. Some new renpy.ast objects are missing, like renpy.ast.Camera. There's not many new objects and they are not so popular, but some rpycfiles can't be decompiled because of this.

[Gouvernathor]

I suggest looking at https://github.com/Gouvernathor/unrpyc/tree/dev3 It's been some time since I worked on it or tried to use it, but I did a lot of tests at the time and made some advances.

[madeddy]

I suggest following fixes (could be done in separate smaller PRs)

Sry i missed last year somehow this post with the suggestions. Will look into it. Could you please move this into a separate issue? As this pull here is just the py3 port and your suggestions going more for extension/expansion. Thanks.

I suggest looking at https://github.com/Gouvernathor/unrpyc/tree/dev3

Yes, thank you. I discovered it already last year and used some of the code if that's ok with you.

[eirnym]

@madeddy yes, this would make sense after PR is merged

[eirnym]

Why PR has been closed?

[madeddy]

Why PR has been closed?

Good question. As i was it myself i can only deduct is was some backlash as i set my fork back and the connected branch disappeared accidentally.

I reopen it but its moot as the pull was to old to be integrated as a whole and the port is already done in dev branch.

[madeddy]

@eirnym Since the version for py3 almost ready is, you can perhaps offer your list again in GH diss if you want. NOTE: 6) is moot and 4) do we have already in unrpyc ... i think 🤔

[CensoredUsername]

We're up to date now, so this pull request is no longer necessary.