search

CensoredUsername / unrpyc

A ren'py script decompiler
Other
861 stars 157 forks source link

[Enhancement] Python 3 | Ren'Py 8.x support #92

Closed madeddy closed 7 months ago

madeddy commented 4 years ago

RenPy's next bigger release will no longer support python 2 in the near future and very possible py3.8. It would be awesome if this little app could get updated to be compatible with python 3.6+.

If any help is needed, say it. As far as it my (limited) coding ability allows i am willing to help.

CensoredUsername commented 4 years ago

My plan is to tackle that problem when that release actually happens, as it's been on the agenda for a long time now. At that point I'll branch a final version for python 2.x games off and switch the development to purely Python 3. Some parts of the decompiler are already compatible with it, like the deserialization code. Then it will be a few unicode-aware fixes in file reading/writing and finding any remaining bugs, but outside of that I don't expect any major pain points.

It is not my intention that the python 3 version of unrpyc is compatible with python 2 games. This would take a lot of headache as ren'py used to be a bit arbitrary about if some things were bytes or unicode. It'd also allow us to get rid of some cruft that's in there for just supporting really old games.

madeddy commented 4 years ago

I thought as much. Good to hear. My view of this could be wrong, however i see the task at hand as two-part and possibly separated: 1) In which py version do i want to run the app? 2) Which version of 'rpy' do i want to decompile? Question if they really can be separated or if some code affects both? My thought was, task 1) could be earlier done but i don't know the internals of unrpyc.

CensoredUsername commented 4 years ago

1 and 2 are indeed possibly separate, however to support things like un.rpyc and other tools (😉) that rely on ren'py's own python version to run it is important that the tool is written in whatever version of python it targets itself.

madeddy commented 4 years ago

Oh, yeah. "un.rpyc" dependence. There was something... :)

Let's see when we get RenPy 8. Could still take some time.

madeddy commented 3 years ago

Ok. We did actually go ahead with a try at this:

I did try the port in spring '20 but got stuck. A user at "F96 - 1" try'd his hands last weeks on a port and managed at least to resolve the error which prevented myself so far. This allowed us finally to go much further with the porting.

In short, I believe we have it working in native Py3. The main module at least(Means everything but un.rpyc). Some info (line numbers relate to my forks "py3 port" branch):

  1. I put it through the 2to3 module, and checked then every change.
  2. Wrapings of all dict use cases in a list call i consider overzealous. I see no value in it. https://docs.python.org/3.9/library/2to3.html?highlight=dict#2to3fixer-dict (last set) Reverted.
  3. 2to3 put additional parens around printstatements so they where doubled. Reverted.
  4. About the change of filter use cases to a generator i am not sure. Both work. I let it stay.
  5. Biggest problem was the change in SL2decompiler from line 313 onward. There we have a in py3 forbidden int/NoneTypecomparison and i got probably a bit overzealous about substituting the possible None cases of the two compare sides. The one side of the comparision needs it not said the other guy.
  6. Another problem was in magic(the error src which i not found) in line 596: The original encoding was bytes and don't works anymore in py3. I followed the code and found it goes in the end in the "pickle module", which allows (py doc)just bytes or ascii.(i think it ends here: https://github.com/python/cpython/blob/cd8dcbc851fcc312722cdb5544c2f25cf46b3f8a/Lib/pickle.py#L1321) The other user replaced with utf-8 and it works also. I am still not sure its, right and shouldn be ascii...
  7. I made no other non essential changes to hold it small and readable. (My fingers wanted though...)
  8. I needed also to put some# nopep8 # noqa comments in to shut my linter up and prevent the autoformater to move some import statements up where they don't belong.
  9. The un.rpyc submodule was naturally not tested. Blame RenPyTom for still trying to ride a dead horse. 😭

If time allows, it would be nice if you (also @jackmcbarn) look it over and test. I am not far enough in py knowledge to be sure about some code decisions and do really not want all this for naught.

My halfway or even better working py3 port: https://github.com/madeddy/unrpyc/tree/py3_v1.1.8

Have fun! 🚀

EwenQuim commented 2 years ago

I tried @madeddy unrpyc.py for python3 in order to test https://github.com/EwenQuim/Renpy-graphviz (sorry for the ad haha), and it worked very well with 8 different games! 🎉

I think some feedback might give some boost to this PR (time to move on!). The Python 3 version of this tool helped me a lot as I only have Python 3 and do not want to install Python 2, so you deserve a lot of respect for the work. Thank you and thanks to @CensoredUsername too :)

Dobby233Liu commented 2 years ago

Since there's ad on the top i will ad too, here's my bad edit of the py3 port https://github.com/Dobby233Liu/unrpyc

Dobby233Liu commented 2 years ago

RenPy's next bigger release

Just coming to say that Ren'Py 8 is officially released: https://www.renpy.org/release/8.0.0

ooofest commented 2 years ago

I tried @madeddy 's Python 3 version and it generally worked on most .rpyc files, but I did encounter some files where it broke where the Python 2 version could otherwise decompile the same files (only after adding @sigio 's changes https://github.com/CensoredUsername/unrpyc/pull/154 )

This is the error dump when a problem was encountered and I can provide the source rpyc if needed - wondering if anyone has ideas on what I might try for resolving this issue for some files being processed:

Traceback (most recent call last): File "c:\game\unrpyc.py", line 200, in worker return decompile_rpyc(filename, args.clobber, args.dump, decompile_python=args.decompile_python, File "c:\game\unrpyc.py", line 164, in decompile_rpyc ast = read_ast_from_file(in_file) File "c:\game\unrpyc.py", line 137, in read_ast_from_file data, stmts = magic.safe_loads(raw_contents, class_factory, {"_ast", "collections"}) File "c:\game\decompiler\magic.py", line 601, in safe_loads return SafeUnpickler(StringIO(string), class_factory, safe_modules, use_copyreg, File "/home/tom/ab/renpy-build/tmp/install.linux-x86_64/lib/python3.9/pickle.py", line 1212, in load File "/home/tom/ab/renpy-build/tmp/install.linux-x86_64/lib/python3.9/pickle.py", line 1589, in load_reduce File "c:\game\decompiler\magic.py", line 113, in new raise FakeUnpicklingError("{0} was instantiated with unexpected arguments {1}, {2}".format(cls, args, kwargs)) decompiler.magic.FakeUnpicklingError: <class 'builtin.set'> was instantiated with unexpected arguments ([],), {}

I did try adding the same changes from https://github.com/CensoredUsername/unrpyc/pull/154 mentioned above to the Python 3 version, but no difference in result.

madeddy commented 1 year ago

Ok. Since Ren'Py 8 and 7.5 for way over 4 months out is and current unrpyc with these versions broken is, i want to poke this again. :innocent: Are there plans and time available, on the main devs side, to tackle this in the near future?

Don't forget i did last year a py3 attempt which seamed to work up to Ren'py 7.4. I just don't know how helpful this for the new situation is. (linked 5 posts above) Greets

CensoredUsername commented 1 year ago

Real life has sadly been quite busy, I'll probably only get to this next year.

madeddy commented 1 year ago

Understandable, this happens to all of us i think. All the same, sometimes i have no desire to tackle some code, other times its big fun. 😁

I have the Python 3 port #159 anyway successful running, also the Ren'Py 8.x + 7.5.x fixes (pull follows) from here do their work greatly.

Best regards and take care. 👋🏻

sebastic commented 1 year ago

Debian has removed Python 2 from the upcoming bookworm stable release increasing the need for Python 3 support.

Baerbeisser commented 1 year ago

How about it? Do you feel up to it again?

CensoredUsername commented 8 months ago

Aight, time to get back into this.

The plan is as follows. I'll first bring the current unrpyc up to date to support up to ren'py 7.5. This will then become the legacy branch, requiring python 2.7, and presenting the various injection techniques for games up to ren'py 7.5.

Then, development on master will switch to targetting ren'py 8.0 and above, running with python 3. This will allow a significant amount of legacy code to be trimmed that is needed for support of older ren'py versions, greatly simplifying the codebase that has become quite unwieldly due to having to support all previous ren'py versions. I'll first remove most of this unnecessary code, and then start porting to Python 3, likely assisted by work already done by the rest of you.

Note: this does mean that the new codebase will not support anything below 8.0. You'll need the legacy branch for that with python 2 still. Due to how ren'py's data storage system is intertwined with basic python types, supporting older releases while running in python 3 adds significant unnecessary complexity. And the codebase has suffered enough from older ren'py's incredible inconsistency in use of str/unicode types which is guaranteed to make any effort of porting that while supporting all previous versions absolute hell. I'd like to just close all that cruft out and use this opportunity of a breaking change to remove all this cruft.

madeddy commented 8 months ago

Hey! Nice to have you back and hope you're well.

The plan is as follows. ...

Basically i figured the same. Seemed logical and oportune to have a nice cut at py3. Feel free to poke me about anything related to my fixes, pulls req, fork branches. I have also some fixes still locally and a unrpyc experimental repo.

I'll first bring the current unrpyc up to date to support up to ren'py 7.5.

I would like to point out that Ren'Py py2 is now going up to v7.7.x(most likely the last v7.x) and will end there in may (official) this year. They stated this will mark also a clean-out of their codebase.

... development will switch to targeting ren'py 8.0 and above... This will allow a significant amount of legacy code to be trimmed... use this opportunity of a breaking change to remove all this cruft.

Absolute understandably for me. I have often problems to read/follow your code from all the legacy parts and also from the complexity of things with renpy.

For consideration:

Ok, hope this helps a bit. Greets

CensoredUsername commented 8 months ago

@madeddy thanks for the summary!

CensoredUsername commented 7 months ago

I do believe that all the 7.5 issues have now been triaged, all with testcases are marked as possibly resolved. SLCustomUse is the only one I haven't bothered fixing, it is just so unlikely that anyone did use that and not made the jump to 8.0. Only thing I might tackle before the move to python 3 is ripping out atl handling from the main decompiler into a subdecompiler for sanity reasons. But the time is nearly there!

CensoredUsername commented 7 months ago

I've performed the refactoring I wanted to do, so I'm going to do a version bump, edit the readme, and cut a release.

madeddy commented 7 months ago

Congrats. I like the way with the new Options class. Good idea.

CensoredUsername commented 7 months ago

🎉 It has landed 🎉

Apologies for the delays, but we have unrpyc.py and un.rpyc fully working in python 3 now!

Please report any issues with it in the issue tracker, but in the meanwhile, and after almost 4 years I'd like to close this issue.