Errors in py3 test with recent updates from py2 branch

[madeddy]

Just so we have this already documented and when it's fresh and you can add it perhaps to a task list. Obviously i cannot guarantee i didn't make any mistakes which cause the errors.

I integrated all recent changes with the py3 port and did some tests with Ren'Py v8.2 and py3.10.

1. Arginfo code for double-/starred_indexes in util.py seems incompatible:

   Error while decompiling /home/olli/.xlib/RPG/_test/AttackOnSurveyCorps-0-16-0-pc/game/scenario/after_load.rpyc:
   Traceback (most recent call last):
   File "/home/olli/Code/Git/unrpyc_exp/unrpyc.py", line 160, in worker
   return decompile_rpyc(
   File "/home/olli/Code/Git/unrpyc_exp/unrpyc.py", line 129, in decompile_rpyc
   decompiler.pprint(
   File "/home/olli/Code/Git/unrpyc_exp/decompiler/__init__.py", line 58, in pprint
   translator=translator).dump(ast, indent_level, init_offset, tag_outside_block)
   File "/home/olli/Code/Git/unrpyc_exp/decompiler/__init__.py", line 140, in dump
   super().dump(ast, indent_level, skip_indent_until_write=True)
   File "/home/olli/Code/Git/unrpyc_exp/decompiler/util.py", line 30, in dump
   self.print_nodes(ast)
   File "/home/olli/Code/Git/unrpyc_exp/decompiler/util.py", line 125, in print_nodes
   self.print_node(node)
   File "/home/olli/Code/Git/unrpyc_exp/decompiler/__init__.py", line 161, in print_node
   self.dispatch.get(type(ast), type(self).print_unknown)(self, ast)
   File "/home/olli/Code/Git/unrpyc_exp/decompiler/__init__.py", line 574, in print_label
   self.print_nodes(ast.block, 1)
   File "/home/olli/Code/Git/unrpyc_exp/decompiler/util.py", line 125, in print_nodes
   self.print_node(node)
   File "/home/olli/Code/Git/unrpyc_exp/decompiler/__init__.py", line 161, in print_node
   self.dispatch.get(type(ast), type(self).print_unknown)(self, ast)
   File "/home/olli/Code/Git/unrpyc_exp/decompiler/__init__.py", line 599, in print_call
   words.append(reconstruct_arginfo(ast.arguments))
   File "/home/olli/Code/Git/unrpyc_exp/decompiler/util.py", line 312, in reconstruct_arginfo
   elif i in list(arginfo.starred_indexes):
   TypeError: 'set' object is not iterable
   
   Decompilation of 1 file failed.

   Really weird for a set, however if i've seen this right Ren'py uses possible the RevertableSet as the normal "set()". -> renpy/minstore.py

2. Case of wrong type compared in deobfuscate.py:

   Error while decompiling /home/olli/.xlib/RPG/_test/AttackOnSurveyCorps-0-16-0-pc/game/scenario/after_load.rpyc:
   Traceback (most recent call last):
   File "/home/olli/Code/Git/unrpyc_exp/unrpyc.py", line 160, in worker
   return decompile_rpyc(
   File "/home/olli/Code/Git/unrpyc_exp/unrpyc.py", line 118, in decompile_rpyc
   ast = deobfuscate.read_ast(in_file)
   File "/home/olli/Code/Git/unrpyc_exp/deobfuscate.py", line 321, in read_ast
   data, stmts, d = try_decrypt_section(raw_data)
   File "/home/olli/Code/Git/unrpyc_exp/deobfuscate.py", line 352, in try_decrypt_section
   newdata = decryptor(raw_data, count)
   File "/home/olli/Code/Git/unrpyc_exp/deobfuscate.py", line 205, in decrypt_hex
   if not all(i in "abcdefABCDEF0123456789" for i in count.keys()):
   File "/home/olli/Code/Git/unrpyc_exp/deobfuscate.py", line 205, in <genexpr>
   if not all(i in "abcdefABCDEF0123456789" for i in count.keys()):
   TypeError: 'in <string>' requires string as left operand, not int
   
   Decompilation of 1 file failed.

   after_load.zip

[CensoredUsername]

Really weird for a set, however if i've seen this right Ren'py uses possible the RevertableSet as the normal "set()". -> renpy/minstore.py

That's only applicable to any set used in rpy files, this ought to be a normal set, and therefore it should just work. So, very confusing. Not sure what is breaking there. Also what is the purpose of the list call around arginfo.starred_indexed? The whole point of a set is a fast contains check, while with a list it's O(n).

Case of wrong type compared in deobfuscate.py:

That is just py 3 porting residue.

[madeddy]

py 3 porting residue.

As thought.

what is the purpose of the list call

Sry, posted from the wrong try. I added it to try to get around the not iterable issue but didn work. I can assure you it errors the same just with your code.

Test possible with https://github.com/madeddy/unrpyc/tree/dev_py3

[madeddy]

I get a lot of errors in the style of e.g. (stdout cite)

Decompiling /home/.../classroom.rpym...
Unknown AST node: <class 'store.ATL.RawIf'>

We have these AST nodes already with leading "renpy.ast.xyz". Don't know if they're just renamed or something like a new namespace in renpy is going on...

Unknown AST nodes list:

• store.ATL.RawIf
• store.ATL.RawBlock
• store.ATL.RawChoice
• store.ATL.RawAction
• store.ATL.RawRepeat

These are the ones i encountered so far. Some are in Ren'Py atl.py but for others i have no idea from where they come. e.g. RawAction

I've taken the code solution to these from @Vepsrp, who maintains his own unrpyc variant:

# fake import
magic.fake_package("store")
import store  # nopep8 # noqa

# These two are just added to their counterparts
@dispatch(store.ATL.RawRepeat)
@dispatch(store.ATL.RawBlock)

# nearly a clone
@dispatch(store.ATL.RawChoice)
def print_atl_rawchoice2(self, ast):
    for loc, chance, block in ast.choices:
        self.indent()
        self.write("choice")
        if chance != "1.0":
            self.write(" %s" % chance)
        self.write(":")
        self.print_atl(block)
    if (self.index + 1 < len(self.block)
            and isinstance(self.block[self.index + 1], store.ATL.RawChoice)):
        self.indent()
        self.write("pass")

# newly written? (removed py2 code from first "if" condition)
@dispatch(renpy.atl.RawIf)
@dispatch(store.ATL.RawIf)
def print_atl_rawif(self, ast):
    statement = First("if %s:", "elif %s:")

    for i, (condition, block) in enumerate(ast.entries):

        if ((i > 0)
            and (i + 1) == len(ast.entries)
                and (not isinstance(condition, str)) or condition == 'True'):
            self.indent()
            self.write("else:")
        else:
            if (hasattr(condition, 'linenumber')):
                self.advance_to_line(condition.linenumber)
            self.indent()
            self.write(statement() % condition)
        self.print_nodes([block], 1)

@dispatch(renpy.atl.RawAction)
@dispatch(store.ATL.RawAction)
def print_atl_rawaction(self, ast):
    self.indent()
    self.write(ast.expr)

If the outcome of this implementation correct is... so far it works. Seemingly.

[CensoredUsername]

That is very odd. The store namespace is where game code normally resides. Are you sure someone didn't monkeypatch that whole module in their own game?

The current ren'py code still defines those things in renpy.atl, and pickles only denote the original instantiation location, and re-export doesn't change that. So alternative classes have been created in store, which is usually where user code hangs out.

[madeddy]

I can ask where this happens(app wise) besides "Attack On Survey Corps(AOSC)" where i did notice it. However AFAIK there where more apps over the last year or so. I add all scripts of AOSC, so you can test it. And yes, the engine is modified. AOSC.zip

Edit: So far also in "Innocent Witches". AFAIK this game is also on a "modified".

If this just in some games happens i see no need to support it. Only if there is some "general way" to write this out into .rpy.

[CensoredUsername]

Yeah Innocent witches has been known to do that. Just redefines completely new ast nodes in the game script. I'll look at that once we have python 3 running.

[madeddy]

Sounds good. Just wanted it noted somewhere.

[CensoredUsername]

Really weird for a set, however if i've seen this right Ren'py uses possible the RevertableSet as the normal "set()". -> renpy/minstore.py

Something weird is afoot. I started the python 3 transition, things work pretty well, except that we get weird set objects, because in the pickles the set object is recorded as a __builtin__.set object. Which is where it resides in python 2. In python 3 it resides in builtins.set. I'm digging through ren'py code atm to figure out where this might be occurring.

[madeddy]

Good to see i'm not THIS crazy. This set() bug stupped me really. I still think this code from minstore.py is weird regarding set()...

# L33> 
python_set = _set = set
...
# L46>
from renpy.revertable import RevertableSet as __renpy__set__
set = __renpy__set__ # @ReservedAssignment
Set = __renpy__set__

[CensoredUsername]

That code, as far as I know, is responsible for providing the default namespace for ren'py files, and when one constructs a set in a ren'py file, it needs to be a RevertableSet to participate in rollback.

That's a different thing. It only affects what set refers to in python code in a ren'py file. Not what set represents in the ren'py engine itself (any engine datastructures do not have to participate in rollback, only the game script has to). Now when users / the engine implement ast nodes / user statements / custom displayables in ren'py files, that kinda blurs the line as now Revertable containers can also end up in the ast.

The weird thing here is that set in the engine datastructures itself has its __module__ set to __builtin__. In Python 3.9 set.__module__ is set to builtins. There isn't even a __builtin__ module in Python 3.9. Python 3 has all builtins in the builtins module, that is normally present in all other module as the builtins variable. meanwhile, Python 2 has all builtins in the __builtin__ module, that is normally present in all other modules as the builtins variable.

I'm particularly confused because to test stuff I downloaded a fresh Ren'py 8.2.0, used that to recompile several scripts, and I still end up finding __builtin__.set in those. How the hell?

[CensoredUsername]

And the reason this doesn't happen to the other containers btw is because pickle special cases tuple, list, and dict to speed up their construction. set meanwhile just has to live with being treated like any other object (until pickle protocol 4, but ren'py keeps to protocol 2 as it is readable by both python 2 and 3, even when operating in ren'py 8 mode for now).

[madeddy]

Just a thought: Could this be from some py2 -> py3 change in python itself and has some relation to the fake classes in magic? Maybe something must be adapted in there to work again with py3 RenPy.

Oh and i read recently in some older issue talk(april'23?) from renpytom, how he now pulled the whole stdlib into RenPy. Maybe because of this changed something with v8.1.x ?

[CensoredUsername]

Could this be from some py2 -> py3 change in python itself

It is a py2 -> py3 change in python itself. The problem is, why isn't that change reflected in ren'py.

Also i found what causes it. It's caused by the fix_imports argument on pickle.dump, when running in protocol 2 it apparently tries to make the imports match the locations of those things in python 2.

[madeddy]

@VepsrP circumvents this in his unrpyc variant in some weird way. He has some _convertast func running and searches then even for double-/starred.. with dunders and also for __iter__

Maybe it helps to take a short look at this: https://github.com/VepsrP/UnRen-Gideon-mod-/blob/e80afed8b83129c6debf132a7b7d6283507bfa3f/decompiler/util.py#L181

[CensoredUsername]

As soon as I found what was causing it the fix was easy, I just added two proxy classes for them:

class oldset(set):
    __module__ = "__builtin__"
oldset.__name__ = "set"
SPECIAL_CLASSES.append(oldset)

class oldfrozenset(frozenset):
    __module__ = "__builtin__"
oldfrozenset.__name__ = "frozenset"
SPECIAL_CLASSES.append(oldfrozenset)

(I overwrite __name__ because I want to keep the actual set accessible normally)

[CensoredUsername]

@VepsrP circumvents this in his unrpyc variant in some weird way. He has some convert_ast func running and searches then even for double-/starred.. with dunders and also for iter

Man that is some incredible overkill lol.

[CensoredUsername]

Got the tutorial and the_question decompiling easily now. We also seem to be missing renpy.test.testast.Scroll. I suppose that one is new.

[madeddy]

Never seen this one. Weird. It's 7 years in, if my find is right: https://github.com/renpy/renpy/blame/1199e7defeb88933204044f6cb7e7b89a1d1dc6e/renpy/test/testparser.py#L144

Testast...testparser... how did you trigger this?

[CensoredUsername]

With a testcase statement! It is very well documented (translation: I couldn't even find documentation for it).

[madeddy]

😂 So much. Well, testing is maybe a insider job for them. Yeah, it's just in the files there.

[madeddy]

Man that is some incredible overkill lol.

I try'd to understand what convert_ast() does, but not sure. There are vals that are handled and not used and only one place where it writes(temp...) some decode() i think and its recursive. To which end... 🤷🏻

The thing is, he likes to integrate also custom problem solutions which affect in ever case just one game. I think this makes it hard to maintain.

Anyway. If the set() bug tackled is, we can close here.

[CensoredUsername]

I try'd to understand what convert_ast() does

It's rewriting the ast, to ensure that object keys are unicode (as they are in py3) and not bytestrings(as they are in py2).

But yeah, closing this one.