fuzzing: Failing cases found with pysource-codegen

[pawamoy]

https://github.com/15r10nk/pysource-codegen :rocket:

FAILED tests/test_visitor.py::test_visit_arbitrary_code[115011] - AttributeError: 'str' object has no attribute 'canonical_path'
FAILED tests/test_visitor.py::test_visit_arbitrary_code[184211] - SystemError: compiler_lookup_arg(name='name_5') with reftype=3 failed in <generic parameters of name_5>; freevars of code name_5: ('.type_params', 'name_0', 'name_2', 'name_3', 'name_...
FAILED tests/test_visitor.py::test_visit_arbitrary_code[904162] - AttributeError: 'str' object has no attribute 'canonical_path'
FAILED tests/test_visitor.py::test_visit_arbitrary_code[621041] - SystemError: compiler_lookup_arg(name='name_5') with reftype=3 failed in <generic parameters of name_5>; freevars of code name_5: ('.type_params', 'name_1', 'name_2', 'name_5')
FAILED tests/test_visitor.py::test_visit_arbitrary_code[243509] - SystemError: compiler_lookup_arg(name='name_0') with reftype=2 failed in <generic parameters of name_5>; freevars of code name_5: ('.type_params', 'name_0', 'name_3', 'name_4')
FAILED tests/test_visitor.py::test_visit_arbitrary_code[402833] - AttributeError: 'str' object has no attribute 'canonical_path'
FAILED tests/test_visitor.py::test_visit_arbitrary_code[451286] - AttributeError: 'str' object has no attribute 'canonical_path'
FAILED tests/test_visitor.py::test_visit_arbitrary_code[79708] - SystemError: compiler_lookup_arg(name='name_0') with reftype=3 failed in <generic parameters of name_3>; freevars of code name_3: ('.type_params', 'name_0')
FAILED tests/test_visitor.py::test_visit_arbitrary_code[874497] - AttributeError: 'str' object has no attribute 'canonical_path'
FAILED tests/test_visitor.py::test_visit_arbitrary_code[920242] - SystemError: compiler_lookup_arg(name='name_2') with reftype=3 failed in <generic parameters of name_0>; freevars of code name_0: ('.type_params', 'name_1', 'name_2', 'name_4', 'name_5')
FAILED tests/test_visitor.py::test_visit_arbitrary_code[83616] - SystemError: compiler_lookup_arg(name='name_1') with reftype=3 failed in <generic parameters of name_1>; freevars of code name_1: ('name_1',)
FAILED tests/test_visitor.py::test_visit_arbitrary_code[76884] - AttributeError: 'str' object has no attribute 'canonical_path'
FAILED tests/test_visitor.py::test_visit_arbitrary_code[217640] - AttributeError: 'str' object has no attribute 'canonical_path'
FAILED tests/test_visitor.py::test_visit_arbitrary_code[825412] - SystemError: compiler_lookup_arg(name='name_3') with reftype=3 failed in <generic parameters of name_5>; freevars of code name_5: ('name_3',)
FAILED tests/test_visitor.py::test_visit_arbitrary_code[847978] - SystemError: compiler_lookup_arg(name='name_0') with reftype=3 failed in <generic parameters of name_2>; freevars of code name_2: ('.type_params', 'name_0', 'name_5')
FAILED tests/test_visitor.py::test_visit_arbitrary_code[614548] - SystemError: compiler_lookup_arg(name='name_1') with reftype=3 failed in <generic parameters of name_5>; freevars of code name_5: ('.type_params', 'name_1', 'name_3')
FAILED tests/test_visitor.py::test_visit_arbitrary_code[596299] - TypeError: 'Name' object is not subscriptable
FAILED tests/test_visitor.py::test_visit_arbitrary_code[545929] - AttributeError: 'str' object has no attribute 'canonical_path'
FAILED tests/test_visitor.py::test_visit_arbitrary_code[767116] - SystemError: compiler_lookup_arg(name='name_4') with reftype=3 failed in <generic parameters of name_3>; freevars of code name_3: ('.type_params', 'name_0', 'name_2', 'name_4')
FAILED tests/test_visitor.py::test_visit_arbitrary_code[216452] - AttributeError: 'str' object has no attribute 'canonical_path'
FAILED tests/test_visitor.py::test_visit_arbitrary_code[500566] - AttributeError: 'str' object has no attribute 'canonical_path'
FAILED tests/test_visitor.py::test_visit_arbitrary_code[893519] - SystemError: compiler_lookup_arg(name='name_0') with reftype=3 failed in <generic parameters of name_4>; freevars of code name_4: ('.type_params', 'name_0', 'name_1', 'name_3', 'name_...
FAILED tests/test_visitor.py::test_visit_arbitrary_code[288102] - SystemError: compiler_lookup_arg(name='name_5') with reftype=3 failed in <generic parameters of name_0>; freevars of code name_0: ('name_5',)
FAILED tests/test_visitor.py::test_visit_arbitrary_code[755094] - SystemError: compiler_lookup_arg(name='name_2') with reftype=3 failed in <generic parameters of name_2>; freevars of code name_2: ('name_2',)
FAILED tests/test_visitor.py::test_visit_arbitrary_code[530575] - TypeError: 'Name' object is not subscriptable
FAILED tests/test_visitor.py::test_visit_arbitrary_code[283245] - AttributeError: 'NoneType' object has no attribute 'canonical_path'
FAILED tests/test_visitor.py::test_visit_arbitrary_code[167320] - SystemError: compiler_lookup_arg(name='name_1') with reftype=3 failed in <generic parameters of name_3>; freevars of code name_3: ('.type_params', 'name_1')
FAILED tests/test_visitor.py::test_visit_arbitrary_code[103847] - AttributeError: 'str' object has no attribute 'canonical_path'

We should reuse these seeds to investigate and fix any legitimate issue. At least we should catch the errors and log debug messages. @15r10nk recommends we minimize the generated code with pysource-minimize before debugging.

[pawamoy]

The SystemErrors seem to come from pysource_codegen itself:

Failing case with seed 538993
Traceback (most recent call last):
  File "/media/data/dev/griffe/duties.py", line 325, in test_seed
    visit("module", filepath=Path("/tmp/module.py"), code=generate(seed))
                                                          ^^^^^^^^^^^^^^
  File "/media/data/dev/griffe/__pypackages__/3.12/lib/pysource_codegen/_codegen.py", line 930, in generate
    tree = generator.generate(root_node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/data/dev/griffe/__pypackages__/3.12/lib/pysource_codegen/_codegen.py", line 886, in generate
    result = fix(result, parents)
             ^^^^^^^^^^^^^^^^^^^^
  File "/media/data/dev/griffe/__pypackages__/3.12/lib/pysource_codegen/_codegen.py", line 752, in fix
    compile(code, "<string>", "exec", dont_inherit=True)
SystemError: compiler_lookup_arg(name='name_1') with reftype=3 failed in <generic parameters of name_0>; freevars of code name_0: ('.type_params', 'name_1', 'name_3', 'name_4')

We will ignore them for now.

[pawamoy]

Erm, it looks like I can't regenerate the same code when debugging through VSCode... :face_exhaling:

[pawamoy]

Oh, OK, it actually depended on the Python version used.

[15r10nk]

Yes, it depends on the python version. Are you still using a 3.12 pre-release? This errors looking familiar to me. https://github.com/python/cpython/issues/109219

I put an quick implementation together based on your tweet which finds some errors #216. the pysource-minimize part is the important part here which gives you this nice small source codes.

Let me know how you want to integrate it. You can also take it from here if you want.

[pawamoy]

Yes I'm still using Python 3.12.0rc1, good catch! Will upgrade :slightly_smiling_face:

[pawamoy]

I eventually made use of pysource_minimize, thank you again for showing me how to use it :slightly_smiling_face: