Genshi doesn't work on Python 3.11: code expected at least 18 arguments, got 16

[vstinner]

Example of error when running tests:

======================================================================
FAIL: OldTextTemplate (genshi.template.text)
Doctest: genshi.template.text.OldTextTemplate
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/doctest.py", line 2216, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for genshi.template.text.OldTextTemplate
  File "/home/vstinner/dev/genshi/genshi/template/text.py", line 239, in OldTextTemplate

----------------------------------------------------------------------
File "/home/vstinner/dev/genshi/genshi/template/text.py", line 244, in genshi.template.text.OldTextTemplate
Failed example:
    tmpl = OldTextTemplate('''Dear $name,

    We have the following items for you:
    #for item in items
     * $item
    #end

    All the best,
    Foobar''')
Exception raised:
    Traceback (most recent call last):
      File "/home/vstinner/python/main/Lib/doctest.py", line 1348, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest genshi.template.text.OldTextTemplate[0]>", line 1, in <module>
        tmpl = OldTextTemplate('''Dear $name,
      File "/home/vstinner/dev/genshi/genshi/template/base.py", line 421, in __init__
        self._stream = self._parse(source, encoding)
      File "/home/vstinner/dev/genshi/genshi/template/text.py", line 293, in _parse
        for kind, data, pos in interpolate(text, self.filepath, lineno,
      File "/home/vstinner/dev/genshi/genshi/template/interpolation.py", line 77, in interpolate
        expr = Expression(chunk.strip(), pos[0], pos[1],
      File "/home/vstinner/dev/genshi/genshi/template/eval.py", line 71, in __init__
        self.code = _compile(node, self.source, mode=self.mode,
      File "/home/vstinner/dev/genshi/genshi/template/eval.py", line 453, in _compile
        return build_code_chunk(code, filename, name, lineno)
      File "/home/vstinner/dev/genshi/genshi/compat.py", line 110, in build_code_chunk
        return CodeType(*params)
    TypeError: code() argument 15 must be bytes, not tuple
----------------------------------------------------------------------
File "/home/vstinner/dev/genshi/genshi/template/text.py", line 253, in genshi.template.text.OldTextTemplate
Failed example:
    print(tmpl.generate(name='Joe', items=[1, 2, 3]).render(encoding=None))
Exception raised:
    Traceback (most recent call last):
      File "/home/vstinner/python/main/Lib/doctest.py", line 1348, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest genshi.template.text.OldTextTemplate[1]>", line 1, in <module>
        print(tmpl.generate(name='Joe', items=[1, 2, 3]).render(encoding=None))
    NameError: name 'tmpl' is not defined

[gvanrossum]

While you're waiting for official docs, here's the new constructor: https://github.com/python/cpython/blob/18f41c04ff4161531f4d08631059fd3ed37c0218/Lib/test/test_code.py#L216-L232

[FelixSchwarz]

@vstinner Would you mind testing this again? In a23f3054b96b487215b04812c680075c5117470a I changed the code so we don't use the constructor anymore and maybe that also fixed the 3.11 problem. (Fedora has now 3.11 package yet so testing this is a bit more complicated for me.)

[vstinner]

@FelixSchwarz: I still get test errors, in another function:

======================================================================
ERROR: test_pickle (genshi.template.tests.eval.ExpressionTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/dev/genshi/genshi/template/tests/eval.py", line 45, in test_pickle
    unpickled = pickle.load(buf)
                ^^^^^^^^^^^^^^^^
  File "/home/vstinner/dev/genshi/genshi/template/eval.py", line 92, in __setstate__
    self.code = CodeType(0, *state['code'])
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: code expected at least 18 arguments, got 16

======================================================================
ERROR: test_pickle (genshi.template.tests.eval.SuiteTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/dev/genshi/genshi/template/tests/eval.py", line 551, in test_pickle
    unpickled = pickle.load(buf)
                ^^^^^^^^^^^^^^^^
  File "/home/vstinner/dev/genshi/genshi/template/eval.py", line 92, in __setstate__
    self.code = CodeType(0, *state['code'])
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: code expected at least 18 arguments, got 16

======================================================================
ERROR: test_pickle (genshi.template.tests.markup.MarkupTemplateTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/dev/genshi/genshi/template/tests/markup.py", line 52, in test_pickle
    unpickled = pickle.load(buf)
                ^^^^^^^^^^^^^^^^
  File "/home/vstinner/dev/genshi/genshi/template/eval.py", line 92, in __setstate__
    self.code = CodeType(0, *state['code'])
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: code expected at least 18 arguments, got 16

Commands:

$ ~/python/main/python -m venv env  # Python 3.11 (dev)
$ env/bin/python setup.py test

Maybe Code.__setstate__() should recompile the code using self.code = _compile(...) rather than serializing CodeType parameters.

[vstinner]

To build Python 3.11:

git clone https://github.com/python/cpython/
cd cpython
./configure
make

No need to install it: just use the ./python build in the root directory of the source code.

[FelixSchwarz]

Ah, that is the part I left open in #49 but @hodgestar is currently working on some fixes as part of #50 (for Python 3.10). Yeah, guess I'll have to set up a Python 3.11 then (though free time is pretty limited - as always :-).

Thank you for re-testing this.

[ztane]

Since I was doing this for my own templating engine Tonnikala and was just casually browsing around, I noticed that Genshi seems to have accidentally swapped co_kwonlyargcount and co_nlocals c.f. CPython help() for code objects...

[FelixSchwarz]

@ztane thank you for reporting this. I filed #52 so this won't get lost.

[FelixSchwarz]

@vstinner Thank you for your comment how to build Python 3.11 locally. It worked and I created PR #54 as a first step to make Genshi compatible with Python 3.11.

[vstinner]

You cannot use CodeType.replace() to avoid being broken each time CodeType constructor is changed?

[hodgestar]

@vstinner We do use CodeType.replace for Python 3.8 in build_code_chunk, but get_code_params is used to pickle and unpickle the code object, so I don't know if CodeType.replace can help in that case.

[vstinner]

Ah right, pickling code objects is non-trivial.

[hodgestar]

@vstinner Do you think CPython would be interested in support for serializing or pickling code objects? That would mean that perhaps one day this continual dance of fixing this code could end.

[brandtbucher]

Hm, could you marshal the code object instead, and use those bytes to "unmarshal" later? Not sure if that would be appropriate for your use-case.

[vstinner]

Do you think CPython would be interested in support for serializing or pickling code objects?

I don't know. You can propose the idea on python-dev or python-ideas.