Fails to build on Python 3.10.0a2

[mgedmin]

Trying to build zodbpickle on Python 3.10.0a2 fails with

  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/mg/src/zodbbrowser/.tox/py310/include -I/home/mg/opt/python310/include/python3.10 -c src/zodbpickle/_pickle_33.c -o build/temp.linux-x86_64-3.10/src/zodbpickle/_pickle_33.o
  src/zodbpickle/_pickle_33.c: In function ‘Pdata_New’:
  src/zodbpickle/_pickle_33.c:179:19: error: lvalue required as left operand of assignment
    179 |     Py_SIZE(self) = 0;
        |                   ^
  src/zodbpickle/_pickle_33.c: In function ‘Pdata_clear’:
  src/zodbpickle/_pickle_33.c:205:19: error: lvalue required as left operand of assignment
    205 |     Py_SIZE(self) = clearto;
        |                   ^
  src/zodbpickle/_pickle_33.c: In function ‘Pdata_pop’:
  src/zodbpickle/_pickle_33.c:247:23: error: lvalue required as decrement operand
    247 |     return self->data[--Py_SIZE(self)];
        |                       ^~
  src/zodbpickle/_pickle_33.c: In function ‘Pdata_push’:
  src/zodbpickle/_pickle_33.c:257:29: error: lvalue required as increment operand
    257 |     self->data[Py_SIZE(self)++] = obj;
        |                             ^~
  src/zodbpickle/_pickle_33.c: In function ‘Pdata_poptuple’:
  src/zodbpickle/_pickle_33.c:283:19: error: lvalue required as left operand of assignment
    283 |     Py_SIZE(self) = start;
        |                   ^
  src/zodbpickle/_pickle_33.c: In function ‘Pdata_poplist’:
  src/zodbpickle/_pickle_33.c:300:19: error: lvalue required as left operand of assignment
    300 |     Py_SIZE(self) = start;
        |                   ^
  src/zodbpickle/_pickle_33.c: In function ‘save_long’:
  src/zodbpickle/_pickle_33.c:1646:18: warning: implicit declaration of function ‘_PyUnicode_AsStringAndSize’; did you mean ‘PyUnicode_FromStringAndSize’? [-Wimplicit-function-declaration]
   1646 |         string = _PyUnicode_AsStringAndSize(repr, &size);
        |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~
        |                  PyUnicode_FromStringAndSize
  src/zodbpickle/_pickle_33.c:1646:16: warning: assignment to ‘char *’ from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
   1646 |         string = _PyUnicode_AsStringAndSize(repr, &size);
        |                ^
  src/zodbpickle/_pickle_33.c: In function ‘save_pers’:
  src/zodbpickle/_pickle_33.c:2942:29: warning: assignment to ‘char *’ from ‘int’ makes pointer from integer without a cast [-Wint-conversion]
   2942 |             pid_ascii_bytes = _PyUnicode_AsStringAndSize(pid_str, &size);
        |                             ^
  src/zodbpickle/_pickle_33.c: In function ‘Pickler_init’:
  src/zodbpickle/_pickle_33.c:3582:9: warning: implicit declaration of function ‘_PyObject_HasAttrId’; did you mean ‘_PyObject_SetAttrId’? [-Wimplicit-function-declaration]
   3582 |     if (_PyObject_HasAttrId((PyObject *)self, &PyId_persistent_id)) {
        |         ^~~~~~~~~~~~~~~~~~~
        |         _PyObject_SetAttrId
  src/zodbpickle/_pickle_33.c: In function ‘load_pop’:
  src/zodbpickle/_pickle_33.c:4840:30: error: lvalue required as left operand of assignment
   4840 |         Py_SIZE(self->stack) = len;
        |                              ^
  src/zodbpickle/_pickle_33.c: In function ‘do_append’:
  src/zodbpickle/_pickle_33.c:5145:38: error: lvalue required as left operand of assignment
   5145 |                 Py_SIZE(self->stack) = x;
        |                                      ^
  src/zodbpickle/_pickle_33.c:5151:30: error: lvalue required as left operand of assignment
   5151 |         Py_SIZE(self->stack) = x;
        |                              ^
  src/zodbpickle/_pickle_33.c: In function ‘Pdata_pop’:
  src/zodbpickle/_pickle_33.c:248:1: warning: control reaches end of non-void function [-Wreturn-type]
    248 | }
        | ^
  error: command '/usr/lib/ccache/gcc' failed with exit code 1

[jamadden]

In Python 3.9 and earlier, Py_SIZE is a macro, and Py_SIZE(self) expands to (((PyVarObject*)self)->ob_size). This is just an integer field of the object, and as such it can be used as a value (rvalue) and assignment target (lvalue).

In 3.10, as part of a broader project to overhaul the C API, some uses of macros have been replaced with functions:

static inline Py_ssize_t _Py_SIZE(const PyVarObject *ob) {
    return ob->ob_size;
}
#define Py_SIZE(ob) _Py_SIZE((PyVarObject*)(ob))

A function call is not a valid assignment target, so all uses of Py_SIZE on the left-hand side have to be changed. Given that the function is declared inline anyway, and thus not part of the stable ABI, it's probably OK to define our own macro equivalent to the original Py_SIZE macro and use that.

_PyUnicode_AsStringAndSize is also a simple define for backwards compatibility:

#define _PyUnicode_AsStringAndSize PyUnicode_AsUTF8AndSize

PyUnicode_AsUTF8AndSize exists all the way back in 3.4 so that should be a simple replacement.

[jamesjer]

Upstream added a macro, Py_SET_SIZE, in python 3.9, in case that is helpful for you. I understand that Py_SIZE will be an lvalue in python 3.10 after all, due to a lot of breakage of this sort.

We are hitting another problem while trying to build Fedora packages with python 3.10 alpha versions:

Traceback (most recent call last):
  File "/builddir/build/BUILD/zodbpickle-2.0.0/setup.py", line 45, in <module>
    setup(
  File "/usr/lib/python3.10/site-packages/setuptools/__init__.py", line 153, in
setup
    return distutils.core.setup(**attrs)
  File "/usr/lib64/python3.10/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib64/python3.10/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python3.10/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.10/site-packages/setuptools/command/test.py", line
232, in run
    self.run_tests()
  File "/usr/lib/python3.10/site-packages/setuptools/command/test.py", line
250, in run_tests
    test = unittest.main(
  File "/usr/lib64/python3.10/unittest/main.py", line 100, in __init__
    self.parseArgs(argv)
  File "/usr/lib64/python3.10/unittest/main.py", line 147, in parseArgs
    self.createTests()
  File "/usr/lib64/python3.10/unittest/main.py", line 158, in createTests
    self.test = self.testLoader.loadTestsFromNames(self.testNames,
  File "/usr/lib64/python3.10/unittest/loader.py", line 220, in
loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib64/python3.10/unittest/loader.py", line 220, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib64/python3.10/unittest/loader.py", line 205, in
loadTestsFromName
    test = obj()
  File
"/builddir/build/BUILD/zodbpickle-2.0.0/src/zodbpickle/tests/test_pickle.py",
line 27, in test_suite
    from .test_pickle_3 import test_suite
  File
"/builddir/build/BUILD/zodbpickle-2.0.0/src/zodbpickle/tests/test_pickle_3.py",
line 7, in <module>
    from .pickletester_3 import AbstractPickleTests
  File
"/builddir/build/BUILD/zodbpickle-2.0.0/src/zodbpickle/tests/pickletester_3.py",
line 10, in <module>
    from test.support import (
ImportError: cannot import name 'TESTFN' from 'test.support'
(/usr/lib64/python3.10/test/support/__init__.py)

test.support is an internal module; it does not provide a stable API. TESTFN was removed:

https://bugs.python.org/issue40275

As a stop-gap measure, it can be imported from test.support.os_helper instead (still internal and unstable).

[pfw]

I posted a pull request (#60) that does build and test fine on 3.9/3.10 but doesn't support older versions - if that is helpful at all.

[jamadden]

The change to Py_SIZE was reverted because it broke the world..