Crash when running the test suite with Python 3.10

[mcepl]

When running the test suite, it fails with this backtrace:

https://gist.github.com/mcepl/43ada31674951bdbd62d8d560575a9f1

This is complete build log describing all packages used and steps taken to reproduce the problem (unfortunately, I had to collect backtrace outside of the build environment).

It is probably blocking https://github.com/mehcode/python-xmlsec/issues/204

[mcepl]

Or there is this backtrace ... not exactly the same: tests/test_xmlsec.py::TestModule::test_reinitialize_module passed and it failed only in tests/test_constants.py::test_transform_str: https://gist.github.com/mcepl/7c0cba8a3b41780dbb3e56ed31bbf299

[dirkmueller]

I have similar one under valgrind:

tests/test_constants.py ==15782== Invalid read of size 8
==15782==    at 0x969DFEF: PyXmlSec_Transform__str__ (constants.c:25)
==15782==    by 0x49AFEA3: PyObject_Str (in /usr/lib64/libpython3.10.so.1.0)
==15782==    by 0x49D1F1A: ??? (in /usr/lib64/libpython3.10.so.1.0)
==15782==    by 0x499C668: ??? (in /usr/lib64/libpython3.10.so.1.0)
==15782==    by 0x499C4E3: _PyObject_MakeTpCall (in /usr/lib64/libpython3.10.so.1.0)
==15782==    by 0x49990D1: _PyEval_EvalFrameDefault (in /usr/lib64/libpython3.10.so.1.0)
==15782==    by 0x4993672: ??? (in /usr/lib64/libpython3.10.so.1.0)
==15782==    by 0x499737E: _PyEval_EvalFrameDefault (in /usr/lib64/libpython3.10.so.1.0)
==15782==    by 0x4993672: ??? (in /usr/lib64/libpython3.10.so.1.0)
==15782==    by 0x49948B9: _PyEval_EvalFrameDefault (in /usr/lib64/libpython3.10.so.1.0)
==15782==    by 0x4993672: ??? (in /usr/lib64/libpython3.10.so.1.0)
==15782==    by 0x49A6A91: PyVectorcall_Call (in /usr/lib64/libpython3.10.so.1.0)
==15782==  Address 0x9963b50 is not stack'd, malloc'd or (recently) free'd
==15782== 
Fatal Python error: Segmentation fault

it appears this is due to

https://github.com/mehcode/python-xmlsec/blob/2460f50a51ae3719cdab5dbeabc18b8ef8c71f52/src/constants.c#L25

where id is suddenly pointing into the wild west. it looks like https://github.com/mehcode/python-xmlsec/blob/2460f50a51ae3719cdab5dbeabc18b8ef8c71f52/src/constants.c#L133 is the only place where id is assigned.

I'm obviously not understanding things, but this is not doing a deep-copy, just copying the pointer. what exactly is supposed to keep the reference alive?

shouldn't this be copied and freed then in __del__?

[mcepl]

Isn't it just that transform->id->href can be uninitialized? I have to admit that I got lost in the bizarre maze of macro hell in src/constants.c.

[dirkmueller]

I don't see how it can be uninitialized, it is unconditionally initialized in the constructing function?

[mcepl]

s/uninitialized/has not been initialized/ I had lost my way in the endless bogs of src/constants.c, but I am just not persuaded still that there are no situations where transform->id->href has not been set at all.

[danigm]

Debugging this problem I've found that the problem is when trying to access transform->id->href, but it's even not null, so it should be something deeper.

25      if (transform->id->href != NULL)
(gdb) p transform
$1 = (PyXmlSec_Transform *) 0x7ffff5704230
(gdb) p transform->id
$2 = (xmlSecTransformId) 0x7ffff5d13220
(gdb) p transform->id->href
Cannot access memory at address 0x7ffff5d13230

I've also detected that this problem doesn't happen to me if I run just the test_constants but it happens when running the full test suite, so it should be some weird state.

[danigm]

Looks like the problem is with test_reinitialize_module, the call to shutdown and a race condition or something like that. I've detected that running the tests with pytest -n 1 doesn't produce this problem.