I think generally multicorn2 works well with Python 3.12, but the test suite cannot be executed in completeness because of the use of plpython in the write_filesystem.sql test.
When this test is executed, the backend crashes. When loaded in a debugger, it is crashing while reporting the error: PyImport_AppendInittab() may not be called after Py_Initialize()", with a stack trace indicating it is being invoked from PLy_initialize in PLPython.
#0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1 0x00007ffff70a2f63 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2 0x00007ffff7052e86 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3 0x00007ffff703b935 in __GI_abort () at abort.c:79
#4 0x00007fffeb303e93 in fatal_error_exit (status=-1) at Python/pylifecycle.c:2735
#5 fatal_error (fd=<optimized out>, header=header@entry=1,
prefix=prefix@entry=0x7fffeb6958f0 <__func__.10.lto_priv.1> "PyImport_AppendInittab",
msg=msg@entry=0x7fffeb6d3c30 "PyImport_AppendInittab() may not be called after Py_Initialize()", status=status@entry=-1)
at Python/pylifecycle.c:2846
#6 0x00007fffeb35be43 in _Py_FatalErrorFunc (func=0x7fffeb6958f0 <__func__.10.lto_priv.1> "PyImport_AppendInittab",
msg=0x7fffeb6d3c30 "PyImport_AppendInittab() may not be called after Py_Initialize()") at Python/pylifecycle.c:2932
#7 0x00007fffeb35f4f8 in PyImport_AppendInittab (name=<optimized out>, initfunc=<optimized out>) at Python/import.c:1499
#8 PyImport_AppendInittab (name=name@entry=0x7ffff6ca4340 "plpy", initfunc=<optimized out>) at Python/import.c:1494
#9 0x00007ffff6c9cb76 in PLy_initialize () at plpy_main.c:129
#10 PLy_initialize () at plpy_main.c:107
... snip ...
Neither multicorn2 nor PLPython call PyImport_AppendInittab after calling Py_Initialize... multicorn2's order is correct:
So I'm left with the initial impression that the two interpreters loaded into the backend are conflicting on each other, which frankly makes sense. There is a change in Python 3.12 (https://github.com/python/cpython/pull/99402) that introduced this error condition:
I don't have any initial thoughts on how to avoid this conflict. Multicorn basically tries to mirror PLPython's initialization (doing a PyImport_AppendInittab("plpy") just because PLPython does it too), and it would be possible for it to detect if the runtime is already initialized and skip it. However, it's equally possible for PLPython to be init'd second (as in this case), in which no changes in Multicorn would prevent the error.
Maybe if Multicorn could runtime detect the presence of the PLy_initialize symbol and invoke it... which stores a static bool for whether it has initialized the runtime... that would avoid the problem. But as it's a static function and this is C, I'm not sure that would be very feasible.
I'll do some more research... but in the short-term this is the problem description at least.
I think generally multicorn2 works well with Python 3.12, but the test suite cannot be executed in completeness because of the use of plpython in the write_filesystem.sql test.
https://github.com/pgsql-io/multicorn2/blob/aebb975d6081ea74310eed88503f8c8f26a6d41c/test-3.9/sql/write_filesystem.sql#L36
When this test is executed, the backend crashes. When loaded in a debugger, it is crashing while reporting the error:
PyImport_AppendInittab() may not be called after Py_Initialize()", with a stack trace indicating it is being invoked from PLy_initialize in PLPython.Neither multicorn2 nor PLPython call PyImport_AppendInittab after calling Py_Initialize... multicorn2's order is correct:
https://github.com/pgsql-io/multicorn2/blob/aebb975d6081ea74310eed88503f8c8f26a6d41c/src/multicorn.c#L136-L144
And Postgres' order is correct:
https://github.com/postgres/postgres/blob/8fea1bd5411b793697a4c9087c403887e050c4ac/src/pl/plpython/plpy_main.c#L116-L117
So I'm left with the initial impression that the two interpreters loaded into the backend are conflicting on each other, which frankly makes sense. There is a change in Python 3.12 (https://github.com/python/cpython/pull/99402) that introduced this error condition:
https://github.com/python/cpython/pull/99402/files#diff-28cfc3e2868980a79d93d2ebdc8747dcb9231f3dd7f2caef96e74107d1ea3bf3R2691-R2694
I don't have any initial thoughts on how to avoid this conflict. Multicorn basically tries to mirror PLPython's initialization (doing a
PyImport_AppendInittab("plpy")just because PLPython does it too), and it would be possible for it to detect if the runtime is already initialized and skip it. However, it's equally possible for PLPython to be init'd second (as in this case), in which no changes in Multicorn would prevent the error.Maybe if Multicorn could runtime detect the presence of the
PLy_initializesymbol and invoke it... which stores a static bool for whether it has initialized the runtime... that would avoid the problem. But as it's a static function and this is C, I'm not sure that would be very feasible.I'll do some more research... but in the short-term this is the problem description at least.