traversaro
commented
11 months ago As mentioned in https://github.com/conda-forge/ctng-compilers-feedstock/issues/114#issuecomment-1718238163 , this problem can be easily reproduced with a custom shared library, for example the one in https://github.com/traversaro/reproduce-python-gomp-deepbind-issue/blob/d69e6a7e70b5be851b0206495ff4ecdc8734e1e1/testso.c .
In particular, one can run:
wget https://github.com/traversaro/reproduce-python-gomp-deepbind-issue/blob/d69e6a7e70b5be851b0206495ff4ecdc8734e1e1/testso.c
gcc -shared -fPIC testso.c -o testso.so
python -c "import ctypes; import os; ctypes._dlopen('./testso.so', os.RTLD_DEEPBIND)"to reproduce the problem. Removing the os.RTLD_DEEPBIND the problem does not appear. It is still not clear in which platform this happens. It happens for all conda-forge builds I tried, while for example on Ubuntu 22.04 apt packages of Python it does not happen, while if I compile the same python version by source as the one installed by apt the problem occurs. On the other hand, on Ubuntu 23.04 and 23.10 with apt packages the problem occurs.
To investigate more, I patched the cpython source code with the following patch:
diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c
index 3ac1034972..e0dbf1171d 100644
--- a/Modules/_ctypes/callproc.c
+++ b/Modules/_ctypes/callproc.c
@@ -1539,6 +1539,10 @@ static PyObject *py_dl_open(PyObject *self, PyObject *args)
return NULL;
}
+ fprintf(stderr, "=====> Calling ctypes_dlopen(%s, %d)\n", name_str, mode);
+ fprintf(stderr, "=====> environ (%p) &environ (%p)\n", environ, &environ);
handle = ctypes_dlopen(name_str, mode);
+ fprintf(stderr, "=====> Called ctypes_dlopen(%s, %d)\n", name_str, mode);
+ fprintf(stderr, "=====> environ (%p) &environ (%p)\n", environ, &environ);
Py_XDECREF(name2);
if (!handle) {
const char *errmsg = ctypes_dlerror();By running the reproducer with this patch, this is the output:
=====> Calling ctypes_dlopen((null), 2)
=====> environ (0x7fff5e89b868) &environ (0x55a647027c20)
=====> Called ctypes_dlopen((null), 2)
=====> environ (0x7fff5e89b868) &environ (0x55a647027c20)
=====> Calling ctypes_dlopen(./testso.so, 10)
=====> environ (0x7fff5e89b868) &environ (0x55a647027c20)
Print debug
environ (nil) &environ 0x7f59ba435200
=====> Called ctypes_dlopen(./testso.so, 10)
=====> environ (0x7fff5e89b868) &environ (0x55a647027c20)So, it is possible to see that it is not the environ variable that change value, but simply that for some reason inside the testso.so a different address is binded to the symbol environ. I tried to understand why this is happend by investigating which of the .so shared libraries contained a environ symbol, and it seems that the dynamic loader contains a local environ symbol, beside the global environ symbol contained in glibc that should be the one used:
traversaro@IITICUBLAP257:~/cpython$ ldd ./install/bin/python3
linux-vdso.so.1 (0x00007ffccf985000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fab811e2000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fab80fba000)
/lib64/ld-linux-x86-64.so.2 (0x00007fab81997000)
traversaro@IITICUBLAP257:~/cpython$ objdump --syms /lib/x86_64-linux-gnu/libc.so.6 | grep environ
000000000021aeb8 l O .bss 0000000000000008 last_environ
0000000000000000 l df *ABS* 0000000000000000 environ.c
0000000000044db0 l F .text 000000000000039d __add_to_environ
0000000000221200 w O .bss 0000000000000008 _environ
0000000000221200 w O .bss 0000000000000008 environ
0000000000221200 g O .bss 0000000000000008 __environ
traversaro@IITICUBLAP257:~/cpython$ objdump --syms /lib64/ld-linux-x86-64.so.2 | grep environ
0000000000000000 l df *ABS* 0000000000000000 dl-environ.c
0000000000000000 l df *ABS* 0000000000000000 environ.c
000000000003b2d0 l O .bss 0000000000000008 _environ
000000000003b2d0 l O .bss 0000000000000008 __environ
000000000003b2d0 l O .bss 0000000000000008 environSo, I am not sure but I am wondering if there is some strange interaction between this two environ symbols?
I also found this old glibc bug that seems related: https://sourceware.org/bugzilla/show_bug.cgi?id=9758 .
Solution to issue cannot be found in the documentation.
Issue
On Linux:
this fails with:
This is due to P1 described in https://github.com/conda-forge/ctng-compilers-feedstock/issues/114#issuecomment-1719077104 .
Installed packages
Environment info