Closed traversaro closed 11 months ago
As mentioned in https://github.com/conda-forge/ctng-compilers-feedstock/issues/114#issuecomment-1718238163 , this problem can be easily reproduced with a custom shared library, for example the one in https://github.com/traversaro/reproduce-python-gomp-deepbind-issue/blob/d69e6a7e70b5be851b0206495ff4ecdc8734e1e1/testso.c .
In particular, one can run:
wget https://github.com/traversaro/reproduce-python-gomp-deepbind-issue/blob/d69e6a7e70b5be851b0206495ff4ecdc8734e1e1/testso.c
gcc -shared -fPIC testso.c -o testso.so
python -c "import ctypes; import os; ctypes._dlopen('./testso.so', os.RTLD_DEEPBIND)"
to reproduce the problem. Removing the os.RTLD_DEEPBIND
the problem does not appear. It is still not clear in which platform this happens. It happens for all conda-forge builds I tried, while for example on Ubuntu 22.04 apt packages of Python it does not happen, while if I compile the same python version by source as the one installed by apt the problem occurs. On the other hand, on Ubuntu 23.04 and 23.10 with apt packages the problem occurs.
To investigate more, I patched the cpython source code with the following patch:
diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c
index 3ac1034972..e0dbf1171d 100644
--- a/Modules/_ctypes/callproc.c
+++ b/Modules/_ctypes/callproc.c
@@ -1539,6 +1539,10 @@ static PyObject *py_dl_open(PyObject *self, PyObject *args)
return NULL;
}
+ fprintf(stderr, "=====> Calling ctypes_dlopen(%s, %d)\n", name_str, mode);
+ fprintf(stderr, "=====> environ (%p) &environ (%p)\n", environ, &environ);
handle = ctypes_dlopen(name_str, mode);
+ fprintf(stderr, "=====> Called ctypes_dlopen(%s, %d)\n", name_str, mode);
+ fprintf(stderr, "=====> environ (%p) &environ (%p)\n", environ, &environ);
Py_XDECREF(name2);
if (!handle) {
const char *errmsg = ctypes_dlerror();
By running the reproducer with this patch, this is the output:
=====> Calling ctypes_dlopen((null), 2)
=====> environ (0x7fff5e89b868) &environ (0x55a647027c20)
=====> Called ctypes_dlopen((null), 2)
=====> environ (0x7fff5e89b868) &environ (0x55a647027c20)
=====> Calling ctypes_dlopen(./testso.so, 10)
=====> environ (0x7fff5e89b868) &environ (0x55a647027c20)
Print debug
environ (nil) &environ 0x7f59ba435200
=====> Called ctypes_dlopen(./testso.so, 10)
=====> environ (0x7fff5e89b868) &environ (0x55a647027c20)
So, it is possible to see that it is not the environ variable that change value, but simply that for some reason inside the testso.so
a different address is binded to the symbol environ
. I tried to understand why this is happend by investigating which of the .so shared libraries contained a environ
symbol, and it seems that the dynamic loader contains a local environ
symbol, beside the global environ
symbol contained in glibc that should be the one used:
traversaro@IITICUBLAP257:~/cpython$ ldd ./install/bin/python3
linux-vdso.so.1 (0x00007ffccf985000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fab811e2000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fab80fba000)
/lib64/ld-linux-x86-64.so.2 (0x00007fab81997000)
traversaro@IITICUBLAP257:~/cpython$ objdump --syms /lib/x86_64-linux-gnu/libc.so.6 | grep environ
000000000021aeb8 l O .bss 0000000000000008 last_environ
0000000000000000 l df *ABS* 0000000000000000 environ.c
0000000000044db0 l F .text 000000000000039d __add_to_environ
0000000000221200 w O .bss 0000000000000008 _environ
0000000000221200 w O .bss 0000000000000008 environ
0000000000221200 g O .bss 0000000000000008 __environ
traversaro@IITICUBLAP257:~/cpython$ objdump --syms /lib64/ld-linux-x86-64.so.2 | grep environ
0000000000000000 l df *ABS* 0000000000000000 dl-environ.c
0000000000000000 l df *ABS* 0000000000000000 environ.c
000000000003b2d0 l O .bss 0000000000000008 _environ
000000000003b2d0 l O .bss 0000000000000008 __environ
000000000003b2d0 l O .bss 0000000000000008 environ
So, I am not sure but I am wondering if there is some strange interaction between this two environ
symbols?
I also found this old glibc bug that seems related: https://sourceware.org/bugzilla/show_bug.cgi?id=9758 .
Ok, I was able to reproduce reliably the problem in a C example, without the need to use Python, thanks to the input from https://sourceware.org/bugzilla/show_bug.cgi?id=9758 . This is what I understood (not an expert, so I could get things wrong).
The file to reproduce the behavior can be found in https://gist.github.com/traversaro/2d12a7067f686472207c942043f8c7b7 .
First of all, the root cause of this is the following behaviour, quoting from https://sourceware.org/bugzilla/show_bug.cgi?id=11717 :
Apparently, when linking a shared library which contains a variable in .bss with an executable referencing that variable, ld allocates space for that variable in the executable's .bss section and creates a duplicate symbol for this variable.
So, in the previous test with C++, I was not accessing environ
from the executable, so there was only one environ
symbol: the one in glibc's BSS (with weak visibility). So, in all cases in the loaded .so, the environ
variable was correctly resolved to the variable in glibc's BSS. If instead the executable (in our case, Python) refers directly to environ
, then a copy of environ
is created in Python's bss section, with global visibility. Without using RTLD_DEEPBIND
, everything works fine, as all the libraries use the environ
from the executable bss, and ignore the environ
from glibc's bss. Note that glibc initialization code use the environ
from the executable bss, while the environ from glibc's bss remains to its initialization value (NULL
).
If instead RTLD_DEEPBIND
is used, then in the shared library the environ
in the executable bss is ignored, and instead the one in glibc's bss is used. However, the environ
from glibc bss is still set to NULL, as glibc initialized the one in the executable bss. Clearly, nothing of this happens if the executable does not use environ directly. I guess all the differences between different versions of Python we experienced is due to the use or not directly of environ (this can be easily checked by looking if there is a B
symbol for __environ
in Python executable).
Interestingly, if the shared library that is opened via RTLD_DEEPBIND access the environment variables via the getenv
or similar functions, everything works fine as inside glibc functions environ is correctly resolved to the environ from executable bss, even if the environ as seen by the loaded shared library is NULL
.
I opened a bug upstream in libgomp at https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111556, and libgomp maintainers confirmed that this is not an issue that can be fixed on libgomp. Given that, I think the options are:
environ
is copy relocated in the Python executable: I am not sure if this is possible, and anyhow it would require a change in Python.WITH_DEEPBIND
CMake option to OFF (see https://github.com/casadi/casadi/blob/7f8f055024ed6d86532612f1b706195c7d41074a/CMakeLists.txt#L316). However, this can create problems if casadi is loaded in MATLAB, see https://github.com/casadi/casadi/issues/1877 . Personally this is a use case I care off, so I would first test if WITH_DEEPBIND CasADi continues to work fine on MATLAB.At the moment, I can't think of other solutions.
At the moment, I can't think of other solutions.
I tought of another solution, added in https://github.com/conda-forge/casadi-feedstock/pull/94 that is working in my local tests.
Solution to issue cannot be found in the documentation.
Issue
On Linux:
this fails with:
This is due to P1 described in https://github.com/conda-forge/ctng-compilers-feedstock/issues/114#issuecomment-1719077104 .
Installed packages
Environment info