jep errors with symbol PyExc_SystemError

[pyNpy]

Describe the bug

1. I use jep to call a package which named "foo" , the package "foo" is compiled by cython ;

2. in windows 10 , jep run ok ,i get the right result ,

3. in centos7 and use python3 to call package "foo", i get the right result

Python 3.7.3 (default, Dec 5 2021, 11:05:33) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux Type "help", "copyright", "credits" or "license" for more information.

import foo from foo import do do <cyfunction do at 0x7fed26c0ecc8> do() Running

4.  in centos 7  , jep  run with errors , these message is : 

Exception in thread "main" jep.JepException: <class 'ImportError'>: /home/administrator/.local/lib/python3.7/site-packages/foo/foo.so: undefined symbol: PyExc_SystemError at /home/administrator/.local/lib/python3.7/site-packages/foo/init.(init.py:3) at .(:1) at jep.Jep.exec(Native Method) at jep.Jep.exec(Jep.java:339) at com.demo.java_call_py.Main.jep_call_foo2(Main.java:97) at com.demo.java_call_py.Main.main(Main.java:121)

5.  the symbol "PyExc_SystemError" is in  lib :  {python_home}/lib/libpython3.7m.so,

nm -D libpython3.7m.so |grep PyExc_SystemError

found  the symbol :  ​0000000000524550 D PyExc_SystemError

6.  I try to use java to load  the lib : libpython3.7m.so  ,  the /proc/{pid}/maps shows that  libpython3.7m.so is loaded in memory,
but still  get those errors ,

[administrator@localhost ~]$ cat /proc/127793/maps |grep python 7f37cf859000-7f37cfb6d000 r-xp 00000000 fd:00 43747550 /usr/local/python3.7/lib/libpython3.7m.so.1.0 7f37cfb6d000-7f37cfd6d000 ---p 00314000 fd:00 43747550 /usr/local/python3.7/lib/libpython3.7m.so.1.0 7f37cfd6d000-7f37cfd70000 r--p 00314000 fd:00 43747550 /usr/local/python3.7/lib/libpython3.7m.so.1.0 7f37cfd70000-7f37cfddc000 rw-p 00317000 fd:00 43747550 /usr/local/python3.7/lib/libpython3.7m.so.1.0

static { System.load("/usr/local/python3.7/lib/libpython3.7m.so"); System.load("/usr/local/python3.7/lib/libpython3.7m.so.1.0"); System.out.println("load:");

    // so path
    jep.MainInterpreter.setJepLibraryPath("/usr/local/python3.7/lib/python3.7/site-packages/jep/libjep.so");

    // python home
    PyConfig pyconfig= new PyConfig();
    pyconfig.setPythonHome("/usr/local/python3.7");
    pyconfig.setVerboseFlag( -1 );
    jep.MainInterpreter.setInitParams(pyconfig);
    // shared lib

    jepconfig_obj.setIncludePath("/usr/local/python3.7/include/python3.7m");
    jepconfig_obj.setIncludePath("/usr/local/python3.7/lib");
    jepconfig_obj.addSharedModules( "/usr/local/python3.7/lib/libpython3.7m.so" );
    jepconfig_obj.addSharedModules( "/usr/local/python3.7/lib/libpython3.7m.so.1.0" );

}

**To Reproduce**

**Expected behavior**

Exception in thread "main" jep.JepException: <class 'ImportError'>: /home/administrator/.local/lib/python3.7/site-packages/foo/foo.so: undefined symbol: PyExc_SystemError at /home/administrator/.local/lib/python3.7/site-packages/foo/init.(init.py:3) at .(:1) at jep.Jep.exec(Native Method) at jep.Jep.exec(Jep.java:339) at com.demo.java_call_py.Main.jep_call_foo2(Main.java:97) at com.demo.java_call_py.Main.main(Main.java:121)

**Environment (please complete the following information):**
- OS Platform, Distribution, and Version: ``` Linux localhost.localdomain 3.10.0-1062.el7.x86_64 #1 SMP Wed Aug 7 18:08:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux ```
- Python Distribution and Version: python 3.7.3 
- Java Distribution and Version:  jdk1.8
- Jep Version: 4.0.0  4.0.1 
- Python packages used (e.g. numpy, pandas, tensorflow): none ,but  package  "foo" build by cython

**Additional context**

want to ask : 
1.  can  jep load  shared  lib ?
2.  the  shared lib can not be loaded  in memory  correctly  or can not find other  dependent  shared lib correctly   ? 

[bsteffensmeier]

We have seen similar errors in the past and it is typically caused because foo.so does not depend on libpython. Although libpython is loaded in the process the symbols from libpython are only made available to libraries that declare a dependency on libpython, if foo.so does not declare this dependency it cannot see the symbols. You can check if this is the case for foo.so by running ldd foo.so. This should include the libpython dependency in the output since it requires symbols from libpython.

The python command is still able to load foo.so because it makes the python related symbols global so all libraries can find the python symbols without declaring a dependency. Libraries without the declared dependency have trouble in embedded environments where the python symbols are not globally available.

If this is the case for foo.so then the simplest solution is to set the LD_PRELOAD env var to /usr/local/python3.7/lib/libpython3.7m.so This will force the library to load globally so the symbols will be available without the dependency.

Jep has code that attempts to make libpython available globally but errors in this process are not currently reported because it should not be a problem if libraries properly declare their dependencies. If something is going wrong with this process it would be helpful if you could troubleshoot it to determine why it isn't working in your environment. The first step would be to rebuild jep but change dlerror() to printf(dlerror()) and see if there is any output from the dynamic linking of libpython.

[pyNpy]

yes , LD_PRELOAD is the the simplest solution . i try this way, and solve these errors , thanks

[pyNpy]

"Jep has code that attempts to make libpython available globally"

i try this way , and printf some words in ifdef and endif , but find no logs to show , maybe the codes between the "#ifdef" and "#endif" , not works , i will try it later

#ifdef _DLFCN_H
 ...
#endif