search

opencog / link-grammar

The CMU Link Grammar natural language parser
GNU Lesser General Public License v2.1
389 stars 119 forks source link

Statically linked python #922

Open christian-storm opened 5 years ago

christian-storm commented 5 years ago

After banging my head on this for a couple days, I finally figured out what was causing link-grammar to segfault. Funny enough I found out 'why' by stumbling on an old issue I had created. It seems I'm SOL since python 3.7+ builds will be statically linked. In researching this I also stumbled upon https://github.com/conda-forge/python-feedstock/issues/222#issuecomment-442496532. In reading it I saw the familiar error from #298.

Fatal Python error: PyThreadState_Get: no current thread

They say if python is compiled as a position independent executable it seems one can compile against it, e.g., by creating a symlink python.so -> python.

I'm a bit over my head here but would it be possible to do this here?

I get the impression that more python builds will be statically linked for speed and security so this might occur more often. Another thought is if it possible to link against libpython as they suggest in the post.

Of note is that there is a quick test to see if python is dynamically or statically linked:

from distutils import sysconfig
sysconfig.get_config_var('Py_ENABLE_SHARED') # 1 if dynamically or 0 if static

At a minimum maybe you could add a test to configure sysconfig.get_config_var('Py_ENABLE_SHARED') to save the user from having to go through this again.

Anaconda python 3.5.4 is statically linked (try otool -L - you will not see the python dynamic library). After a recent update at my /pkgs/ directory there is also python version is 3.5.2 which is dynamically linked).

% ls anaconda/pkgs/*/bin/python
anaconda/pkgs/python-3.5.2-0/bin/python
anaconda/pkgs/python-3.5.4-hb8880cc_19/bin/python

Try to see if you have 3.5.2 too (but I guess it may disappear on the next update).

% anaconda/bin/python
Python 3.5.4 |Anaconda, Inc.| (default, Oct  5 2017, 02:58:14) 
...
% % otool -L anaconda/bin/python
anaconda/bin/python:
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 853.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

As you can see, there is no dynamic python library there. However:

% anaconda/pkgs/python-3.5.2-0/bin/python
Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:52:12)
...
% otool -L anaconda/pkgs/python-3.5.2-0/bin/python
anaconda/pkgs/python-3.5.2-0/bin/python:
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 635.21.0)
    @rpath/libpython3.5m.dylib (compatibility version 3.5.0, current version 3.5.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.1.0)

So unless you can see (using otool -L that your python is dynamically linked with the python library, there is no need even to try LG with it - it will fail.

Conclusion: You cannot use LG with an Anaconda Python version (or any other one) if it is statically linked with libpython.

Originally posted by @ampli in https://github.com/opencog/link-grammar/issues/298#issuecomment-350246315

linas commented 5 years ago

I don't understand what you are trying to say here. It's very confusing to read the above.

In general: dynamic libraries are supposed to be compatible over time, but it is possible that python no longer guarantees this.

Static libraries solve this compatibility problem, by permanently linking in the desired version. That way, if you upgrade python, your binary will still be linked to the old, correct library.

If you updated python, and now things don't work, then it should be possible to fix things by removing the old link-grammar, and recomplling all over again. Did you try this?

ampli commented 5 years ago

The problem is that the provided statically-linked python is just not able to load the dynamic clinkgrammar module that the LG build creates (compiled against a dynamic library that is also provided in the distribution, but is not used by the python binary because it is statically-linked).

I understood that @christian-storm suggests that configure will check 'Py_ENABLE_SHARED' and warn about this situation. He also suggests to try to link clinkgrammar against the Python binary itself, using it as if it was a shared library. I can try that.

christian-storm commented 5 years ago

Thanks Amir. That is it exactly.

On Thu, Feb 28, 2019 at 3:54 PM Amir Plivatsky notifications@github.com wrote:

The problem is that the provided statically-linked python is just not able to load the dynamic clinkgrammar module that the LG build creates (compiled against a dynamic library that is also provided in the distribution, but is not used by the python binary because it is statically-linked).

I understood that @christian-storm https://github.com/christian-storm suggests that configure will check 'Py_ENABLE_SHARED' and warn about this situation. He also suggests to try to link clinkgrammar against the Python binary itself, using it as if it was a shared library. I can try that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/opencog/link-grammar/issues/922#issuecomment-468487105, or mute the thread https://github.com/notifications/unsubscribe-auth/ARKIHO8V_Icmv2ezGlE3SmSYHme1VYmLks5vSGw_gaJpZM4bV40y .