search

opencobra / cobrapy

COBRApy is a package for constraint-based modeling of metabolic networks.
http://opencobra.github.io/cobrapy/
GNU General Public License v2.0
465 stars 218 forks source link

symengine will conflict with libsbml on Python 3.9 and Linux #1056

Closed cdiener closed 3 years ago

cdiener commented 3 years ago

Problem description

This will hit us when we support Python 3.9. There seems to be serious bug in libSBML that breaks reading of SBO terms completely. This will make several of our tests fail.

Code Sample

Create a minimal, complete, verifiable example.

This uses the E. coli model from our repo.

from cobra.io import read_sbml_model

mod = read_sbml_model("iJO1366.xml.gz")
mod.reactions.LDH_D.annotation

On Python < 3.9

{'sbo': 'SBO:0000375', 'bigg.reaction': 'LDH_D', 'biocyc': 'META:DLACTDEHYDROGNAD-RXN', 'ec-code': '1.1.1.28', 'kegg.reaction': 'R00704', 'metanetx.reaction': 'MNXR101037', 'rhea': ['16369', '16370', '16371', '16372']}

On Python 3.9

{'sbo': 'SBO:',  # <- see here
 'bigg.reaction': 'LDH_D',
 'biocyc': 'META:DLACTDEHYDROGNAD-RXN',
 'ec-code': '1.1.1.28',
 'kegg.reaction': 'R00704',
 'metanetx.reaction': 'MNXR101037',
 'rhea': ['16369', '16370', '16371', '16372']}

This is due to a bug in libsbml where libsbml.SBase.getSBOTermID() always returns "SBO:" (without the ID). libsbml.SBase.getSBOTerm() still works (returns 375 in this case).

Context

I think depinfo is a bit buggy as well. Here is the correct libsbml version:

ython 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libsbml
>>> libsbml.__version__
'5.19.0'
``` System Information ================== OS Linux OS-release 5.10.0-4-amd64 Python 3.9.2 Package Versions ================ appdirs~=1.4 not installed black 20.8b1 bumpversion 0.6.0 cobra 0.21.0 depinfo 1.6.0 diskcache~=5.0 not installed future 0.18.2 httpx~=0.14 not installed importlib_resources 3.3.0 isort; extra == "development" not installed numpy~=1.13 not installed optlang<1.4.6 not installed pandas~=1.0 not installed pip 21.0.1 pydantic~=1.6 not installed python-libsbml==5.19.0 not installed rich~=6.0 not installed ruamel.yaml~=0.16 not installed scipy 1.6.1 setuptools 54.1.2 six 1.15.0 swiglpk 5.0.3 tox 3.23.0 wheel 0.36.2 ```
Midnighter commented 3 years ago

So is this something we should address or do we need an upstream fix in libSBML?

(Please open an issue on depinfo when you can say clearly what is going wrong.)

cdiener commented 3 years ago

I think this is upstream but there is a potential workaround by using libsbml.SBase.getSBOTerm() instead and assembling the ID from that.

No idea about depinfo. It shows the correct versions for some deps but not others. Maybe it's something with my setup.

cdiener commented 3 years ago

Okay, it is not upstream per se, but there is something weird going on with our imports because the following fixes it:

Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import libsbml

In [2]: from cobra.io import read_sbml_model

In [3]: mod = read_sbml_model("src/cobra/test/data/iJO1366.xml.gz")

In [4]: mod.reactions.LDH_D.annotation
Out[4]: 
{'sbo': 'SBO:0000375',
 'bigg.reaction': 'LDH_D',
 'biocyc': 'META:DLACTDEHYDROGNAD-RXN',
 'ec-code': '1.1.1.28',
 'kegg.reaction': 'R00704',
 'metanetx.reaction': 'MNXR101037',
 'rhea': ['16369', '16370', '16371', '16372']}

So just by importing libsbml before cobrapy it has no issues anymore. I could also reproduce this in a fresh Python env.

Midnighter commented 3 years ago

Strange, I don't have that issue at all in the following environment:

Python 3.9.0 (default, Nov 29 2020, 14:58:16)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from cobra.io import read_sbml_model

In [2]: mod = read_sbml_model("src/cobra/test/data/iJO1366.xml.gz")

In [3]: mod.reactions.LDH_D.annotation
Out[3]:
{'sbo': 'SBO:0000375',
 'bigg.reaction': 'LDH_D',
 'biocyc': 'META:DLACTDEHYDROGNAD-RXN',
 'ec-code': '1.1.1.28',
 'kegg.reaction': 'R00704',
 'metanetx.reaction': 'MNXR101037',
 'rhea': ['16369', '16370', '16371', '16372']}
System Information
==================
OS                      Linux
OS-release 5.8.0-7642-generic
Python                  3.9.0

Package Versions
================
appdirs                                      1.4.4
black ; extra == 'development'       not installed
bumpversion ; extra == 'development' not installed
cobra                                       0.21.0
depinfo                                      1.7.0
diskcache                                    5.2.1
future                                      0.18.2
httpx                                       0.17.1
importlib-resources                          5.1.2
isort ; extra == 'development'       not installed
numpy                                       1.20.1
optlang                                      1.4.4
pandas                                       1.2.3
pip                                         20.2.3
pydantic                                     1.8.1
python-libsbml                              5.19.0
rich                                         6.2.0
ruamel.yaml                                0.16.13
scipy ; extra == 'array'             not installed
setuptools                                  49.2.1
six                                         1.15.0
swiglpk                                      5.0.3
tox ; extra == 'development'         not installed
cdiener commented 3 years ago

Here is a more minimal version that triggers the error for me:

Python 3.9.0 (default, Nov 15 2020, 14:28:56) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import cobra

In [2]: import libsbml

In [3]: doc = libsbml.readSBML("iJO1366.xml.gz")

In [4]: doc.getModel().getReaction("R_LDH_D").getSBOTermID()
Out[4]: 'SBO:'
cdiener commented 3 years ago

The problem is a conflict between libsbml and symengine. Uninstalling symengine fixes the bug.

Midnighter commented 3 years ago 
Python 3.9.0 (default, Nov 29 2020, 14:58:16) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import libsbml

In [2]: doc = libsbml.readSBML("cobrapy/src/cobra/test/data/iJO1366.xml.gz")

In [3]: doc.getModel().getReaction("R_LDH_D").getSBOTermID()
Out[3]: 'SBO:0000375'

😆 which libSBML is that using?

Midnighter commented 3 years ago

The problem is a conflict between libsbml and symengine. Uninstalling symengine fixes the bug.

Wow, what?

cdiener commented 3 years ago

I had those issues before. Something in the symengine wheels breaks other packages, I'll open a report there.

See for instance: https://github.com/numba/llvmlite/issues/587

cdiener commented 3 years ago

Minimal example:

In [1]: import symengine

In [2]: import libsbml

In [3]: doc = libsbml.readSBML("iJO1366.xml.gz")

In [4]: doc.getModel().getReaction("R_LDH_D").getSBOTermID()
Out[4]: 'SBO:'

Removing the symengine import will change the output to:

Python 3.9.0 (default, Nov 15 2020, 14:28:56) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import libsbml

In [2]: doc = libsbml.readSBML("iJO1366.xml.gz")

In [3]: doc.getModel().getReaction("R_LDH_D").getSBOTermID()
Out[3]: 'SBO:0000375'
Midnighter commented 3 years ago

I was wondering if symengine causes libsbml to load a different shared library but as far as I can tell that is not the case.

❯ ldd python3.9/site-packages/libsbml/_libsbml.cpython-39-x86_64-linux-gnu.so                                                                                
        linux-vdso.so.1 (0x00007ffe9fb8a000)                                                   
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5b691ce000)            
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5b6907f000)                      
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5b69064000)              
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5b68e72000)                      
        /lib64/ld-linux-x86-64.so.2 (0x00007f5b6a986000)
❯ ldd python3.9/site-packages/symengine.libs/libsymengine-054f8ecc.so.0.7.0                                                                                  
        linux-vdso.so.1 (0x00007ffd109f5000)
        libgmp-b6b552a9.so.10.4.1 => not found
        libflint-b75bea04.so.15.0.1 => not found                                               
        libmpc-d5c9e4c1.so.3.1.0 => not found
        libmpfr-0d3b9336.so.6.0.2 => not found
        libz-1f4891ad.so.1.2.11 => not found
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4557b2e000)                    
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4557b0b000)          
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f45579bc000)                      
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f45579a1000)              
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f45577af000)                                                                                                                     
        /lib64/ld-linux-x86-64.so.2 (0x00007f4559bc6000)

Unless it's a local linux-vdso.so.1? Not really my area of strength.

cdiener commented 3 years ago

But one is a manylinux1 image and the other manylinux2010, so there may be different versions of the same lib. But if they are built cleanly they should not export overlapping symbols.

matthiaskoenig commented 3 years ago

Most likely this is due to C++ symbols being overwritten within the swig packages. Basically all swig C++ symbols are imported in the same namespace so that conflicting namespaces will overwrite symbols from other packages. So the last package import wins basically. Something similar happened between libsbml and libsedml which have many similar names. That the import order is fixing this is a strong indicator. As a workaround a reload of the library works within the respective code areas. E.g. just do a reload(libsbml) or reload(symengine) before accessing objects from the library.

cdiener commented 3 years ago

Symengine is not a swig package though. But I do think you are right, ergo same symbols. I thought that the point of the new many linux images was to isolate this better but that does not seem to be the case.

cdiener commented 3 years ago

Fixed in symengine 0.7.1.

Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import symengine

In [2]: import libsbml

In [3]: doc = libsbml.readSBML("iJO1366.xml")

In [4]: doc.getModel().getReaction("R_LDH_D").getSBOTermID()
Out[4]: 'SBO:0000375'