search

pysam-developers / pysam

Pysam is a Python package for reading, manipulating, and writing genomics data such as SAM/BAM/CRAM and VCF/BCF files. It's a lightweight wrapper of the HTSlib API, the same one that powers samtools, bcftools, and tabix.
https://pysam.readthedocs.io/en/latest/
MIT License
774 stars 274 forks source link

pysam-0.11.2.1 error on Mac OS/X Sierra #475

Closed diekhans closed 6 years ago

diekhans commented 7 years ago

pip install of pysam-0.11.2.1 on Mac OS/X Sierra with python installed with macports results in a broken installation due to dynamic library reference to the wrong location

Python 2.7.13 (default, Apr 23 2017, 16:50:35) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
>>> import pysam
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pysam/__init__.py", line 5, in <module>
    from pysam.libchtslib import *
  File "pysam/libchtslib.pyx", line 1, in init pysam.libchtslib (pysam/libchtslib.c:12515)
ImportError: dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pysam/libcutils.so, 2): Library not loaded: @rpath/pysam-0.11.2.1-py2.7-macosx-10.12-x86_64.egg/pysam/libcsamtools.so
  Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pysam/libcutils.so
  Reason: image not found

libcsamtools.so is actually installed in: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pysam/libcsamtools.so

diekhans commented 7 years ago

manually check pysam out of git and compile still requires

cd /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages sudo ln -s pysam-0.11.2.1-py2.7-macosx-10.12-x86_64.egg/pysam .

bioinformed commented 7 years ago

Thanks for the report. I'm working on a fix.

maximilianpress commented 7 years ago

Same issue with a pip install, except pysam 0.11.2.2 and system Python (a virtualenv Python installed elsewhere shows same problem), MacOSX Sierra. I was able to resolve by uninstalling the pip version and just using git-->compile but I am not totally clear on the causality there.

Possibly I did something stupid while setting my environment up in the first place.

Unlike @diekhans, as far as I can tell everything is in the right location for both pip and the manual build but in the pip install it seems that Python is not loading the library even though it is looking in the right place.

Thanks, max

# (pip install of pysam), same behavior observed with Python 2.7.13
Python 2.7.10 (default, Feb  7 2017, 00:08:15) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pysam
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/pysam/__init__.py", line 5, in <module>
    from pysam.libchtslib import *
  File "pysam/libchtslib.pyx", line 1, in init pysam.libchtslib (pysam/libchtslib.c:12515)
ImportError: dlopen(/usr/local/lib/python2.7/site-packages/pysam/libcutils.so, 2): Library not loaded: @rpath/pysam/libcsamtools.so
  Referenced from: /usr/local/lib/python2.7/site-packages/pysam/libcutils.so
  Reason: image not found

# libcsamtools.so is in the right place?
$ ls /usr/local/lib/python2.7/site-packages/pysam/
Pileup.py       csamtools_util.h    libcbcftools.pxd    libcsamtools.pxd    libcvcf.so      utils.py
Pileup.pyc      htslib_util.h       libcbcftools.so     libcsamtools.so     namedtuple.py       utils.pyc
__init__.py     include         libcbgzf.so     libctabix.pxd       namedtuple.pyc      version.py
__init__.pyc        libcalignedsegment.pxd  libcfaidx.pxd       libctabix.so        pysam_stream.h      version.pyc
bcftools.py     libcalignedsegment.so   libcfaidx.so        libctabixproxies.pxd    pysam_util.h
bcftools.pyc        libcalignmentfile.pxd   libchtslib.pxd      libctabixproxies.so samfile_util.h
cbcftools_util.h    libcalignmentfile.so    libchtslib.so       libcutils.pxd       samtools.py
config.py       libcbcf.pxd     libcsamfile.pxd     libcutils.so        samtools.pyc
config.pyc      libcbcf.so      libcsamfile.so      libcvcf.pxd     tabix_util.h
diekhans commented 7 years ago

@maximilianpress , this is the same thing I am seeing. libcutils.so is reference libsamtools.so, which can't be found because it is looking for @rpath/pysam/libcsamtools.so. This has something to do with the ay libcutils.so was built, most likely using the wrong -rpath value.

bioinformed commented 7 years ago

@diekhans, @maximilianpress: Thanks for the interest in tracking down the source of this bug. For more insight into what is happening, look at https://github.com/pysam-developers/pysam/blob/master/cy_build.py . Therein is some hairy code that attempts to infer which one of the multitude of installation methods and path adjustments to use to set the correct rpath. I've been looking into how to improve our logic to properly set rpath, but every potential fix I've found ends up breaking other installation methods.

The best workaround is to install from a wheel file. That case is working. Once @AndreasHeger posts an OSX wheel file (hint hint) then installation from pypi will work once more.

kyleabeauchamp commented 7 years ago

Have you tried the bioconda builds on Sierra? Worth a try, it does some funky rpath modifying stuff that may or may not be different.

bioinformed commented 7 years ago

@kyleabeauchamp: I've reviewed the bioconda modifications and it works for how bioconda installs packages, but is not a general fix since it breaks other installation modes.

splaisan commented 7 years ago

pysam failure to install prevents me to install CrossMap

can you please help me fix this?

sudo -H pip install Crossmap --upgrade Collecting Crossmap Using cached CrossMap-0.2.6-py2-none-any.whl Collecting pysam (from Crossmap) Using cached pysam-0.11.2.2.tar.gz Exception: Traceback (most recent call last): File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/Library/Python/2.7/site-packages/pip/commands/install.py", line 324, in run requirement_set.prepare_files(finder) File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 380, in prepare_files ignore_dependencies=self.ignore_dependencies)) File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 634, in _prepare_file abstract_dist.prep_for_dist() File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 129, in prep_for_dist self.req_to_install.run_egg_info() File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 412, in run_egg_info self.setup_py, self.name, File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 387, in setup_py import setuptools # noqa File "/Library/Python/2.7/site-packages/setuptools/init.py", line 12, in import setuptools.version File "/Library/Python/2.7/site-packages/setuptools/version.py", line 1, in import pkg_resources File "/Library/Python/2.7/site-packages/pkg_resources/init.py", line 72, in import packaging.requirements File "/Library/Python/2.7/site-packages/packaging/requirements.py", line 59, in MARKER_EXPR = originalTextFor(MARKER_EXPR())("marker") TypeError: call() takes exactly 2 arguments (1 given)

bioinformed commented 7 years ago

@splaisan: This appears to be an unrelated issue and likely not one with pysam. Can you please open a new issue so that we can track the status properly?

AndreasHeger commented 7 years ago

... will look into the OsX wheels.

diekhans commented 7 years ago

I am trying to dig into this a bit more. It doesn't fail on a pip install to a virtualenv, which I found off.

In both a system and virtualenv install, libcsamtools.so is in: lib/python2.7/site-packages/pysam/libcsamtools.so

In virtualenv, objdump -p shows libcutils.so has the dependency specified as @rpath/pysam/libcsamtools.so

while in the system install, it's specified as @rpath/pysam-0.11.2.2-py2.7-macosx-10.12-x86_64.egg/pysam/libcsamtools.so

All I have time for now; it's unclear why @path is being set to the wrong location here. Can you explain the details?

splaisan commented 7 years ago

Dear,

First thanks a lot for your efforts to help. I remember I have once set virtualenv on my mac but found it too complicated to keep and this has left scars which from time to time prevent me from maintaining the system. I also used mac port to install more recent than system versions of python and this is also likely involved. I now just set python to the apple build but it did not help in this case.

I am not enough of a python guy to fix these and would really like to perform a clean install of python and remove all non OS traces of it to fix these but fear to break the apple python by doing so. most of the python pmackages I have installed do work, it is only a few like this one which are bad. I think time has come for a full clean install of my mac….

Sorry for this issue which is clearly linked to my computer settings.

Best Stephane

Stéphane Plaisance – Staff Scientist | Bioinformatician VIB Nucleomics Core
Campus Gasthuisberg Herestraat 49 – Post Box 816 – 3000 Leuven – Belgium O&N4 Building – 8th Floor – Room 08.440 Tel. +32 16 37 31 26
Lync. +32 16 32 00 60
www.nucleomics.be  http://www.nucleomics.be/ http://www.nucleomics.be/ http://www.nucleomics.be/ http://www.nucleomics.be/ http://www.vib.be/

On 16 Jun 2017, at 17:18, Mark Diekhans notifications@github.com wrote:

I am trying to dig into this a bit more. It doesn't fail on a pip install to a virtualenv, which I found off.

In both a system and virtualenv install, libcsamtools.so is in: lib/python2.7/site-packages/pysam/libcsamtools.so

In virtualenv, objdump -p shows libcutils.so has the dependency specified as @rpath/pysam/libcsamtools.so

while in the system install, it's specified as @rpath/pysam-0.11.2.2-py2.7-macosx-10.12-x86_64.egg/pysam/libcsamtools.so

All I have time for now; it's unclear why @path https://github.com/path is being set to the wrong location here. Can you explain the details?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pysam-developers/pysam/issues/475#issuecomment-309054244, or mute the thread https://github.com/notifications/unsubscribe-auth/AA0ZlNr-AYe0c9Vef8YvGD-e79mYVPCzks5sEpy7gaJpZM4Nu9wO.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/pysam-developers/pysam","title":"pysam-developers/pysam","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/pysam-developers/pysam"}},"updates":{"snippets":[{"icon":"PERSON","message":"@diekhans in #475: I am trying to dig into this a bit more. It doesn't fail on a pip install to a virtualenv, which I found\r\noff. \r\n\r\nIn both a system and virtualenv install, libcsamtools.so is in: lib/python2.7/site-packages/pysam/libcsamtools.so\r\n\r\nIn virtualenv, objdump -p shows libcutils.so has the dependency specified as @rpath/pysam/libcsamtools.so\r\n\r\nwhile in the system install, it's specified as @rpath/pysam-0.11.2.2-py2.7-macosx-10.12-x86_64.egg/pysam/libcsamtools.so\r\n\r\nAll I have time for now; it's unclear why @path is being set to the wrong location here. Can you explain the details?\r\n\r\n\r\n\r\n\r\n\r\n"}],"action":{"name":"View Issue","url":"https://github.com/pysam-developers/pysam/issues/475#issuecomment-309054244"}}}

splaisan commented 7 years ago

I made one step progress by aliasing the macport lzma and lzma.h dependencies from /opt/local/include to /usr/local/include (they were absent from the /usr/path) After that pip install pysam worked as well as pip install CrossMap I am now stuck when running CrossMap with: CrossMap.py

Traceback (most recent call last):
  File "/usr/local/bin/CrossMap.py", line 22, in <module>
    import pysam
  File "/Library/Python/2.7/site-packages/pysam/__init__.py", line 5, in <module>
    from pysam.libchtslib import *
  File "pysam/libchtslib.pyx", line 1, in init pysam.libchtslib (pysam/libchtslib.c:12515)
ImportError: dlopen(/Library/Python/2.7/site-packages/pysam/libcutils.so, 2): Library not loaded: @rpath/pysam-0.11.2.2-py2.7-macosx-10.10-intel.egg/pysam/libcsamtools.so
  Referenced from: /Library/Python/2.7/site-packages/pysam/libcutils.so
  Reason: image not found

which I find elsewhere too (@maximilianpress). Any idea how I can do better than aliasing the lzma dependencies, reinstall all and get it running? I would prefer a solution without additional package installers (bioconda) as my system get too complex with these extra layers.

The pysam package folder is there and contains the expected libraries (I think?)

$ ll /Library/Python/2.7/site-packages/pysam
total 11M
drwxr-xr-x  51 root wheel  1.7K Jun 26 15:13 .
drwxr-xr-x 196 root wheel  6.6K Jun 26 15:18 ..
-rw-r--r--   1 root wheel  8.2K Oct 31  2016 Pileup.py
-rw-r--r--   1 root wheel  7.6K Jun 26 15:13 Pileup.pyc
-rw-r--r--   1 root wheel  3.1K Apr 11 20:08 __init__.py
-rw-r--r--   1 root wheel  2.8K Jun 26 15:13 __init__.pyc
-rw-r--r--   1 root wheel   428 Oct 31  2016 bcftools.py
-rw-r--r--   1 root wheel   577 Jun 26 15:13 bcftools.pyc
-rw-r--r--   1 root wheel   102 Apr 11 20:08 cbcftools_util.h
-rw-r--r--   1 root wheel   138 Jun 26 15:10 config.py
-rw-r--r--   1 root wheel   350 Jun 26 15:13 config.pyc
-rw-r--r--   1 root wheel   102 Apr 11 20:08 csamtools_util.h
-rw-r--r--   1 root wheel  3.0K Apr 11 20:08 htslib_util.h
drwxr-xr-x   7 root wheel   238 Jun 26 15:13 include
-rw-r--r--   1 root wheel  2.7K Apr 11 20:08 libcalignedsegment.pxd
-rwxr-xr-x   1 root wheel 1006K Jun 26 15:11 libcalignedsegment.so
-rw-r--r--   1 root wheel  3.5K Apr  9 21:52 libcalignmentfile.pxd
-rwxr-xr-x   1 root wheel  806K Jun 26 15:11 libcalignmentfile.so
-rw-r--r--   1 root wheel  4.0K May 12 20:59 libcbcf.pxd
-rwxr-xr-x   1 root wheel  1.5M Jun 26 15:12 libcbcf.so
-rw-r--r--   1 root wheel    84 Apr 11 20:08 libcbcftools.pxd
-rwxr-xr-x   1 root wheel  1.5M Jun 26 15:13 libcbcftools.so
-rwxr-xr-x   1 root wheel  150K Jun 26 15:12 libcbgzf.so
-rw-r--r--   1 root wheel  2.1K Oct 31  2016 libcfaidx.pxd
-rwxr-xr-x   1 root wheel  245K Jun 26 15:12 libcfaidx.so
-rw-r--r--   1 root wheel  102K May 12 20:59 libchtslib.pxd
-rwxr-xr-x   1 root wheel  1.5M Jun 26 15:11 libchtslib.so
-rw-r--r--   1 root wheel  1.5K Oct 31  2016 libcsamfile.pxd
-rwxr-xr-x   1 root wheel   83K Jun 26 15:11 libcsamfile.so
-rw-r--r--   1 root wheel    84 Apr 11 20:08 libcsamtools.pxd
-rwxr-xr-x   1 root wheel  772K Jun 26 15:12 libcsamtools.so
-rw-r--r--   1 root wheel  2.6K May 12 20:59 libctabix.pxd
-rwxr-xr-x   1 root wheel  419K Jun 26 15:11 libctabix.so
-rw-r--r--   1 root wheel  1.3K May 12 20:59 libctabixproxies.pxd
-rwxr-xr-x   1 root wheel  465K Jun 26 15:11 libctabixproxies.so
-rw-r--r--   1 root wheel  1.6K Apr 11 20:08 libcutils.pxd
-rwxr-xr-x   1 root wheel  264K Jun 26 15:13 libcutils.so
-rw-r--r--   1 root wheel     2 Oct 31  2016 libcvcf.pxd
-rwxr-xr-x   1 root wheel  1.3M Jun 26 15:12 libcvcf.so
-rw-r--r--   1 root wheel  5.0K Oct 31  2016 namedtuple.py
-rw-r--r--   1 root wheel  4.9K Jun 26 15:13 namedtuple.pyc
-rw-r--r--   1 root wheel   255 Oct 31  2016 pysam_stream.h
-rw-r--r--   1 root wheel   895 Apr 11 20:08 pysam_util.h
-rw-r--r--   1 root wheel    80 Apr 11 20:08 samfile_util.h
-rw-r--r--   1 root wheel  1.4K Oct 31  2016 samtools.py
-rw-r--r--   1 root wheel  1.5K Jun 26 15:13 samtools.pyc
-rw-r--r--   1 root wheel   218 Oct 31  2016 tabix_util.h
-rw-r--r--   1 root wheel  3.2K Apr 11 20:08 utils.py
-rw-r--r--   1 root wheel  3.7K Jun 26 15:13 utils.pyc
-rw-r--r--   1 root wheel   205 Jun  8 21:02 version.py
-rw-r--r--   1 root wheel   279 Jun 26 15:13 version.pyc
AndreasHeger commented 7 years ago

@splaisan , the build directory looks good. I assume this issue is related to #489 ?

oscarestrada05 commented 7 years ago

Same error: Error: Could not import required module 'pysam':

$ pip install pysam Requirement already satisfied: pysam in /usr/local/lib/python2.7/site-packages

splaisan commented 7 years ago

Digging in it from a separate post I ran the next command to identify what libcutils.so relies on. Does it help the experts among us fix the @rpath issue ?

otool -L /Library/Python/2.7/site-packages/pysam/libcutils.so
/Library/Python/2.7/site-packages/pysam/libcutils.so:
    @rpath/pysam-0.11.2.2-py2.7-macosx-10.10-intel.egg/pysam/libcutils.so (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
    /usr/lib/liblzma.5.dylib (compatibility version 6.0.0, current version 6.3.0)
    /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.5)
    /usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 8.0.0)
    @rpath/pysam-0.11.2.2-py2.7-macosx-10.10-intel.egg/pysam/libchtslib.so (compatibility version 0.0.0, current version 0.0.0)
    @rpath/pysam-0.11.2.2-py2.7-macosx-10.10-intel.egg/pysam/libcsamtools.so (compatibility version 0.0.0, current version 0.0.0)
    @rpath/pysam-0.11.2.2-py2.7-macosx-10.10-intel.egg/pysam/libcbcftools.so (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

importing pysam from within python confirms

Python 2.7.10 (default, Jul 14 2015, 19:46:27) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pysam
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pysam/__init__.py", line 5, in <module>
    from pysam.libchtslib import *
  File "pysam/libchtslib.pyx", line 1, in init pysam.libchtslib (pysam/libchtslib.c:12515)
ImportError: dlopen(pysam/libcutils.so, 2): Library not loaded: @rpath/pysam-0.11.2.2-py2.7-macosx-10.10-intel.egg/pysam/libcsamtools.so
  Referenced from: /Library/Python/2.7/site-packages/pysam/libcutils.so

As said above by @diekhans the @rpath for the system is wrong as there is no folder called pysam-0.11.2.2-py2.7-macosx-10.10-intel.egg in there. I created it as alias to pysam but it does not improve. Could better aliasing elsewhere fix this issue? Where is @rpath on my machine? how can I get/modify it?

Is install_name_tool the way to go and HOW? (quite cryptic to me)?

ChrisHIV commented 7 years ago

Sounds like this is my error too. On a new Mac:

brew install samtools
brew install python
sudo easy_install pip
sudo -H pip install pysam --upgrade

Then in python

import pysam
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/pysam/__init__.py", line 5, in <module>
    from pysam.libchtslib import *
  File "pysam/libchtslib.pyx", line 1, in init pysam.libchtslib (pysam/libchtslib.c:12515)
ImportError: dlopen(/Library/Python/2.7/site-packages/pysam/libcutils.so, 2): Library not loaded: @rpath/pysam-0.11.2.2-py2.7-macosx-10.12-intel.egg/pysam/libcsamtools.so
  Referenced from: /Library/Python/2.7/site-packages/pysam/libcutils.so
  Reason: image not found
terrycojones commented 7 years ago

I just ran into this too (OS X 10.10.5, python 3.5). I got pysam working (in a virtualenv) by:

  1. Remove anything in the virtualenv installed by pip install pysam (pip uninstall pysam might be enough).
  2. pip install cython (if you don't have it already)
  3. git clone https://github.com/pysam-developers/pysam.git
  4. cd pysam
  5. python setup.py build
  6. python setup.py install

Then python -c 'import pysam' does the import without the error mentioned originally above.

splaisan commented 7 years ago

Today i found a new version of pysam (0.12) which built OK under Yosemite. I could finally build CrossMap too. Although I did not test the commands more than printing the help, the building issue at least seems to be fixed. Thanks for this new release.

AndreasHeger commented 6 years ago

Thanks, closing this for now.