search

pysam-developers / pysam

Pysam is a Python package for reading, manipulating, and writing genomics data such as SAM/BAM/CRAM and VCF/BCF files. It's a lightweight wrapper of the HTSlib API, the same one that powers samtools, bcftools, and tabix.
https://pysam.readthedocs.io/en/latest/
MIT License
775 stars 274 forks source link

pysam-0.15.2: TypeError: 'output_dir' must be a string or None #759

Open mmokrejs opened 5 years ago

mmokrejs commented 5 years ago

Hi, Gentoo Linux and especially @SoapGentoo (now as @SoapZA ?) has put an effort in the past to compile&link pysam against site-wide htslib and samtools. Since pysam-0.12.x series the setup.py has been rewamped, which is fine. The past changes need were https://gitweb.gentoo.org/repo/gentoo.git/tree/sci-biology/pysam/pysam-0.12.0.1.ebuild?id=5283acc5b86973e40e3a455a1eeaff7b81aa3232 with https://gitweb.gentoo.org/repo/gentoo.git/tree/sci-biology/pysam/files/pysam-0.12-fix-buildsystem.patch?id=5283acc5b86973e40e3a455a1eeaff7b81aa3232 on top of the sources.

The patch is not applicable due to the complete rewamp as of now. The changes to the ebuild file (the parent process calling setup.py) are maybe still needed. Would you mind thinking of the differences between the 0.12.0.1 and 0.15.2 and adjusting the setup.py in a way it works out-of-the-box for everybody? I believe other Linux distros have same need to get rid of the bundled htslib/samtools stuff to avoid mess with runtime library incompatibilities.

Basically, setup.py should detect site-wide htslib and samtools and not rely on the bundled copies. They can be different, no matter if older or newer, simply, different.

Currently I get: 

>>> Emerging (1 of 1) sci-biology/pysam-0.15.2::science
 * pysam-0.15.2.tar.gz BLAKE2B SHA512 size ;-) ...                                                                                                                                                                                    [ ok ]
>>> Unpacking source...
>>> Unpacking pysam-0.15.2.tar.gz to /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work
>>> Source unpacked in /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work
>>> Preparing source in /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2 ...
>>> Source prepared.
>>> Configuring source in /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2 ...
>>> Source configured.
>>> Compiling source in /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2 ...
>>> Source compiled.
>>> Test phase [not enabled]: sci-biology/pysam-0.15.2

>>> Install pysam-0.15.2 into /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/image/ category sci-biology
 * python2_7: running distutils-r1_run_phase distutils-r1_python_install
python2.7 setup.py install --root=/scratch/var/tmp/portage/sci-biology/pysam-0.15.2/image/_python2.7
# pysam: cython is available - using cythonize if necessary
# pysam: htslib mode is external
# pysam: HTSLIB_CONFIGURE_OPTIONS=None
running install
running build
running build_py
creating /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/Pileup.py -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/bcftools.py -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/version.py -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/utils.py -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/config.py -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/__init__.py -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/samtools.py -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
creating /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include
copying pysam/include/__init__.py -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include
package init file 'samtools/__init__.py' not found (or not a regular file)
package init file 'bcftools/__init__.py' not found (or not a regular file)
package init file 'samtools/win32/__init__.py' not found (or not a regular file)
copying pysam/libcalignmentfile.pxd -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/libcutils.pxd -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/libcfaidx.pxd -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/libcbcftools.pxd -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/libcalignedsegment.pxd -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/libchtslib.pxd -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/libcsamfile.pxd -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/libctabix.pxd -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/libctabixproxies.pxd -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/libcvcf.pxd -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/libcbcf.pxd -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/libcsamtools.pxd -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/pysam_stream.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/csamtools_util.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/htslib_util.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/pysam_util.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
copying pysam/cbcftools_util.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam
creating /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/bam2bcf.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/stats_isize.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/bam.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/bedidx.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/bam_plbuf.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/samtools.pysam.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/bam_lpileup.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/version.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/tmp_file.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/config.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/sample.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/sam_opts.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/sam.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/sam_header.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/bam_endian.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
copying samtools/samtools.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools
creating /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/vcfbuf.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/convert.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/gvcf.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/bam2bcf.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/filter.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/rbuf.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/prob1.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/mw.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/hclust.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/tsv2vcf.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/call.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/smpl_ilist.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/vcmp.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/version.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/config.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/HMM.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/khash_str2str.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/kmin.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/ploidy.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/kheap.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/bam_sample.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/regidx.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/bin.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/bcftools.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
copying bcftools/bcftools.pysam.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/bcftools
creating /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools/win32
copying samtools/win32/zconf.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools/win32
copying samtools/win32/zlib.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools/win32
copying samtools/win32/xcurses.h -> /scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2-python2_7/lib/pysam/include/samtools/win32
running build_ext
skipping 'pysam/libchtslib.c' Cython extension (up-to-date)
building 'pysam.libchtslib' extension
Traceback (most recent call last):
  File "setup.py", line 439, in <module>
    dist = setup(**metadata)
  File "/usr/lib64/python2.7/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib64/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib64/python2.7/site-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/usr/lib64/python2.7/distutils/command/install.py", line 563, in run
    self.run_command('build')
  File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib64/python2.7/distutils/command/build.py", line 127, in run
    self.run_command(cmd_name)
  File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib64/python2.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/usr/lib64/python2.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/usr/lib64/python2.7/site-packages/Cython/Distutils/old_build_ext.py", line 194, in build_extensions
    self.build_extension(ext)
  File "/scratch/var/tmp/portage/sci-biology/pysam-0.15.2/work/pysam-0.15.2/cy_build.py", line 86, in build_extension
    build_ext.build_extension(self, ext)
  File "/usr/lib64/python2.7/distutils/command/build_ext.py", line 499, in build_extension
    depends=ext.depends)
  File "/usr/lib64/python2.7/distutils/ccompiler.py", line 566, in compile
    depends, extra_postargs)
  File "/usr/lib64/python2.7/distutils/ccompiler.py", line 329, in _setup_compile
    raise TypeError, "'output_dir' must be a string or None"
TypeError: 'output_dir' must be a string or None

It dies inside Gentoo code but is 

    # unbundle htslib
    export HTSLIB_MODE="external"
    export HTSLIB_INCLUDE_DIR="${EPREFIX}"/usr/include
    export HTSLIB_LIBRARY_DIR="${EPREFIX}"/usr/$(get_libdir)
    rm -r htslib || die

not possible anymore before calling setup.py?

brobr commented 5 years ago

This seems to be available:

from https://github.com/pysam-developers/pysam/blob/master/INSTALL

External

pysam can be combined with an externally installed htslib_ library. This is a good way to avoid duplication of libraries. To link against an externally installed library, set the environment variables HTSLIB_LIBRARY_DIR and HTSLIB_INCLUDE_DIR before installing::

export HTSLIB_LIBRARY_DIR=/usr/local/lib export HTSLIB_INCLUDE_DIR=/usr/local/include pip install pysam

Note that the location of the file :file:libhts.so needs to be known to the linker once you run pysam, for example by setting the environment-varirable LD_LIBRARY_PATH.

But: Is this also needed with python setup.py install instead of pip install pysam? That is: can we do without those export lines?

This because I do not see a different package being built when htslib and headers are installed in a way that fits the distribution set-up (i.e. configure should be able to find all required stuff)

EDIT: without native htslib installed pysam won't compile using above export statements, thus it works (but having the same versions of the libraries either installed or coming with pysam, a difference won't show in the installed pysam python-ware)