pipenv install trouble

[cmatKhan]

I am having trouble installing pybedtools using pipenv:

> pipenv install pybedtools 
Installing pybedtools...
Error:  An error occurred while installing pybedtools!
Error text: Collecting pybedtools
  Using cached pybedtools-0.9.0.tar.gz (12.5 MB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: six in ./.venv/lib/python3.9/site-packages (from pybedtools->-r /tmp/pipenv-bv7wz308-requirements/pipenv-8j872enn-requirement.txt (line 1)) (1.16.0)
Requirement already satisfied: pysam in ./.venv/lib/python3.9/site-packages (from pybedtools->-r /tmp/pipenv-bv7wz308-requirements/pipenv-8j872enn-requirement.txt (line 1)) (0.19.1)
Building wheels for collected packages: pybedtools
  Building wheel for pybedtools (setup.py): started
  Building wheel for pybedtools (setup.py): finished with status 'error'
  Running setup.py clean for pybedtools
Failed to build pybedtools
Installing collected packages: pybedtools
  Running setup.py install for pybedtools: started
  Running setup.py install for pybedtools: finished with status 'error'

  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [170 lines of output]
      running bdist_wheel
      The [wheel] section is deprecated. Use [bdist_wheel] instead.
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-39
      creating build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/logger.py -> build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/genome_registry.py -> build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/settings.py -> build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/__init__.py -> build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/helpers.py -> build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/bedtool.py -> build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/paths.py -> build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/parallel.py -> build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/stats.py -> build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/filenames.py -> build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/version.py -> build/lib.linux-x86_64-cpython-39/pybedtools
      creating build/lib.linux-x86_64-cpython-39/pybedtools/test
      copying pybedtools/test/test_issues.py -> build/lib.linux-x86_64-cpython-39/pybedtools/test
      copying pybedtools/test/test_contrib.py -> build/lib.linux-x86_64-cpython-39/pybedtools/test
      copying pybedtools/test/test_cbedtools.py -> build/lib.linux-x86_64-cpython-39/pybedtools/test
      copying pybedtools/test/test_pathlib.py -> build/lib.linux-x86_64-cpython-39/pybedtools/test
      copying pybedtools/test/__init__.py -> build/lib.linux-x86_64-cpython-39/pybedtools/test
      copying pybedtools/test/test_len_leak.py -> build/lib.linux-x86_64-cpython-39/pybedtools/test
      copying pybedtools/test/tfuncs.py -> build/lib.linux-x86_64-cpython-39/pybedtools/test
      copying pybedtools/test/test_iter.py -> build/lib.linux-x86_64-cpython-39/pybedtools/test
      copying pybedtools/test/regression_tests.py -> build/lib.linux-x86_64-cpython-39/pybedtools/test
      copying pybedtools/test/test_1.py -> build/lib.linux-x86_64-cpython-39/pybedtools/test
      copying pybedtools/test/test_gzip_support.py -> build/lib.linux-x86_64-cpython-39/pybedtools/test
      copying pybedtools/test/genomepy_integration.py -> build/lib.linux-x86_64-cpython-39/pybedtools/test
      copying pybedtools/test/test_helpers.py -> build/lib.linux-x86_64-cpython-39/pybedtools/test
      creating build/lib.linux-x86_64-cpython-39/pybedtools/contrib
      copying pybedtools/contrib/__init__.py -> build/lib.linux-x86_64-cpython-39/pybedtools/contrib
      copying pybedtools/contrib/plotting.py -> build/lib.linux-x86_64-cpython-39/pybedtools/contrib
      copying pybedtools/contrib/intersection_matrix.py -> build/lib.linux-x86_64-cpython-39/pybedtools/contrib
      copying pybedtools/contrib/long_range_interaction.py -> build/lib.linux-x86_64-cpython-39/pybedtools/contrib
      copying pybedtools/contrib/bigbed.py -> build/lib.linux-x86_64-cpython-39/pybedtools/contrib
      copying pybedtools/contrib/venn_maker.py -> build/lib.linux-x86_64-cpython-39/pybedtools/contrib
      copying pybedtools/contrib/bigwig.py -> build/lib.linux-x86_64-cpython-39/pybedtools/contrib
      creating build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/__init__.py -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      running egg_info
      creating pybedtools.egg-info
      writing pybedtools.egg-info/PKG-INFO
      writing dependency_links to pybedtools.egg-info/dependency_links.txt
      writing requirements to pybedtools.egg-info/requires.txt
      writing top-level names to pybedtools.egg-info/top_level.txt
      writing manifest file 'pybedtools.egg-info/SOURCES.txt'
      reading manifest file 'pybedtools.egg-info/SOURCES.txt'
      adding license file 'LICENSE.txt'
      writing manifest file 'pybedtools.egg-info/SOURCES.txt'
      /home/oguzkhan/code/callingCardsTools/.venv/lib/python3.9/site-packages/setuptools/command/build_py.py:153: SetuptoolsDeprecationWarning:     Installing 'pybedtools.include' as data is deprecated, please list it in `packages`.
          !!

          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'pybedtools.include' as an importable package,
          but it is not listed in the `packages` configuration of setuptools.

          'pybedtools.include' has been automatically added to the distribution only
          because it may contain data files, but this behavior is likely to change
          in future versions of setuptools (and therefore is considered deprecated).

          Please make sure that 'pybedtools.include' is included as a package by using
          the `packages` configuration field or the proper discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).

          You can read more about "package discovery" and "data files" on setuptools
          documentation page.

      !!

        check.warn(importable)
      copying pybedtools/_Window.pyx -> build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/cbedtools.cpp -> build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/cbedtools.pxd -> build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/cbedtools.pyx -> build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/featurefuncs.cpp -> build/lib.linux-x86_64-cpython-39/pybedtools
      copying pybedtools/featurefuncs.pyx -> build/lib.linux-x86_64-cpython-39/pybedtools
      creating build/lib.linux-x86_64-cpython-39/pybedtools/include
      copying pybedtools/include/bedFile.cpp -> build/lib.linux-x86_64-cpython-39/pybedtools/include
      copying pybedtools/include/fileType.cpp -> build/lib.linux-x86_64-cpython-39/pybedtools/include
      copying pybedtools/include/gzstream.cpp -> build/lib.linux-x86_64-cpython-39/pybedtools/include
      copying pybedtools/test/data/reads.bam -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/d.gff -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/CTCF_Kc_Bushey_2009.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/small.bam -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/hg19.gff -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/genome.fa -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/gdc.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/SuHw_Kc_Bushey_2009.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/bedpe2.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/dm3-chr2L-5M.gff.gz -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/gdc.sorted.bam -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/BEAF_Mbn2_Bushey_2009.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/test.fa.fai -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/a.links.html -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/Cp190_Mbn2_Bushey_2009.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/gdc.bam -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/Cp190_Kc_Bushey_2009.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/democonfig.yaml -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/y.bam -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/CTCF_Mbn2_Bushey_2009.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/vcf-stderr-test.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/gdc.gff.gz -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/small.fastq -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/a.igv_script -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/gdc.50.200.bam.bai -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/v.vcf -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/test_bedpe.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/x.bam -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/multibamcov_test.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/issue319.vcf.gz -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/vcf-stderr-test.vcf -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/mm9.bed12 -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/gdc.sorted.bam.bai -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/gdc.1.100.bam -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/test.fa -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/rmsk.hg18.chr21.small.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/c.gff -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/gdc.cram -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/164.gtf -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/gdc.1.100.bam.bai -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/dm3-chr2L-5M-invalid.gff.gz -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/a.bed.gz -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/tag_test1.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/hg38-problem.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/x.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/a.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/issue_121.bam -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/gdc.gff -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/venn.c.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/1000genomes-example.vcf -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/BEAF_Kc_Bushey_2009.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/issue319.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/exons.gff -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/gdc.othersort.bam -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/SuHw_Mbn2_Bushey_2009.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/tag_test2.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/snps.bed.gz -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/venn.b.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/gdc.50.200.bam -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/hg38-base.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/issue319.out.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/a.bed.gz.tbi -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/expand_test.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/test_tsses.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/bedpe.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/test_peaks.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/m1.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/rmsk.hg18.chr21.small.bed.gz -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      copying pybedtools/test/data/b.bed -> build/lib.linux-x86_64-cpython-39/pybedtools/test/data
      warning: build_py: byte-compiling is disabled, skipping.

      running build_ext
      building 'pybedtools.cbedtools' extension
      creating build/temp.linux-x86_64-cpython-39
      creating build/temp.linux-x86_64-cpython-39/pybedtools
      creating build/temp.linux-x86_64-cpython-39/pybedtools/include
      x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -Ipybedtools/include/ -I/home/oguzkhan/code/callingCardsTools/.venv/include -I/usr/include/python3.9 -c pybedtools/cbedtools.cpp -o build/temp.linux-x86_64-cpython-39/pybedtools/cbedtools.o
      pybedtools/cbedtools.cpp:35:10: fatal error: Python.h: No such file or directory
         35 | #include "Python.h"
            |          ^~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pybedtools
  error: subprocess-exited-with-error

  × python setup.py clean did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
         or: setup.py --help [cmd1 cmd2 ...]
         or: setup.py --help-commands
         or: setup.py cmd --help

      error: option --all not recognized
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed cleaning build dir for pybedtools
  error: subprocess-exited-with-error

  × Running setup.py install for pybedtools did not run successfully.
  │ exit code: 1
  ╰─> [50 lines of output]
      running install
      /home/oguzkhan/code/callingCardsTools/.venv/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      copying pybedtools/version.py -> build/lib.linux-x86_64-cpython-39/pybedtools
      running egg_info
      writing pybedtools.egg-info/PKG-INFO
      writing dependency_links to pybedtools.egg-info/dependency_links.txt
      writing requirements to pybedtools.egg-info/requires.txt
      writing top-level names to pybedtools.egg-info/top_level.txt
      reading manifest file 'pybedtools.egg-info/SOURCES.txt'
      adding license file 'LICENSE.txt'
      writing manifest file 'pybedtools.egg-info/SOURCES.txt'
      /home/oguzkhan/code/callingCardsTools/.venv/lib/python3.9/site-packages/setuptools/command/build_py.py:153: SetuptoolsDeprecationWarning:     Installing 'pybedtools.include' as data is deprecated, please list it in `packages`.
          !!

          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'pybedtools.include' as an importable package,
          but it is not listed in the `packages` configuration of setuptools.

          'pybedtools.include' has been automatically added to the distribution only
          because it may contain data files, but this behavior is likely to change
          in future versions of setuptools (and therefore is considered deprecated).

          Please make sure that 'pybedtools.include' is included as a package by using
          the `packages` configuration field or the proper discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).

          You can read more about "package discovery" and "data files" on setuptools
          documentation page.

      !!

        check.warn(importable)
      warning: build_py: byte-compiling is disabled, skipping.

      running build_ext
      building 'pybedtools.cbedtools' extension
      x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -Ipybedtools/include/ -I/home/oguzkhan/code/callingCardsTools/.venv/include -I/usr/include/python3.9 -c pybedtools/cbedtools.cpp -o build/temp.linux-x86_64-cpython-39/pybedtools/cbedtools.o
      pybedtools/cbedtools.cpp:35:10: fatal error: Python.h: No such file or directory
         35 | #include "Python.h"
            |          ^~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> pybedtools

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

✘ Installation Failed 

I'd appreciate any suggestions -- thank you!

[daler]

pybedtools needs compilation of some C++/Cython code. Do you happen to have Cython in that environment?

[cmatKhan]

I ended up switching to Poetry, rather than Pipenv, but the error message is the same. I do have cython in my environment:

> poetry show cython
name         : cython
version      : 0.29.30
description  : The Cython compiler for writing C extensions for the Python language.

It seems like this is the problem:

        pybedtools/cbedtools.cpp:35:10: fatal error: Python.h: No such file or directory
           35 | #include "Python.h"
              |          ^~~~~~~~~~
        compilation terminated.
        error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
        [end of output]

But, by the same token, it seems like if there was a missing file, there would be other issues reports. So, not sure.

[cmatKhan]

Ok -- my fault. I hadn't updated my python3-dev when i upgraded from 3.8 to 3.9...my bad. If anyone else happens to run into this, on a debian OS, do this:

sudo apt-get install python<version>-dev

eg, I am currently using python 3.9.13:

sudo apt-get install python3.9-dev