bam2stats error - AttributeError: module 'pysam.libcalignmentfile' has no attribute 'PileupColumn'

[lucygarner]

Hi,

I installed cgat-apps in a fresh Conda environment as follows:

conda create -n cgat-apps
conda activate cgat-apps
conda install -c conda-forge -c bioconda cgat-apps

I did not use mamba as this gave me an error (mamba installs python 3.9.7 by default):

Looking for: ['cgat-apps']

bioconda/noarch          [====================] (00m:00s) Done
pkgs/main/linux-64       [====================] (00m:00s) Done
bioconda/linux-64        [====================] (00m:01s) Done
pkgs/r/noarch            [====================] (00m:00s) Done
pkgs/r/linux-64          [====================] (00m:00s) Done
pkgs/main/noarch         [====================] (00m:00s) Done
conda-forge/noarch       [====================] (00m:01s) Done
conda-forge/linux-64     [====================] (00m:05s) Done

Pinned packages:
  - python 3.9.*

Encountered problems while solving:
  - package cgat-apps-0.6.4-py38hc5b2f15_0 requires python >=3.8,<3.9.0a0, but none of the providers can be installed

When trying to run bam2stats, I get the following error:

Traceback (most recent call last):
  File "/home/lucy/conda/envs/cgat-apps/bin/cgat", line 11, in <module>
    sys.exit(main())
  File "/home/lucy/conda/envs/cgat-apps/lib/python3.8/site-packages/cgat/cgat.py", line 129, in main
    module = imp.load_module(command, file, pathname, description)
  File "/home/lucy/conda/envs/cgat-apps/lib/python3.8/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/home/lucy/conda/envs/cgat-apps/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/lucy/conda/envs/cgat-apps/lib/python3.8/site-packages/cgat/tools/bam2stats.py", line 315, in <module>
    from cgat.BamTools.bamtools import bam2stats_count
  File "cgat/BamTools/bamtools.pyx", line 1, in init cgat.BamTools.bamtools
AttributeError: module 'pysam.libcalignmentfile' has no attribute 'PileupColumn'

Do I require a different version of pysam?

conda list:

# packages in environment at /home/lucy/conda/envs/cgat-apps:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
aiohttp                   3.7.4.post0      py38h497a2fe_0    conda-forge
alignlib-lite             0.3              py38h9b08285_4    bioconda
apsw                      3.36.0.r1        py38he41bca5_0    conda-forge
async-timeout             3.0.1                   py_1000    conda-forge
attrs                     21.2.0             pyhd8ed1ab_0    conda-forge
bcrypt                    3.2.0            py38h497a2fe_1    conda-forge
bedtools                  2.30.0               h7d7f7ad_2    bioconda
biopython                 1.79             py38h497a2fe_0    conda-forge
boto3                     1.19.2             pyhd8ed1ab_0    conda-forge
botocore                  1.22.2             pyhd8ed1ab_0    conda-forge
brotlipy                  0.7.0           py38h497a2fe_1001    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.17.2               h7f98852_0    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
cachetools                4.2.4              pyhd8ed1ab_0    conda-forge
certifi                   2021.10.8        py38h578d9bd_0    conda-forge
cffi                      1.14.6           py38h3931269_1    conda-forge
cgat-apps                 0.6.4            py38hc5b2f15_1    bioconda
cgatcore                  0.6.9              pyhdfd78af_0    bioconda
chardet                   4.0.0            py38h578d9bd_1    conda-forge
charset-normalizer        2.0.0              pyhd8ed1ab_0    conda-forge
coreutils                 8.31                 h516909a_0    conda-forge
cryptography              35.0.0           py38h3e25421_1    conda-forge
cycler                    0.10.0                     py_2    conda-forge
drmaa                     0.7.9                   py_1000    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
ftputil                   5.0.1              pyhd8ed1ab_0    conda-forge
future                    0.18.2           py38h578d9bd_3    conda-forge
gevent                    21.8.0           py38h497a2fe_0    conda-forge
google-api-core           1.31.2             pyhd8ed1ab_0    conda-forge
google-auth               1.35.0             pyh6c4a22f_0    conda-forge
google-cloud-core         1.7.1              pyhd3eb1b0_0  
google-cloud-sdk          361.0.0          py38h578d9bd_0    conda-forge
google-cloud-storage      1.19.0                     py_0    conda-forge
google-crc32c             1.1.2            py38h8838a9a_0    conda-forge
google-resumable-media    2.0.3              pyh6c4a22f_0    conda-forge
googleapis-common-protos  1.53.0           py38h578d9bd_0    conda-forge
greenlet                  1.1.2            py38h709712a_0    conda-forge
grep                      3.4                  h9d02d08_1    bioconda
grpcio                    1.41.0           py38hdd6454d_0    conda-forge
htslib                    1.12                 h9093b5e_1    bioconda
idna                      3.1                pyhd3deb0d_0    conda-forge
jbig                      2.1               h7f98852_2003    conda-forge
jinja2                    3.0.2              pyhd8ed1ab_0    conda-forge
jmespath                  0.10.0             pyh9f0ad1d_0    conda-forge
joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
jpeg                      9d                   h36c2ea0_0    conda-forge
kiwisolver                1.3.2            py38h1fd1430_0    conda-forge
krb5                      1.19.2               hcc1bbae_2    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      2.2.1                h9c3ff4c_0    conda-forge
libblas                   3.9.0           12_linux64_openblas    conda-forge
libcblas                  3.9.0           12_linux64_openblas    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcurl                   7.79.1               h2574ce0_1    conda-forge
libdeflate                1.7                  h7f98852_5    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h9c3ff4c_4    conda-forge
libgcc-ng                 11.2.0              h1d223b6_11    conda-forge
libgfortran-ng            11.2.0              h69a702a_11    conda-forge
libgfortran5              11.2.0              h5c6108e_11    conda-forge
libgomp                   11.2.0              h1d223b6_11    conda-forge
liblapack                 3.9.0           12_linux64_openblas    conda-forge
libnghttp2                1.43.0               h812cca2_1    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.18          pthreads_h8fe5266_0    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libprotobuf               3.18.1               h780b84a_0    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libssh2                   1.10.0               ha56f1ee_2    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_11    conda-forge
libtiff                   4.3.0                hf544144_1    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libuv                     1.42.0               h7f98852_0    conda-forge
libwebp-base              1.2.1                h7f98852_0    conda-forge
libzlib                   1.2.11            h36c2ea0_1013    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
markupsafe                2.0.1            py38h497a2fe_0    conda-forge
matplotlib-base           3.4.3            py38hf4fb855_1    conda-forge
multidict                 5.2.0            py38h497a2fe_0    conda-forge
mysql-connector-c         6.1.11            h6eb9d5d_1007    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
nomkl                     1.0                  h5ca1d4c_0    conda-forge
numpy                     1.21.3           py38he2449b9_0    conda-forge
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openssl                   1.1.1l               h7f98852_0    conda-forge
packaging                 21.0               pyhd8ed1ab_0    conda-forge
pandas                    1.3.4            py38h43a58ef_0    conda-forge
paramiko                  2.8.0              pyhd8ed1ab_0    conda-forge
pcre                      8.45                 h9c3ff4c_0    conda-forge
pillow                    8.3.2            py38h8e6f84c_0    conda-forge
pip                       21.3.1             pyhd8ed1ab_0    conda-forge
protobuf                  3.18.1           py38h709712a_0    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pybedtools                0.8.2            py38h69e0bdc_1    bioconda
pybigwig                  0.3.18           py38h5ebd311_1    bioconda
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pynacl                    1.4.0            py38h497a2fe_2    conda-forge
pyopenssl                 21.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pysam                     0.17.0           py38hf7546f9_0    bioconda
pysftp                    0.2.9                      py_1    conda-forge
pysocks                   1.7.1            py38h578d9bd_3    conda-forge
python                    3.8.12          hb7a2778_2_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.8                      2_cp38    conda-forge
pytz                      2021.3             pyhd8ed1ab_0    conda-forge
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
pyyaml                    6.0              py38h497a2fe_0    conda-forge
quicksect                 0.2.2            py38h4a8c8d9_4    bioconda
readline                  8.1                  h46c0cb4_0    conda-forge
requests                  2.26.0             pyhd8ed1ab_0    conda-forge
rsa                       4.7.2              pyh44b312d_0    conda-forge
ruffus                    2.8.4              pyh864c0ab_1    bioconda
s3transfer                0.5.0              pyhd8ed1ab_0    conda-forge
scikit-learn              1.0              py38hacb3eff_1    conda-forge
scipy                     1.7.1            py38h56a6a73_0    conda-forge
setuptools                58.2.0           py38h578d9bd_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
sqlalchemy                1.4.26           py38h497a2fe_0    conda-forge
sqlite                    3.36.0               h9cd32fc_2    conda-forge
threadpoolctl             3.0.0              pyh8a188c0_0    conda-forge
time                      1.8                  h516909a_0    conda-forge
tk                        8.6.11               h27826a3_1    conda-forge
tornado                   6.1              py38h497a2fe_1    conda-forge
typing-extensions         3.10.0.2             hd8ed1ab_0    conda-forge
typing_extensions         3.10.0.2           pyha770c72_0    conda-forge
ucsc-bedgraphtobigwig     377                  h0b8a92a_2    bioconda
ucsc-bedtobigbed          377                  h0b8a92a_2    bioconda
ucsc-wigtobigwig          377                  h0b8a92a_2    bioconda
urllib3                   1.26.7             pyhd8ed1ab_0    conda-forge
wheel                     0.37.0             pyhd8ed1ab_1    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h516909a_0    conda-forge
yarl                      1.7.0            py38h497a2fe_0    conda-forge
zlib                      1.2.11            h36c2ea0_1013    conda-forge
zope.event                4.5.0              pyh9f0ad1d_0    conda-forge
zope.interface            5.4.0            py38h497a2fe_0    conda-forge
zstd                      1.5.0                ha95c52a_0    conda-forge

[Acribbs]

So to answer your first issue regarding the conda install, what do you have in your base environment? Do you have python installed? In your base environment you should only have mamba installed and nothing else.

For the pysam issue, it may take me a little time to debug, do you have the command and the data that you are using to run bam2stats?

[lucygarner]

Thanks @Acribbs. My revision 0 of the base environment seems to contain python, and if I try to remove python, it will also remove conda. Is there any way I can easily get back to a "clean" base environment?

The command that I used was: cgat bam2stats --force-output --output-filename-pattern=bam2stats.dir/sample_1.bamstats < mark_duplicates.dir/sample_1.dupmarked.bam > bam2stats.dir/sample_1.bamstats

Note: I wrote this command for a pipeline a few years ago now, so I hope the format is still correct!

I can send you one of my example BAM files.

[Acribbs]

Ah damn, unfortunately there isn't a good way to clean the base environment. The only reliable way is to reinstall miniconda.

Anything that is in your base environment usually gets copied to all your subsequent environment, so its pretty important not to install anything there

Command looks good, if you can send me the file I can try on my end.

Adam

[lucygarner]

Hi @Acribbs,

I just reinstalled miniconda, but python gets installed as a dependency.

This is what I ran:

curl -o Miniconda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh 
bash Miniconda.sh -b -p conda

And these are the packages installed:

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-4.5-1_gnu
  brotlipy           pkgs/main/linux-64::brotlipy-0.7.0-py39h27cfd23_1003
  ca-certificates    pkgs/main/linux-64::ca-certificates-2021.7.5-h06a4308_1
  certifi            pkgs/main/linux-64::certifi-2021.5.30-py39h06a4308_0
  cffi               pkgs/main/linux-64::cffi-1.14.6-py39h400218f_0
  chardet            pkgs/main/linux-64::chardet-4.0.0-py39h06a4308_1003
  conda              pkgs/main/linux-64::conda-4.10.3-py39h06a4308_0
  conda-package-han~ pkgs/main/linux-64::conda-package-handling-1.7.3-py39h27cfd23_1
  cryptography       pkgs/main/linux-64::cryptography-3.4.7-py39hd23ed53_0
  idna               pkgs/main/noarch::idna-2.10-pyhd3eb1b0_0
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.35.1-h7274673_9
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.3.0-h5101ec6_17
  libgomp            pkgs/main/linux-64::libgomp-9.3.0-h5101ec6_17
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.3.0-hd4cf53a_17
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1k-h27cfd23_0
  pip                pkgs/main/linux-64::pip-21.1.3-py39h06a4308_0
  pycosat            pkgs/main/linux-64::pycosat-0.6.3-py39h27cfd23_0
  pycparser          pkgs/main/noarch::pycparser-2.20-py_2
  pyopenssl          pkgs/main/noarch::pyopenssl-20.0.1-pyhd3eb1b0_1
  pysocks            pkgs/main/linux-64::pysocks-1.7.1-py39h06a4308_0
  python             pkgs/main/linux-64::python-3.9.5-h12debd9_4
  readline           pkgs/main/linux-64::readline-8.1-h27cfd23_0
  requests           pkgs/main/noarch::requests-2.25.1-pyhd3eb1b0_0
  ruamel_yaml        pkgs/main/linux-64::ruamel_yaml-0.15.100-py39h27cfd23_0
  setuptools         pkgs/main/linux-64::setuptools-52.0.0-py39h06a4308_0
  six                pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_0
  sqlite             pkgs/main/linux-64::sqlite-3.36.0-hc218d9a_0
  tk                 pkgs/main/linux-64::tk-8.6.10-hbc83047_0
  tqdm               pkgs/main/noarch::tqdm-4.61.2-pyhd3eb1b0_1
  tzdata             pkgs/main/noarch::tzdata-2021a-h52ac0ba_0
  urllib3            pkgs/main/noarch::urllib3-1.26.6-pyhd3eb1b0_1
  wheel              pkgs/main/noarch::wheel-0.36.2-pyhd3eb1b0_0
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  yaml               pkgs/main/linux-64::yaml-0.2.5-h7b6447c_0
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3

[sebastian-luna-valero]

Hi,

I am able to reproduce this with

cgat bam2stats -h

Full steps (after ensuring that my shell does not have conda already available):

mktemp -d
cd /tmp/<folder>
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b -p conda-install
conda install -c conda-forge mamba
mamba create -n cgat-apps -c conda-forge -c bioconda cgat-apps
conda activate cgat-apps
cgat bam2stats -h

This installs pysam-0.17.0 for me as well. If I do:

mamba install -c conda-forge -c bioconda 'pysam<0.17'

I get cgat bam2stats -h working again.

I hope it helps!

Best regards, Sebastian

[Acribbs]

Hi Sebastian,

Thanks for this, you beat me to it. I also found that the latest pysam build led to issues with bam2stats.

Now working out fi this is a cgat-apps issue or a pysam issue.

[lucygarner]

Thanks both!

[jmarshall]

AttributeError: module 'pysam.libcalignmentfile' has no attribute 'PileupColumn'

PileupColumn was moved from libcalignmentfile to libcalignedsegment some years ago, and since pysam-developers/pysam#1007 (first released in 0.17.0) it no longer appears in libcalignmentfile.pxd at all — hence this error. So this particular error would go away if cgat-apps was rebuilt against current pysam.

Moreover it appears that nothing in cgat-apps actually uses PileupColumn — at least, the text PileupColumn does not appear in any source file. However cgat/BamTools/bamtools.pyx has

from pysam.libcalignmentfile cimport *

which (if built against pysam 0.16.0.1 or earlier) is presumably what is causing the need for (an outdated version of) PileupColumn. So I suspect if you reduce the * to only import what you actually need from libcalignmentfile (and similarly in the other *.pyx files that have a similar overly wide cimport), this problem will disappear entirely (even if cgat-apps were rebuilt against 0.16.0.1 again).

[Acribbs]

Many thanks for your comment @jmarshall. I will make changes and go through the modifications made to pysam and hopefully catch other errors that may occur within our code. Thanks again!

[jmarshall]

Note that after PR #110 there are still several instances of cimport * elsewhere:

$ git grep 'cimport\ \*'
cgat/BamTools/bamtools.pyx:from pysam.libchtslib cimport *
cgat/BamTools/bamtools.pyx:from pysam.libcfaidx cimport *
cgat/BamTools/geneprofile.pyx:from pysam.libchtslib cimport *
cgat/BamTools/geneprofile.pyx:from pysam.libcalignmentfile cimport *
cgat/BamTools/peakshape.pyx:from pysam.libcalignmentfile cimport *
cgat/FastqTools/fastqtools.pyx:from pysam.libchtslib cimport *
cgat/GeneModelAnalysis.pyx:from pysam.libchtslib cimport *
cgat/GeneModelAnalysis.pyx:from pysam.libcalignmentfile cimport *
cgat/VCFTools/vcftools.pyx:from pysam.libchtslib cimport *
doc/developing.rst:      from csamtools cimport *

[Acribbs]

Hi @jmarshall, thanks and indeed I am aware but don't have time to change all the cimports yet so just wanted to fix this issue and fix the others at the weekend.