Error while running Concoct_covergae_table.py

[dannyionescu]

Hi. I am trying to use concoct. I am getting errors running the script concoct_coverage_table.py. If I run this in a python 2.7 environment (concoct) I am getting the following error:dionescu@allegro:/data/scratch/dionescu/IK/data/scratch/dionescu/IK/Concoct_binning$ concoct_coverage_table.py contigs_10k.bed mapping/issyk_kul_megahit.contigs.fa.IS.bam.bam > coverage_table.tsv Traceback (most recent call last): File "/home/dionescu/miniconda3/envs/concoct/bin/concoct_coverage_table.py", line 18, in import pandas as pd File "/home/dionescu/.local/lib/python2.7/site-packages/pandas/init.py", line 35, in "the C extensions first.".format(module)) ImportError: C extension: /home/dionescu/.local/lib/python2.7/site-packages/pandas/_libs/tslibs/conversion.so: undefined symbol: PyFPE_jbuf not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first. (concoct) dionescu@allegro:/data/scratch/dionescu/IK/data/scratch/dionescu/IK/Concoct_binning$ pip install --upgrade pandas If I am running it in a python 3.6.7 environment then the error is different: (Concoct) dionescu@allegro:/data/scratch/dionescu/IK/data/scratch/dionescu/IK/Concoct_binning$ concoct_coverage_table.py contigs_10k.bed mapping/issyk_kul_megahit.contigs.fa.IS.bam.bam > c overage_table.csv ERROR: fail to open index BAM file 'mapping/issyk_kul_megahit.contigs.fa.IS10_S3.bam.bam' Traceback (most recent call last): File "/home/dionescu/CONCOCT/scripts/concoct_coverage_table.py", line 77, in generate_input_table(args.bedfile, args.bamfiles, samplenames=samplenames) File "/home/dionescu/CONCOCT/scripts/concoct_coverage_table.py", line 28, in generate_input_table sys.stderr.write(out) TypeError: write() argument must be str, not bytes Please advise on how to solve this.

[alneberg]

Hello @dannyionescu,

I think the first error is due to incorrectly installed pandas. Reinstalling it with pip or conda should hopefully fix that.

The second error, I'm guessing is caused by specifying the wrong input files: ".bam.bam"?

[ReneKat]

Hi @alneberg, I am actually having this issue too. I created the concoct_env as specified and am using: python 3.6.7; pandas 0.25.3; numpy 1.17.3 Loaded packages in environment: packages in environment at /Users/Owl/miniconda3/envs/concoct_env: #

Name Version Build Channel

alabaster 0.7.12 py_0 conda-forge babel 2.7.0 py_0 conda-forge biopython 1.74 py36h0b31af3_0 conda-forge bzip2 1.0.8 h01d97ff_1 conda-forge ca-certificates 2019.9.11 hecc5488_0 conda-forge certifi 2019.9.11 py36_0 conda-forge cffi 1.13.2 py36h33e799b_0 conda-forge chardet 3.0.4 py36_1003 conda-forge concoct 0.4.1 py36hd1ccf6d_0 bioconda cryptography 2.8 py36hafa8578_0 conda-forge curl 7.65.3 h22ea746_0 conda-forge cython 0.29.14 py36h4a8c4bd_0 conda-forge docutils 0.15.2 py36_0 conda-forge gsl 2.2.1 h002c638_3
htslib 1.9 h3a161e8_7 bioconda idna 2.8 py36_1000 conda-forge imagesize 1.1.0 py_0 conda-forge jinja2 2.10.3 py_0 conda-forge joblib 0.14.0 py_0 conda-forge krb5 1.16.3 hcfa6398_1001 conda-forge libblas 3.8.0 11_openblas conda-forge libcblas 3.8.0 11_openblas conda-forge libcurl 7.65.3 h16faf7d_0 conda-forge libcxx 9.0.0 h89e68fa_1 conda-forge libdeflate 1.0 h1de35cc_1 bioconda libedit 3.1.20170329 hcfe32e1_1001 conda-forge libffi 3.2.1 1 bioconda libgfortran 3.0.1 0 conda-forge liblapack 3.8.0 11_openblas conda-forge libopenblas 0.3.6 hdc02c5d_2
libssh2 1.8.2 hcdc9a53_2 conda-forge llvm-openmp 9.0.0 h40edb58_0 conda-forge markupsafe 1.1.1 py36h0b31af3_0 conda-forge ncurses 6.1 h0a44026_1002 conda-forge nose 1.3.7 py36_1003 conda-forge numpy 1.17.3 py36hde6bac1_0 conda-forge openblas 0.2.20 8 conda-forge openssl 1.1.1d h0b31af3_0 conda-forge packaging 19.2 py_0 conda-forge pandas 0.25.3 py36h4f17bb1_0 conda-forge pip 19.3.1 py36_0 conda-forge pycparser 2.19 py36_1 conda-forge pygments 2.4.2 py_0 conda-forge pyopenssl 19.0.0 py36_0 conda-forge pyparsing 2.4.5 py_0 conda-forge pysocks 1.7.1 py36_0 conda-forge python 3.6.7 h4285619_1006 conda-forge python-dateutil 2.8.1 py_0 conda-forge pytz 2019.3 py_0 conda-forge readline 8.0 hcfe32e1_0 conda-forge requests 2.22.0 py36_1 conda-forge samtools 1.9 h8aa4d43_12 bioconda scikit-learn 0.21.3 py36hd4ffd6c_0 conda-forge scipy 1.3.1 py36hab3da7d_2 conda-forge setuptools 41.6.0 py36_1 conda-forge six 1.13.0 py36_0 conda-forge snowballstemmer 2.0.0 py_0 conda-forge sphinx 2.2.1 py_0 conda-forge sphinx_rtd_theme 0.4.3 py_0 conda-forge sphinxcontrib-applehelp 1.0.1 py_0 conda-forge sphinxcontrib-devhelp 1.0.1 py_0 conda-forge sphinxcontrib-htmlhelp 1.0.2 py_0 conda-forge sphinxcontrib-jsmath 1.0.1 py_0 conda-forge sphinxcontrib-qthelp 1.0.2 py_0 conda-forge sphinxcontrib-serializinghtml 1.1.3 py_0 conda-forge sqlite 3.30.1 h93121df_0 conda-forge tk 8.6.9 h2573ce8_1003 conda-forge urllib3 1.25.7 py36_0 conda-forge wheel 0.33.6 py36_0 conda-forge xz 5.2.4 h1de35cc_1001 conda-forge zlib 1.2.11 h0b31af3_1006 conda-forge

Here is my code: /Users/Owl/Git/CONCOCT/scripts/concoct_coverage_table.py contigs_10K.bed cSMMC_MH.srt.bam > cSMMC_coverage_table.tsv

ERROR: fail to open index BAM file 'cSMMC_MH.srt.bam' Traceback (most recent call last): File "/Users/Owl/Git/CONCOCT/scripts/concoct_coverage_table.py", line 91, in generate_input_table(args.bedfile, args.bamfiles, samplenames=samplenames) File "/Users/Owl/Git/CONCOCT/scripts/concoct_coverage_table.py", line 41, in generate_input_table sys.stderr.write(out) TypeError: write() argument must be str, not bytes

Any guidance would be appreciated, I'm a newby. Best, René

[alneberg]

Hi René!

I think important part of the error message is: ERROR: fail to open index BAM file 'cSMMC_MH.srt.bam'. Could you please make sure there are .bai-files for this .bam-files?

[ReneKat]

Hello @alneberg, Thank you so much for your quick reply! :) No, there are no .bai files. I used samtools to convert from .sam files...

[ReneKat]

I just indexed my .bam file using samtools: samtools index cSMMC_MH.srt.bam cSMMC_MH.srt.bam.bai and reran the code using the indexed bam file: /Users/Owl/Git/CONCOCT/scripts/concoct_coverage_table.py contigs_10K.bed cSMMC_MH.srt.bam.bai > cSMMC_coverage_table.tsv and got the same error message: [E::hts_hopen] Failed to open file cSMMC_MH.srt.bam.bai [E::hts_open_format] Failed to open file cSMMC_MH.srt.bam.bai ERROR: fail to open index BAM file 'cSMMC_MH.srt.bam.bai' Traceback (most recent call last): File "/Users/Owl/Git/CONCOCT/scripts/concoct_coverage_table.py", line 91, in generate_input_table(args.bedfile, args.bamfiles, samplenames=samplenames) File "/Users/Owl/Git/CONCOCT/scripts/concoct_coverage_table.py", line 41, in generate_input_table sys.stderr.write(out) TypeError: write() argument must be str, not bytes

[ReneKat]

Okay, got it! I misunderstood, it's not that the index file replace the .bam file it is only that they have to be in the pwd. So, after I created the .bai index of the .bam file, and reran the original command: /Users/Owl/Git/CONCOCT/scripts/concoct_coverage_table.py contigs_10K.bed cSMMC_MH.srt.bam > cSMMC_coverage_table.tsv it worked! Thank you for your time and patience. Best, René

[alneberg]

Yes, your last description is correct!

You mostly solved it yourself, Cheers