Open palomo11 opened 6 years ago
Thank you for your feedback. This is definitely a problem I need to fix. However it is tricky for me to diagnose because there are a couple different things going on. Do you mind providing some more information?
Can you please provide the full standard out (the stuff metaWRAP prints to screen) of when you run metawrap binning
with the --maxbin2
and --concoct
options? (you can run with a subset of your data so you don't have to rerun anything)
metawrap binning --concoct
Crash after depth file is produced:
head concoct_depth.txt contig-1 53.4787 0.117991 0.0328043 contig-2 1.32157 2.16182 1.39231 contig-3 18.5849 0.246922 0.120634
Attached the standard out
Can you please set the following variable export LD_LIBRARY_PATH=/your/path/to/miniconda2/lib/:$LD_LIBRARY_PATH
and then re-run your concoct test? Don't forget to change the the bath to your actual miniconda2/lib path.
Also, can you please run conda list
and give me the output? I am unable to replicate this error so I am trying to understand what is different between our systems.
conda list:
MaxBin2 is still running.
I will also try now your suggestion about set the variable
Thanks!
metawrap binning --MaxBin2
This time it seems it has worked. The difference with the previous run was: Here just running MaxBin2, min contig 2000 instead of 1000, and 3 samples instead of 24. It has produced 132 bins. But still in the standard out it says:
Finished
readline() on closed filehandle FILE at /home/people/alpal/.conda/envs/metawrap-env/bin/run_MaxBin.pl line 1334.
Here you can see also the end of the log file
....
========== Job finished ==========
Yielded 132 bins for contig (scaffold) file binning_max/work_files/assembly.fa
Here are the output files for this run.
Please refer to the README file for further details.
Summary file: binning_max/work_files/maxbin2_out/bin.summary
Genome abundance info file: binning_max/work_files/maxbin2_out/bin.abundance
Marker counts: binning_max/work_files/maxbin2_out/bin.marker
Marker genes for each bin: binning_max/work_files/maxbin2_out/bin.marker_of_each_gene.tar.gz
Bin files: binning_max/work_files/maxbin2_out/bin.001.fasta - binning_max/work_files/maxbin2_out/bin.132.fasta
Unbinned sequences: binning_max/work_files/maxbin2_out/bin.noclass
========== Elapsed Time ==========
1 hours 13 minutes and 33 seconds.
------------------------------------------------------------------------------------------------------------------------
----- MaxBin2 finished successfully, and found 132 bins! -----
------------------------------------------------------------------------------------------------------------------------
########################################################################################################################
##### BINNING PIPELINE SUCCESSFULLY FINISHED!!! #####
########################################################################################################################
Fantastic, thank you! The problems with CONCOCT and MaxBin2 are two completely unrelated issues.
MaxBin2: It is unfortunate that you cannot reproduce the error with the subset data... Do you mind re-running the full thing and giving me the full stdout like you did for the concoct run? I can't patch it because it is unclear what happened exactly from the fragment you posted earlier.
CONCOCT: There is a problem with your versions of gcc and gcc libraries, so scipy fails to load/run. I cannot reproduce this problem on my end, so if the LD_LIBRARY_PATH trick doesn't do it, ill ask you to try a diagnose on your end. To start, can you give me the output of the following:
conda list
within the metawrap-envwhich gcc
gcc -v
echo $PATH
within the metawrap-envecho $LD_LIBRARY_PATH
within the metawrap-envNote: You might be able to diagnose this bug without actually running concoct all over again. Just enter python2.7, and tun __import__('pkg_resources').run_script('concoct==0.4.0', 'concoct')
(this should give you the same error). Not sure if it will work, however.
conda list within the metawrap-env conda_list_metaWRAP.txt
which gcc
~/.conda/envs/metawrap-env/bin/gcc
gcc -v
Reading specs from /home/people/name/.conda/envs/metawrap-env/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.8.5/specs
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/home/people/alpal/.conda/envs/metawrap-env/bin/../libexec/gcc/x86_64-unknown-linux-gnu/4.8.5/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: ./configure --prefix=/home/ray/mc-x64-2.7/conda-bld/gcc_1479223211463/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold --with-gxx-include-dir=/home/ray/mc-x64-2.7/conda-bld/gcc_1479223211463/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/gcc/include/c++ --bindir=/home/ray/mc-x64-2.7/conda-bld/gcc_1479223211463/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/bin --datarootdir=/home/ray/mc-x64-2.7/conda-bld/gcc_1479223211463/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/share --libdir=/home/ray/mc-x64-2.7/conda-bld/gcc_1479223211463/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib --with-gmp=/home/ray/mc-x64-2.7/conda-bld/gcc_1479223211463/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold --with-mpfr=/home/ray/mc-x64-2.7/conda-bld/gcc_1479223211463/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold --with-mpc=/home/ray/mc-x64-2.7/conda-bld/gcc_1479223211463/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold --with-isl=/home/ray/mc-x64-2.7/conda-bld/gcc_1479223211463/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold --with-cloog=/home/ray/mc-x64-2.7/conda-bld/gcc_1479223211463/_b_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold --enable-checking=release --with-tune=generic --disable-multilib
Thread model: posix
gcc version 4.8.5 (GCC)
python2.7
Python 2.7.11 |Continuum Analytics, Inc.| (default, Dec 6 2015, 18:08:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> __import__('pkg_resources').run_script('concoct==0.4.0', 'concoct')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 748, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1517, in run_script
exec(code, namespace, namespace)
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/concoct-0.4.0-py2.7-linux-x86_64.egg/EGG-INFO/scripts/concoct", line 10, in <module>
from concoct.cluster import cluster
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/concoct-0.4.0-py2.7-linux-x86_64.egg/concoct/cluster.py", line 4, in <module>
from sklearn.mixture import GMM
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/sklearn/__init__.py", line 134, in <module>
from .base import clone
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/sklearn/base.py", line 11, in <module>
from scipy import sparse
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/__init__.py", line 229, in <module>
from .csr import *
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/csr.py", line 15, in <module>
from ._sparsetools import csr_tocsc, csr_tobsr, csr_count_blocks, \
ImportError: /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/../../../../libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/_sparsetools.so)
In relation to export LD_LIBRARY_PATH=/your/path/to/miniconda2/lib/:$LD_LIBRARY_PATH Which path should I put there? I'm not sure about it.
OK, I'm going to run MaxBin2 with all the contigs and sample, I will let you know. By the way, I have just seen that I have run this before:
conda config --add channels ursky
Could that be an issue?
Thanks!
Update:
MaxBin2 run well with all contigs and samples. Besides running binning --maxbin2 alone, the only modification I did was to add module load idba_ud as it seems it appeared as an error last time (although I'm not sure for what purpose idba ud is used).
So any idea about how to solve concoct? Plase, have a look to what I copied you about gcc and I copy here what I said in the last comment:
In relation to export LD_LIBRARY_PATH=/your/path/to/miniconda2/lib/:$LD_LIBRARY_PATH Which path should I put there? I'm not sure about it.
By the way, I have just seen that I have run this before:
conda config --add channels ursky
Thank you very much for your help. I hope it can be solved and I can fully use metaWRAP.
That's great that MaxBin2 worked! Although its unfortunate we weren't able to catch why it failed that one time. If you ever run it again and it does't work, please copy paste the entire std out from metaWRAP so I can find out what happened.
For CONCOCT, it looks like your conda environment is identical to mine, which means that something about your system is messing with your conda's gcc... To try to fix it, find the full path of where your metawrap-env conda installation directory is, and then run export LD_LIBRARY_PATH=/yourfullpath/miniconda2/envs/metawrap-env/lib/:$LD_LIBRARY_PATH
.
For me, it was export LD_LIBRARY_PATH=/scratch/miniconda2/envs/metawrap-env/lib/:$LD_LIBRARY_PATH
. For you, I think it might be export LD_LIBRARY_PATH=/home/people/name/.conda/envs/metawrap-env/lib/:$LD_LIBRARY_PATH
, but check.
Run echo $LD_LIBRARY_PATH
to see if it worked. Then, in the same window, run python2.7
and then __import__('pkg_resources').run_script('concoct==0.4.0', 'concoct')
, just as you did before.
Don't worry, we will get it. Its apparently a common issue with conda packages.
Also, can you provide the outputs to the following (all within the metawrap-env):
echo $PATH
which python
which python2.7
which gcc
ldd /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/../../../../libstdc++.so.6
ldd /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/_sparsetools.so
export LD_LIBRARY_PATH=/home/people/name/.conda/envs/metawrap-env/lib/:$LD_LIBRARY_PATH
(metawrap-env) echo $LD_LIBRARY_PATH
/home/people/name/.conda/envs/metawrap-env/lib/:/usr/local/lib:/cm/local/apps/gcc/5.1.0/lib:/cm/local/apps/gcc/5.1.0/lib64
(metawrap-env) python2.7
Python 2.7.11 |Continuum Analytics, Inc.| (default, Dec 6 2015, 18:08:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> __import__('pkg_resources').run_script('concoct==0.4.0', 'concoct')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 748, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1517, in run_script
exec(code, namespace, namespace)
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/concoct-0.4.0-py2.7-linux-x86_64.egg/EGG-INFO/scripts/concoct", line 10, in <module>
from concoct.cluster import cluster
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/concoct-0.4.0-py2.7-linux-x86_64.egg/concoct/cluster.py", line 4, in <module>
from sklearn.mixture import GMM
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/sklearn/__init__.py", line 134, in <module>
from .base import clone
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/sklearn/base.py", line 11, in <module>
from scipy import sparse
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/__init__.py", line 229, in <module>
from .csr import *
File "/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/csr.py", line 15, in <module>
from ._sparsetools import csr_tocsc, csr_tobsr, csr_count_blocks, \
ImportError: /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/../../../../libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/_sparsetools.so)
echo $PATH
/home/people/name/.conda/envs/metawrap-env/bin:/services/tools/anaconda2/4.0.0/bin:/services/tools/ngsqctoolkit/2.3.3/scripts:/services/tools/perl/5.24.0/bin:/cm/shared/apps/xmsub/1.0/bin:/cm/shared/apps/moab/9.1.0.1/sbin:/cm/shared/apps/moab/9.1.0.1/bin:/home/people/name/perl5/bin:/cm/shared/apps/xqsub/1.0/bin:/cm/local/apps/gcc/5.1.0/bin:/usr/local/bin:/usr/local/sbin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/ibutils/bin:/sbin:/usr/sbin:/cm/local/apps/environment-modules/3.2.10/bin:/home/people/name/bin:/home/projects/env_10000/people/name/scripts
(metawrap-env) which python
~/.conda/envs/metawrap-env/bin/python
(metawrap-env) which python2.7
~/.conda/envs/metawrap-env/bin/python2.7
(metawrap-env) which gcc
~/.conda/envs/metawrap-env/bin/gcc
(metawrap-env) ldd /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/../../../../libstdc++.so.6
linux-vdso.so.1 => (0x00002aaaaaaab000)
libm.so.6 => /lib64/libm.so.6 (0x00002aaaaade8000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaab0eb000)
/lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
libgcc_s.so.1 => /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/../../../.././libgcc_s.so.1 (0x00002aaaab4ae000)
(metawrap-env) ldd /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/_sparsetools.so
/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/_sparsetools.so: /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/../../../../libstdc++.so.6: version **CXXABI_1.3.9' not found (required by /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/_sparsetools.so)**
/home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/_sparsetools.so: /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/../../../../libstdc++.so.6: version **GLIBCXX_3.4.21' not found (required by /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/_sparsetools.so)**
linux-vdso.so.1 => (0x00002aaaaaaab000)
libpython2.7.so.1.0 => /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/../../../../libpython2.7.so.1.0 (0x00002aaaaafea000)
libstdc++.so.6 => /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/../../../../libstdc++.so.6 (0x00002aaaab3d2000)
libgcc_s.so.1 => /home/people/name/.conda/envs/metawrap-env/lib/python2.7/site-packages/scipy/sparse/../../../../libgcc_s.so.1 (0x00002aaaab6e8000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaab922000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002aaaabce6000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaabf02000)
libutil.so.1 => /lib64/libutil.so.1 (0x00002aaaac106000)
libm.so.6 => /lib64/libm.so.6 (0x00002aaaac30a000)
/lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
Ok, I believe I might have found the issue! Patched in conda's metawrap-binning=0.6.4
.
Please delete your environment (just in case) conda remove --name metawrap-env --all
, and then remake the environment and re-install metawrap (make sure its v=0.6.4) as per installation instructions. Please let me know how it goes.
it worked fine!
Thank you very much for the help! Now running the next modules, I will let you know if there is any issue.
Having same problem with concoct.
Updated Metawrap to version 1.0.4-0 and got similar error as above user doing binning. Made fresh install following the provided instructions and still got a problem. scipy was only version 0.17.1 in Metawrap version 1.0.4-0 according to conda list. downgraded to 1.0.3 and everything worked.
Hope this helps, Gisle
Thanks for that Gisle! Looks like accidentally removed libopenblas
as a dependency when I was playing around with the dependency versions... The error should be gone in metawrap=1.0.5
.
Hi I have similar problem. My metaWRAP version is 1.2. metabat2 ran just fine, but concoct and maxbin2 returned error.
Maxbin2 = no glibc_2.14 (although there was no error during installation) concoct = vbgmm module was not found (even though samtools is installed)
did the --update-deps but no luck :(, fresh installation didn't do trick either. Any ideas?
Did you try re-installing libopenblas and gllibc 2.14? For the concoct issue, see https://github.com/BinPro/CONCOCT/issues/137. You can contact the concoct developer directly if it still does not work.
Hi Ursky, I have found out about the glibc issue. it seems that I have to install glibc=2.14 in the metawrap-env, export LD_LIBRARY_PATH (I set the PBS directive to do this in whenever I started the environment), then run the tools.
wget https://ftp.gnu.org/gnu/libc/glibc-2.14.tar.gz
tar zvfg glibc-2.14.tar.gz
mkdir build glibc
cd build
../glibc-2.14/configure --prefix=path/to/anaconda/glibc CC=/path/to/gcc/>4.9 (or specified in INSTALL)
make ; make install
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/glibc/lib
Hi,
I have run the binning module.
MetaBat2 run without problems and output around 200 bins, but both MaxBin2 and Concoct called for errors, see below the error file:
In the log file for MaxBin2, it says:
Both MaxBin2 and concoct abundance files seems to be fine:
Do you know what could it be going on?
Thanks in advance.