search

bxlab / metaWRAP

MetaWRAP - a flexible pipeline for genome-resolved metagenomic data analysis
MIT License
381 stars 188 forks source link

Something wrong when using metawrap bin_refinement #343

Open aiguoguo1 opened 3 years ago

aiguoguo1 commented 3 years ago

When I use metawrap bin_refinement, something wrong happened after BIN REFINEMENT and begin checkm, detailed information is below:

  Running CheckM on binsA bins                                         -----
------------------------------------------------------------------------------------------------------------------------

*******************************************************************************
 [CheckM - tree] Placing bins in reference genome tree.
*******************************************************************************

  Identifying marker genes in 35 bins with 8 threads:
Process SyncManager-1:
Traceback (most recent call last):
  File "/home/hanwei/miniconda3/envs/metawrap-env/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/home/hanwei/miniconda3/envs/metawrap-env/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hanwei/miniconda3/envs/metawrap-env/lib/python2.7/multiprocessing/managers.py", line 550, in _run_server
    server = cls._Server(registry, address, authkey, serializer)
  File "/home/hanwei/miniconda3/envs/metawrap-env/lib/python2.7/multiprocessing/managers.py", line 162, in __init__
    self.listener = Listener(address=address, backlog=16)
  File "/home/hanwei/miniconda3/envs/metawrap-env/lib/python2.7/multiprocessing/connection.py", line 132, in __init__
    self._listener = SocketListener(address, family, backlog)
  File "/home/hanwei/miniconda3/envs/metawrap-env/lib/python2.7/multiprocessing/connection.py", line 256, in __init__
    self._socket.bind(address)
  File "/home/hanwei/miniconda3/envs/metawrap-env/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 22] Invalid argument

Unexpected error: <type 'exceptions.EOFError'>
Traceback (most recent call last):
  File "/home/hanwei/miniconda3/envs/metawrap-env/bin/checkm", line 711, in <module>
    checkmParser.parseOptions(args)
  File "/home/hanwei/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/main.py", line 1252, in parseOptions
    self.tree(options)
  File "/home/hanwei/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/main.py", line 134, in tree
    options.bCalledGenes)
  File "/home/hanwei/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/markerGeneFinder.py", line 67, in find
    binIdToModels = mp.Manager().dict()
  File "/home/hanwei/miniconda3/envs/metawrap-env/lib/python2.7/multiprocessing/__init__.py", line 99, in Manager
    m.start()
  File "/home/hanwei/miniconda3/envs/metawrap-env/lib/python2.7/multiprocessing/managers.py", line 528, in start
    self._address = reader.recv()
EOFError

************************************************************************************************************************
*****                             Something went wrong with running CheckM. Exiting...                             *****
************************************************************************************************************************
ursky commented 3 years ago

Im not sure, but it seems to be a CheckM problem. I would consult with their team and open an issue on their github.

EacoChen commented 3 years ago

I also met this problem. And I found what was going wrong. The point is the checkm parameter '--tmpdir' when the path include symbolic link this error can be seen.

I hope this may help.

Caiyulu-818 commented 3 years ago

I also met this problem. And I found what was going wrong. The point is the checkm parameter '--tmpdir' when the path include symbolic link this error can be seen.

I hope this may help.

hi. how to set the parameter '--tmpdir' during bin_refinement?

EacoChen commented 3 years ago

hi. how to set the parameter '--tmpdir' during bin_refinement?

Hi, This is the error from the "checkm"

check the '/bin/metawrap-modules/bin_refinement.sh' line 301 and 303

checkm lineage_wf -x fa $bin_set ${bin_set}.checkm -t $threads --tmpdir ${bin_set}.tmp --pplacer_threads $p_threads checkm lineage_wf -x fa $bin_set ${bin_set}.checkm -t $threads --tmpdir ${bin_set}.tmp --pplacer_threads $p_threads

if you do not set the parameter "--skip-check" you will get this error, just because the ${bin_set} is a symbolic link

I used ln -s to save the usage of the disk, so I got this error.

Caiyulu-818 commented 3 years ago 
font{
    line-height: 1.6;
}
ul,ol{
    padding-left: 20px;
    list-style-position: inside;
}

Thank you for your reply timelyI find the bin_refinement.sh about the —tmpdirI think.i have solved this error YoursCaiyu 

                            lucyncs123

                                ***@***.***

    签名由
    网易邮箱大师
    定制

On 06/17/2021 ***@***.***> wrote：

hi. how to set the parameter '--tmpdir' during bin_refinement?

Hi, This is the error from the "checkm"

check the '/bin/metawrap-modules/bin_refinement.sh' line 301 and 303

checkm lineage_wf -x fa $bin_set ${bin_set}.checkm -t $threads --tmpdir ${bin_set}.tmp --pplacer_threads $p_threads

checkm lineage_wf -x fa $bin_set ${bin_set}.checkm -t $threads --tmpdir ${bin_set}.tmp --pplacer_threads $p_threads

if you do not set the parameter "--skip-check" you will get this error, just because the ${bin_set} is a symbolic link

I used ln -s to save the usage of the disk, so I got this error.

因为你在运行metawrap bin_refinement的时候-A -B -C可能给了一些“软链接”地址，他这里的bin_set变量得是实际路径，不然导致checkm报错

bin_refinement 不能指定--tmpdir，你要不就把你bin_refinement.sh脚本里面这个tmpdir参数删掉，line301,303，402,404应该是这四处如果我没有遗漏的话

Sorry for replied by Chinese.

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe.

Caiyulu-818 commented 2 years ago

hi. how to set the parameter '--tmpdir' during bin_refinement?

Hi, This is the error from the "checkm"

check the '/bin/metawrap-modules/bin_refinement.sh' line 301 and 303

checkm lineage_wf -x fa $bin_set ${bin_set}.checkm -t $threads --tmpdir ${bin_set}.tmp --pplacer_threads $p_threads checkm lineage_wf -x fa $bin_set ${bin_set}.checkm -t $threads --tmpdir ${bin_set}.tmp --pplacer_threads $p_threads

if you do not set the parameter "--skip-check" you will get this error, just because the ${bin_set} is a symbolic link

I used ln -s to save the usage of the disk, so I got this error.

hi i change the parameter of --tmpdir but i miss the error with the message: can't find the file "binA.checkm", how can i solve the problem?

Caiyulu-818 commented 2 years ago

hi i again meet some problem after change the parameter, here is the error message: Aligning marker genes with multiple hits in a single bin: Finished processing 0 of 4463 (0.00%) bins. Unexpected error: <type 'exceptions.AttributeError'> Traceback (most recent call last): File "/data2/lcy/miniconda3/envs/metawrap-env/bin/checkm", line 708, in checkmParser.parseOptions(args) File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/main.py", line 1253, in parseOptions self.analyze(options) File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/main.py", line 331, in analyze os.path.join(options.out_folder, 'storage', 'aai_qa') File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/hmmerAligner.py", line 176, in makeAlignmentsOfMultipleHits p.terminate() File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/multiprocessing/process.py", line 140, in terminate self._popen.terminate() AttributeError: 'NoneType' object has no attribute 'terminate'

EacoChen commented 2 years ago

hi i again meet some problem after change the parameter, here is the error message: Aligning marker genes with multiple hits in a single bin: Finished processing 0 of 4463 (0.00%) bins. Unexpected error: <type 'exceptions.AttributeError'> Traceback (most recent call last): File "/data2/lcy/miniconda3/envs/metawrap-env/bin/checkm", line 708, in checkmParser.parseOptions(args) File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/main.py", line 1253, in parseOptions self.analyze(options) File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/main.py", line 331, in analyze os.path.join(options.out_folder, 'storage', 'aai_qa') File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/hmmerAligner.py", line 176, in makeAlignmentsOfMultipleHits p.terminate() File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/multiprocessing/process.py", line 140, in terminate self._popen.terminate() AttributeError: 'NoneType' object has no attribute 'terminate'

Hi, what script are you running? can you offer the command line?

hi i change the parameter of --tmpdir but i miss the error with the message: can't find the file "binA.checkm", how can i solve the problem?

and how you change the parameter?

Caiyulu-818 commented 2 years ago 
font{
    line-height: 1.6;
}
ul,ol{
    padding-left: 20px;
    list-style-position: inside;
}

the command I use：nohup metawrap bin_refinement -o /data1/lucy/srabin_refine2q_2 -t 10 -A /public/home/bma/lcy/SRA9_199_1 -B /data2/bma/lcy/sramaxbin2_1  -C /data1/lucy/lcybin/sra9_199_1 -c 50 -x 10 —quick & And I delete the parameter —tempdir (four places) in the  bin_refinement.sh in the metawrap_modules  !Best Lucy

                            lucyncs123

                                ***@***.***

    签名由
    网易邮箱大师
    定制

On 07/12/2021 ***@***.***> wrote：

hi i again meet some problem after change the parameter, here is the error message: Aligning marker genes with multiple hits in a single bin: Finished processing 0 of 4463 (0.00%) bins. Unexpected error: <type 'exceptions.AttributeError'> Traceback (most recent call last): File "/data2/lcy/miniconda3/envs/metawrap-env/bin/checkm", line 708, in checkmParser.parseOptions(args) File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/main.py", line 1253, in parseOptions self.analyze(options) File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/main.py", line 331, in analyze os.path.join(options.out_folder, 'storage', 'aai_qa') File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/hmmerAligner.py", line 176, in makeAlignmentsOfMultipleHits p.terminate() File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/multiprocessing/process.py", line 140, in terminate self._popen.terminate() AttributeError: 'NoneType' object has no attribute 'terminate'

Hi, what script are you running? can you offer the command line?

hi i change the parameter of --tmpdir but i miss the error with the message: can't find the file "binA.checkm", how can i solve the problem?

and how you change the parameter?

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe.

EacoChen commented 2 years ago

I mostly know the error, but I am not so sure. Can you also offer your nohop.out?

hi i again meet some problem after change the parameter, here is the error message: Aligning marker genes with multiple hits in a single bin: Finished processing 0 of 4463 (0.00%) bins. Unexpected error: <type 'exceptions.AttributeError'> Traceback (most recent call last): File "/data2/lcy/miniconda3/envs/metawrap-env/bin/checkm", line 708, in checkmParser.parseOptions(args) File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/main.py", line 1253, in parseOptions self.analyze(options) File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/main.py", line 331, in analyze os.path.join(options.out_folder, 'storage', 'aai_qa') File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/hmmerAligner.py", line 176, in makeAlignmentsOfMultipleHits p.terminate() File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/multiprocessing/process.py", line 140, in terminate self._popen.terminate() AttributeError: 'NoneType' object has no attribute 'terminate'

Caiyulu-818 commented 2 years ago

I mostly know the error, but I am not so sure. Can you also offer your nohop.out?

hi i again meet some problem after change the parameter, here is the error message: Aligning marker genes with multiple hits in a single bin: Finished processing 0 of 4463 (0.00%) bins. Unexpected error: <type 'exceptions.AttributeError'> Traceback (most recent call last): File "/data2/lcy/miniconda3/envs/metawrap-env/bin/checkm", line 708, in checkmParser.parseOptions(args) File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/main.py", line 1253, in parseOptions self.analyze(options) File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/main.py", line 331, in analyze os.path.join(options.out_folder, 'storage', 'aai_qa') File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/hmmerAligner.py", line 176, in makeAlignmentsOfMultipleHits p.terminate() File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/multiprocessing/process.py", line 140, in terminate self._popen.terminate() AttributeError: 'NoneType' object has no attribute 'terminate'

here is the nohup.out nohup.out.zip

Caiyulu-818 commented 2 years ago 
font{
    line-height: 1.6;
}
ul,ol{
    padding-left: 20px;
    list-style-position: inside;
}

I load it on the github(https://github.com/bxlab/metaWRAP/issues/343)

                            lucyncs123

                                ***@***.***

    签名由
    网易邮箱大师
    定制

On 07/12/2021 ***@***.***> wrote：

I mostly know the error, but I am not so sure. Can you also offer your nohop.out?

hi i again meet some problem after change the parameter, here is the error message: Aligning marker genes with multiple hits in a single bin: Finished processing 0 of 4463 (0.00%) bins. Unexpected error: <type 'exceptions.AttributeError'> Traceback (most recent call last): File "/data2/lcy/miniconda3/envs/metawrap-env/bin/checkm", line 708, in checkmParser.parseOptions(args) File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/main.py", line 1253, in parseOptions self.analyze(options) File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/main.py", line 331, in analyze os.path.join(options.out_folder, 'storage', 'aai_qa') File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/site-packages/checkm/hmmerAligner.py", line 176, in makeAlignmentsOfMultipleHits p.terminate() File "/data2/lcy/miniconda3/envs/metawrap-env/lib/python2.7/multiprocessing/process.py", line 140, in terminate self._popen.terminate() AttributeError: 'NoneType' object has no attribute 'terminate'

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe.

EacoChen commented 2 years ago

what happen if you do not delete --tmpdir and try to use the absolute path to the file. Because it used in the following script,

Jamesped commented 3 days ago

deletion of two '--tmpdir binsO.tmp' and two '--tmpdir binsO.tmp' in bin_refinement.sh solved my problem. Thanks.