search

MattiaPandolfoVR / MetaPhage

GNU General Public License v3.0
36 stars 9 forks source link

An error at the graphanalyzer step #72

Closed ilnamkang closed 9 months ago

ilnamkang commented 10 months ago

Hi,

I encountered an error likely at the graphanalyzer step. The excerpt of the log file is as below.

How can I avoid this error?

According to the error message, I may be able to try setting "low_memory=False". But, I don't know how to do it.

Thanks.

Dec-08 02:18:15.086 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 682; name: vcontact2 (megahit); status: COMPLETED; exit: 0; error: -; workDir: /home/johy/Virome/MetaPhage/work/e8/5b76587e3dad3cfc3f66e2241cf432] Dec-08 02:18:15.113 [Task submitter] DEBUG nextflow.executor.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run Dec-08 02:18:15.114 [Task submitter] INFO nextflow.Session - [4a/c0e4fb] Submitted process > graphanalyzer (megahit) Dec-08 02:22:13.345 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 02:27:13.400 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 02:32:13.458 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 02:37:13.512 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 02:42:13.575 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 02:47:13.639 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 02:52:13.695 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 02:57:13.751 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 03:02:13.803 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 03:07:13.855 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 03:12:13.906 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 03:17:13.962 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 03:22:13.993 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 03:27:14.045 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 03:32:14.099 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 03:37:14.154 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 03:42:14.210 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 03:47:14.272 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 03:52:14.336 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 03:57:14.385 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 04:02:14.439 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 04:07:14.480 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 04:12:14.525 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 04:18:33.782 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 04:25:33.114 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 04:30:42.370 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 04:35:59.280 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 686; name: graphanalyzer (megahit); status: RUNNING; exit: -; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 04:36:13.864 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 686; name: graphanalyzer (megahit); status: COMPLETED; exit: 1; error: -; workDir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4] Dec-08 04:36:14.458 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'graphanalyzer (megahit)'

Caused by: Process graphanalyzer (megahit) terminated with an error exit status (1)

Command executed:

python /home/johy/Virome/MetaPhage/bin/python/graphanalyzer.py --threads 60 --graph c1.ntw --csv genome_by_genome_overview.csv --metas /home/johy/Virome/MetaPhage/db/inphared/*data_excluding_refseq.tsv --prefix vOTU --output ./ --suffix megahit cp custom_taxonomy_table_megahit_mqc.txt custom_taxonomy_table_mqc.txt python /home/johy/Virome/MetaPhage/bin/python/taxonomy_table_namer.py cp custom_taxonomy_table_mqc.txt custom_taxonomy_table_megahit_mqc.csv sed -i 1,5d custom_taxonomy_table_megahit_mqc.csv mv custom_taxonomy_table_megahit_mqc.csv taxonomy_table.csv rm custom_taxonomy_table_megahit_mqc.txt rm custom_taxonomy_table_mqc.txt

Command exit status: 1

Command output: Completed subgraph 1981/21876 (vOTU_12161)
Completed subgraph 1982/21876 (vOTU_12409)
Completed subgraph 1983/21876 (vOTU_12285)
Completed subgraph 1984/21876 (vOTU_12435)
Completed subgraph 1985/21876 (vOTU_12437)
Completed subgraph 1986/21876 (vOTU_12440)
Completed subgraph 1987/21876 (vOTU_12414)
Completed subgraph 1988/21876 (vOTU_12442)
Completed subgraph 1989/21876 (vOTU_1238)
Completed subgraph 1990/21876 (vOTU_12353)
Completed subgraph 1991/21876 (vOTU_12443)
Completed subgraph 1992/21876 (vOTU_12322)
Completed subgraph 1993/21876 (vOTU_12428)
Completed subgraph 1994/21876 (vOTU_12448)
Completed subgraph 1995/21876 (vOTU_12431)
Completed subgraph 1996/21876 (vOTU_12438)
Completed subgraph 1997/21876 (vOTU_12445)
Completed subgraph 1998/21876 (vOTU_12454)
Completed subgraph 1999/21876 (vOTU_12452)
Completed subgraph 2000/21876 (vOTU_12447)
Completed subgraph 2001/21876 (vOTU_12458)
Completed subgraph 2002/21876 (vOTU_12397)
Completed subgraph 2003/21876 (vOTU_1240)
Completed subgraph 2004/21876 (vOTU_12123)
Completed subgraph 2005/21876 (vOTU_12398)
Completed subgraph 2006/21876 (vOTU_12467)
Completed subgraph 2007/21876 (vOTU_1187)
Completed subgraph 2008/21876 (vOTU_12401)
Completed subgraph 2009/21876 (vOTU_12371)
Completed subgraph 2010/21876 (vOTU_12457)
Completed subgraph 2011/21876 (vOTU_12474)
Completed subgraph 2012/21876 (vOTU_12477)
Completed subgraph 2013/21876 (vOTU_12380)
Completed subgraph 2014/21876 (vOTU_12462)
Completed subgraph 2015/21876 (vOTU_12479)
Completed subgraph 2016/21876 (vOTU_12471)
Completed subgraph 2017/21876 (vOTU_12455)
Completed subgraph 2018/21876 (vOTU_1200)
Completed subgraph 2019/21876 (vOTU_12483)
Completed subgraph 2020/21876 (vOTU_12468)
Completed subgraph 2021/21876 (vOTU_12488)
Completed subgraph 2022/21876 (vOTU_12490)
Completed subgraph 2023/21876 (vOTU_12449)
Completed subgraph 2024/21876 (vOTU_12494)
Completed subgraph 2025/21876 (vOTU_12433)
Completed subgraph 2026/21876 (vOTU_12485)
Completed subgraph 2027/21876 (vOTU_12481)
Completed subgraph 2028/21876 (vOTU_1241)
Completed subgraph 2029/21876 (vOTU_12497)
Completed subgraph 2030/21876 (vOTU_12432)

Command error: sys:1: DtypeWarning: Columns (5,8) have mixed types. Specify dtype option on import or set low_memory=False. Traceback (most recent call last): File "/home/johy/Virome/MetaPhage/bin/python/graphanalyzer.py", line 1105, in subgraphCreator(graph, csv_edit, df_results, parameters.output, parameters.suffix, max_weight, parameters.prefix, parameters.threads) File "/home/johy/Virome/MetaPhage/bin/python/graphanalyzer.py", line 986, in subgraphCreator scaffold = future.result() # blocking call: wait for this task to completed File "/home/johy/miniconda3/envs/metaphage2/lib/python3.7/concurrent/futures/_base.py", line 428, in result return self.get_result() File "/home/johy/miniconda3/envs/metaphage2/lib/python3.7/concurrent/futures/_base.py", line 384, in get_result raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Work dir: /home/johy/Virome/MetaPhage/work/4a/c0e4fbbd73533920c5fc5905cc5be4

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh Dec-08 04:36:14.498 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process graphanalyzer (megahit) terminated with an error exit status (1) Dec-08 04:36:14.527 [Actor Thread 344] WARN nextflow.processor.TaskProcessor - Input tuple does not match input set cardinality declared by process phylo_obj -- offending value: [] Dec-08 04:36:14.604 [Task monitor] DEBUG nextflow.Session - The following nodes are still active:

ilnamkang commented 9 months ago

This error might have been due to a bad edit of the graphanalyzer part of the original "nextflow.config" by me.

I've resumed the run with an updated config file.

If the error is resolved, then I'll update this issue with more detailed information.

ilnamkang commented 9 months ago

Unfortunately, the error has not been resolved.

At first, I edited the graphanalyzer part of the original "nextflow.config" file as below to assign more cores and memory.

  withLabel: graphanalyzer {
        cpus = 60
        memory = 384.GB
        time = 2400.h

But, the run stopped with likely out of memory error. (I was monitoring memory usage of graphanalyzer.py.)

So, I resumed the run with the original configuration of cpus and memory.

  withLabel: graphanalyzer {
        cpus = 2
        memory = 48.GB
        time = 2400.h

But, the resumed run also stopped with out of memory error.

Because I ran Metaphage on an Ubuntu server with 512 GB of memory and there were no other processes that used >50 GB of memory, I think that the setting of the config file had not been applied to the graphanalyzer. Also, at the last time I monitored the memory usage, each of the graphanalyzer processes occupied more than >150 GB of memory.

Are there any ways to avoid this error?

Thanks.

MattiaPandolfoVR commented 9 months ago

Hi there, this is a known issue bound to an old version of graphanalyzer. I'm now testing MP with an updated version of the tool (1.5.1) which should fix the issue. If everything runs smooth, i'll update the dev branch and let you know here :) If you don't wanna wait, feel free to download the graphanalyzer.py script of the 1.5.1 version from here and replace it in your metaphage bin/python/graphanalyzer.py ;)

ilnamkang commented 9 months ago

Hi,

The graphanalyzer.py script of the 1.5.1 version solved this error.

Thank you for your help.