search

jtamames / SqueezeMeta

A complete pipeline for metagenomic analysis
GNU General Public License v3.0
348 stars 81 forks source link

unable to restart SqueezeMeta #679

Closed koutnymarek closed 1 year ago

koutnymarek commented 1 year ago

Hi I somehow make a mistake when trying to restart the process:

/home/jovyan/my-conda-envs/SqueezeMeta) rstudio@jupyter-marekkoutny--mkoutnysqueezemeta:~$ SqueezeMeta.pl -p test31 --restart -step 7 Unknown option: restart Unknown option: step Usage: SqueezeMeta.pl -m -p -s -f [options]

MISSING ARGUMENT: -f|-seq: Fastq read files' directory MISSING ARGUMENT: -s|-samples: Samples file MISSING ARGUMENT: -m: Run mode (sequential, coassembly, merged) UNRECOGNIZED mode (valid ones are sequential, coassembly, merged or seqmerge

Please, do you have an idea where is my problem?

MK

jtamames commented 1 year ago

Hello Don't put the "-p" to specify the project name. Best, J

fpusan commented 1 year ago

Note that in the current version you would need to use -p as you did, but you seem to be using an old one so as Javier says the syntax is different

koutnymarek commented 1 year ago

I have tried to install version 1.6 but it did not work:

conda-forge/linux-64 Using cache conda-forge/noarch Using cache bioconda/linux-64 Using cache bioconda/noarch Using cache fpusan/linux-64 No change fpusan/noarch No change Encountered problems while solving:

Then without specifying the version I was successful and thought that I was installing the latest version.

MK

Marek Koutný Professor, department director Department of Environmental Protection Engineering Faculty of Technology Tomas Bata University in Zlín Nad Ovčírnou 3685, Zlín 76001, Czech Republic

Email: @.**@.> Phone: +420 576 031 208<tel:576%20031%20208> Mobile: +420 739 003 706<tel:739%20003%20706>

From: Fernando Puente-Sánchez @.> Sent: Saturday, May 6, 2023 9:38 To: jtamames/SqueezeMeta @.> Cc: Marek Koutný @.>; Author @.> Subject: Re: [jtamames/SqueezeMeta] unable to restart SqueezeMeta (Issue #679)

Note that in the current version you would need to use -p as you did, but you seem to be using an old one so as Javier says the syntax is different

— Reply to this email directly, view it on GitHubhttps://github.com/jtamames/SqueezeMeta/issues/679#issuecomment-1537079663, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AS5XH3KAPPC4Q6OFGLT4CXTXEX5XTANCNFSM6AAAAAAXX46KCM. You are receiving this because you authored the thread.Message ID: @.***>

fpusan commented 1 year ago

What command did you use exactly to try and install it?

koutnymarek commented 1 year ago

This:

(base) @.***:~$ mamba create -n SqueezeMeta -c conda-forge -c bioconda -c fpusan squeezemeta=1.6 --no-channel-priority

MK

Marek Koutný Professor, department director Department of Environmental Protection Engineering Faculty of Technology Tomas Bata University in Zlín Nad Ovčírnou 3685, Zlín 76001, Czech Republic

Email: @.**@.> Phone: +420 576 031 208<tel:576%20031%20208> Mobile: +420 739 003 706<tel:739%20003%20706>

From: Fernando Puente-Sánchez @.> Sent: Saturday, May 6, 2023 10:08 To: jtamames/SqueezeMeta @.> Cc: Marek Koutný @.>; Author @.> Subject: Re: [jtamames/SqueezeMeta] unable to restart SqueezeMeta (Issue #679)

What command did you use exactly to try and install it?

— Reply to this email directly, view it on GitHubhttps://github.com/jtamames/SqueezeMeta/issues/679#issuecomment-1537085424, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AS5XH3OKM7VQGNVLCLS5FVDXEYBIXANCNFSM6AAAAAAXX46KCM. You are receiving this because you authored the thread.Message ID: @.***>

fpusan commented 1 year ago

That should have worked. Does the problem keep happening in a fresh miniconda3 install?

koutnymarek commented 1 year ago

Sorry, I do not know. I use jupyterhub servers on a computer cluster.

MK

Marek Koutný Professor, department director Department of Environmental Protection Engineering Faculty of Technology Tomas Bata University in Zlín Nad Ovčírnou 3685, Zlín 76001, Czech Republic

Email: @.**@.> Phone: +420 576 031 208<tel:576%20031%20208> Mobile: +420 739 003 706<tel:739%20003%20706>

From: Fernando Puente-Sánchez @.> Sent: Saturday, May 6, 2023 10:22 To: jtamames/SqueezeMeta @.> Cc: Marek Koutný @.>; Author @.> Subject: Re: [jtamames/SqueezeMeta] unable to restart SqueezeMeta (Issue #679)

That should have worked. Does the problem keep happening in a fresh miniconda3 install?

— Reply to this email directly, view it on GitHubhttps://github.com/jtamames/SqueezeMeta/issues/679#issuecomment-1537088163, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AS5XH3P23G6W5NWXHGOQD4LXEYC6DANCNFSM6AAAAAAXX46KCM. You are receiving this because you authored the thread.Message ID: @.***>

fpusan commented 1 year ago

Then maybe you should discuss this with the person managing the servers.

koutnymarek commented 1 year ago

Hello Fernando,

After an exchange with the administrators, I was advised to add "-c anaconda" to my mamba create command and it worked.

Now everything looks fine.

Thanks and all the best,

MK

Marek Koutný Professor, department director Department of Environmental Protection Engineering Faculty of Technology Tomas Bata University in Zlín Nad Ovčírnou 3685, Zlín 76001, Czech Republic

Email: @.**@.> Phone: +420 576 031 208<tel:576%20031%20208> Mobile: +420 739 003 706<tel:739%20003%20706>

From: Marek Koutný @.> Sent: Saturday, May 6, 2023 10:34 To: jtamames/SqueezeMeta @.> Subject: Re: [jtamames/SqueezeMeta] unable to restart SqueezeMeta (Issue #679)

Sorry, I do not know. I use jupyterhub servers on a computer cluster.

MK

Marek Koutný Professor, department director Department of Environmental Protection Engineering Faculty of Technology Tomas Bata University in Zlín Nad Ovčírnou 3685, Zlín 76001, Czech Republic

Email: @.**@.> Phone: +420 576 031 208<tel:576%20031%20208> Mobile: +420 739 003 706<tel:739%20003%20706>

From: Fernando Puente-Sánchez @.> Sent: Saturday, May 6, 2023 10:22 To: jtamames/SqueezeMeta @.> Cc: Marek Koutný @.>; Author @.> Subject: Re: [jtamames/SqueezeMeta] unable to restart SqueezeMeta (Issue #679)

That should have worked. Does the problem keep happening in a fresh miniconda3 install?

— Reply to this email directly, view it on GitHubhttps://github.com/jtamames/SqueezeMeta/issues/679#issuecomment-1537088163, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AS5XH3P23G6W5NWXHGOQD4LXEYC6DANCNFSM6AAAAAAXX46KCM. You are receiving this because you authored the thread.Message ID: @.***>

koutnymarek commented 1 year ago

Hello Fernando,

Just to give you some feedback.

After successful installation, I run Squeezemeta first with the test data then with another small dataset. Everything was OK.

Then I tried with a bigger dataset (1 sample pair 18 GB each file) and got an error message:

[5 hours, 42 minutes, 10 seconds]: STEP4 -> HOMOLOGY SEARCHES: 04.rundiamond.pl

Setting block size for Diamond

AVAILABLE (free) RAM memory: 481.95 Gb

We will set Diamond block size to 16 (Gb RAM/8, Max 16).

You can override this setting using the -b option when starting the project, or changing

the $blocksize variable in SqueezeMeta_conf.pl

Working with taxonomy database in /home/rstudio/sqdb/db/nr.dmnd

taxash: line 1: 3780 Killed /home/rstudio/my-conda-envs/SqueezeMeta/SqueezeMeta/bin/diamond blastp -q /home/rstudio/Data/test2/results/03.test2.faa -p 12 -d /home/rstudio/sqdb/db/nr.dmnd -e 0.001 --id 50 -f tab -b 16 -o /home/rstudio/Data/test2/intermediate/04.test2.nr.diamond > /home/rstudio/Data/test2/temp/diamond.nr.log 2>&1

Error running command: /home/rstudio/my-conda-envs/SqueezeMeta/SqueezeMeta/bin/diamond blastp -q /home/rstudio/Data/test2/results/03.test2.faa -p 12 -d /home/rstudio/sqdb/db/nr.dmnd -e 0.001 --id 50 -f tab -b 16 -o /home/rstudio/Data/test2/intermediate/04.test2.nr.diamond > /home/rstudio/Data/test2/temp/diamond.nr.log 2>&1 at /home/rstudio/my-conda-envs/SqueezeMeta/SqueezeMeta/scripts/04.rundiamond.pl line 77.

Stopping in STEP4 -> 04.rundiamond.pl. Program finished abnormally

If you don't know what went wrong or want further advice, please look for similar issues in https://github.com/jtamames/SqueezeMeta/issues

Feel free to open a new issue if you don't find the answer there. Please add a brief description of the problem and upload the /home/rstudio/Data/test2/syslog file (zip it first)

Died at /home/jovyan/my-conda-envs/SqueezeMeta/bin/SqueezeMeta.pl line 941.

I used 16 CPU and 64 GB RAM.

Then I tried again with 32 CPU 256 GB RAM and everything was OK.

Please, do you see some logic in the happenings above?

Is there some limit to the data volume that can be processed? Some datasets could be quite big.

Marek

Marek Koutný Professor, department director Department of Environmental Protection Engineering Faculty of Technology Tomas Bata University in Zlín Nad Ovčírnou 3685, Zlín 76001, Czech Republic

Email: @.**@.> Phone: +420 576 031 208<tel:576%20031%20208> Mobile: +420 739 003 706<tel:739%20003%20706>

From: Marek Koutný @.> Sent: Saturday, May 13, 2023 12:41 To: jtamames/SqueezeMeta @.> Subject: Re: [jtamames/SqueezeMeta] unable to restart SqueezeMeta (Issue #679)

Hello Fernando,

After an exchange with the administrators, I was advised to add "-c anaconda" to my mamba create command and it worked.

Now everything looks fine.

Thanks and all the best,

MK

Marek Koutný Professor, department director Department of Environmental Protection Engineering Faculty of Technology Tomas Bata University in Zlín Nad Ovčírnou 3685, Zlín 76001, Czech Republic

Email: @.**@.> Phone: +420 576 031 208<tel:576%20031%20208> Mobile: +420 739 003 706<tel:739%20003%20706>

From: Marek Koutný @.> Sent: Saturday, May 6, 2023 10:34 To: jtamames/SqueezeMeta @.> Subject: Re: [jtamames/SqueezeMeta] unable to restart SqueezeMeta (Issue #679)

Sorry, I do not know. I use jupyterhub servers on a computer cluster.

MK

Marek Koutný Professor, department director Department of Environmental Protection Engineering Faculty of Technology Tomas Bata University in Zlín Nad Ovčírnou 3685, Zlín 76001, Czech Republic

Email: @.**@.> Phone: +420 576 031 208<tel:576%20031%20208> Mobile: +420 739 003 706<tel:739%20003%20706>

From: Fernando Puente-Sánchez @.> Sent: Saturday, May 6, 2023 10:22 To: jtamames/SqueezeMeta @.> Cc: Marek Koutný @.>; Author @.> Subject: Re: [jtamames/SqueezeMeta] unable to restart SqueezeMeta (Issue #679)

That should have worked. Does the problem keep happening in a fresh miniconda3 install?

— Reply to this email directly, view it on GitHubhttps://github.com/jtamames/SqueezeMeta/issues/679#issuecomment-1537088163, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AS5XH3P23G6W5NWXHGOQD4LXEYC6DANCNFSM6AAAAAAXX46KCM. You are receiving this because you authored the thread.Message ID: @.***>

fpusan commented 1 year ago

Yes, I can explain this. See this excerpt from the ReadMe

  • Make sure to request enough memory. See the previous section for a rough guide on what is "enough". If you get a crash during the assembly or during the annotation step, it will be likely because you ran out of memory.
  • Make sure to manually set the -b parameter so that it matches the amount of memory that you requested divided by 8. Otherwise, SqueezeMeta will assume that it can use all the free memory in the node in which it is running. This is fine if you got a full node for yourself, but will lead to crashes otherwise.

In general, try asking for full nodes when running SqueezeMeta in a cluster, particularly if your datasets are large. The more RAM you have, the larger datasets you can process, but it is not a linear relationship and also depends on how complex are the samples.