search

nf-core / mag

Assembly and binning of metagenomes
https://nf-co.re/mag
MIT License
213 stars 109 forks source link

Provide example successful output of "nf-core/mag -profile test,singularity" #278

Closed prehensilecode closed 2 years ago

prehensilecode commented 2 years ago

Is your feature request related to a problem? Please describe

Running the self-test for singularity

nextflow run nf-core/mag -profile test,singularity

frequently fails, and at different points.

Describe the solution you'd like

Full output of a completed successful test so that I have something to compare with.

(I am a sysadmin, and do not know anything about the components of this workflow.)

Describe alternatives you've considered

Just blindly keep re-running the test.

Additional context

d4straub commented 2 years ago

Hi! I typically try to assist with problem solving when there are issues with the pipeline. Pipeline output is described in detail in https://nf-co.re/mag/output and example output can be found at https://nf-co.re/mag/results. However, both of those resources do not specify exactly the test run output, so its not what you are asking for, I assume. https://nf-co.re/mag/results displays results of -profile test_full, that accesses data on AWS, typically (but not always) more reliable source than github files. You can find the test specifications in https://github.com/nf-core/mag/blob/master/conf/. Profile test_full has meaningful results and "real" datasets, in contrast to profile test, which has very small, meaningless data that is solely used for simple & fast testing or pipeline development. So, essentially, I would recommend to use -profile test_full for your purpose with https://nf-co.re/mag/results. Does that solve your issue for output examples?

prehensilecode commented 2 years ago

@d4straub Thanks for responding. I have "-profile test_full,singularity" currently running for a few hours. I think possibly part of the reason my prior runs failed was that I was running on a local partition with only about 100 GiB of free space. After moving to a BeeGFS filesystem with 170 TiB available, the run has continued for a few hours. Currently, the work directory is consuming 719 GiB, while the Singularity cache directory takes up 2.4 GiB.

I'll do a comparison of outputs once the test run has completed.

May I suggest adding a note about the requirements (disk space, etc.) needed to run the test suite in the "Quick Start" section of README.md.

d4straub commented 2 years ago

Well, -profile test runs even on my laptop in like 20 min, requires a few GB disk space, no need to over-complicate here. I agree though that it might be helpful to document somewhere requirements for -profile test_full. Is there a specific place you would have found/expected it? Sorry, now I see you think Quick start would be a good place, I'll check requirements (or you could provide them to me, after all you are running it right now) and put it there.

prehensilecode commented 2 years ago

Current -profile test_full,singularity run failed on NFCORE_MAG:MAG:PROKKA (MEGAHIT-CAPES_S7.14). Output is too long to paste: please see the gist.

At the end of the run of test_full, work directory consumes 787 GiB, with 14 tests yet incomplete or failed. Singularity cache takes up 4.1 GiB.

Meanwhile, I'll try another run of -profile test,singularity.

d4straub commented 2 years ago

14 tests? Than your number for disk usage is pretty useless, isn't it? There seems to be a problem that is unrelated to the pipeline code. The pipeline actually progresses quite far, but doesn't finish. Did you ever try the nextflow command -resume? Or in this case append --skip_prokka -resume? Also, the command line log is helpful, but the .nextflow.log file is more valuable (not that I am saying it would be of particular help here).

Running the self-test for singularity nextflow run nf-core/mag -profile test,singularity frequently fails, and at different points.

Also indicates a non-pipeline related problem. test (and related profiles) work when your system works, is compatible with nextflow and uses compatible settings.

prehensilecode commented 2 years ago

OK. I'll check that Nextflow does work properly before trying nf-core/mag again. Thanks for taking time to explain.

Removing the test_full stuff, and running/re-running -profile test,singularity --skip_prokka -resume gave:

executor >  local (37)
[ad/cd71a3] process > NFCORE_MAG:MAG:FASTQC_RAW (test_minigut_sample2)                                                           [100%] 2 of 2, cached: 2 ✔
[c5/26ea49] process > NFCORE_MAG:MAG:FASTP (test_minigut_sample2)                                                                [100%] 2 of 2, cached: 2 ✔
[75/a22545] process > NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_BUILD (GCA_002596845.1_ASM259684v1_genomic.fna.gz)                     [100%] 1 of 1, cached: 1 ✔
[ce/143466] process > NFCORE_MAG:MAG:BOWTIE2_PHIX_REMOVAL_ALIGN (test_minigut_sample2-phix_removed)                              [100%] 2 of 2, cached: 2 ✔
[f3/ec3030] process > NFCORE_MAG:MAG:FASTQC_TRIMMED (test_minigut_sample2)                                                       [100%] 2 of 2, cached: 2 ✔
[-        ] process > NFCORE_MAG:MAG:NANOPLOT_RAW                                                                                -
[-        ] process > NFCORE_MAG:MAG:PORECHOP                                                                                    -
[-        ] process > NFCORE_MAG:MAG:NANOLYSE                                                                                    -
[-        ] process > NFCORE_MAG:MAG:FILTLONG                                                                                    -
[-        ] process > NFCORE_MAG:MAG:NANOPLOT_FILTERED                                                                           -
[d4/2c3e56] process > NFCORE_MAG:MAG:CENTRIFUGE_DB_PREPARATION                                                                   [100%] 1 of 1, cached: 1 ✔
[7b/58e464] process > NFCORE_MAG:MAG:CENTRIFUGE (test_minigut_sample2-minigut_cf)                                                [100%] 2 of 2, cached: 2 ✔
[43/0ce38b] process > NFCORE_MAG:MAG:KRAKEN2_DB_PREPARATION                                                                      [100%] 1 of 1, cached: 1 ✔
[e8/371857] process > NFCORE_MAG:MAG:KRAKEN2 (test_minigut_sample2-minigut_kraken)                                               [100%] 2 of 2, cached: 2 ✔
[97/723149] process > NFCORE_MAG:MAG:MEGAHIT (test_minigut_sample2)                                                              [100%] 2 of 2, cached: 2 ✔
[13/b1dda7] process > NFCORE_MAG:MAG:SPADES (test_minigut_sample2)                                                               [100%] 2 of 2, cached: 2 ✔
[-        ] process > NFCORE_MAG:MAG:SPADESHYBRID                                                                                -
[67/b3d96a] process > NFCORE_MAG:MAG:QUAST (MEGAHIT-test_minigut_sample2)                                                        [100%] 4 of 4, cached: 4 ✔
[0a/34e0b7] process > NFCORE_MAG:MAG:PRODIGAL (test_minigut_sample2)                                                             [100%] 4 of 4, cached: 4 ✔
[b0/e2443e] process > NFCORE_MAG:MAG:METABAT2_BINNING:BOWTIE2_ASSEMBLY_BUILD (SPAdes-test_minigut_sample2)                       [100%] 4 of 4, cached: 4 ✔
[e0/bc403c] process > NFCORE_MAG:MAG:METABAT2_BINNING:BOWTIE2_ASSEMBLY_ALIGN (MEGAHIT-test_minigut_sample2-test_minigut_sample2) [100%] 8 of 8, cached: 8 ✔
[1c/19c810] process > NFCORE_MAG:MAG:METABAT2_BINNING:METABAT2 (MEGAHIT-test_minigut_sample2)                                    [100%] 4 of 4, cached: 1 ✔
[8a/8951b1] process > NFCORE_MAG:MAG:METABAT2_BINNING:MAG_DEPTHS (MEGAHIT-test_minigut_sample2)                                  [100%] 4 of 4 ✔
[14/929cf7] process > NFCORE_MAG:MAG:METABAT2_BINNING:MAG_DEPTHS_PLOT (MEGAHIT-test_minigut_sample2)                             [100%] 4 of 4 ✔
[0d/5df2df] process > NFCORE_MAG:MAG:METABAT2_BINNING:MAG_DEPTHS_SUMMARY                                                         [100%] 1 of 1 ✔
[2e/56598a] process > NFCORE_MAG:MAG:BUSCO_QC:BUSCO_DB_PREPARATION (bacteria_odb10.2020-03-06.tar)                               [100%] 1 of 1, cached: 1 ✔
[e7/e39932] process > NFCORE_MAG:MAG:BUSCO_QC:BUSCO (MEGAHIT-test_minigut.1.fa)                                                  [100%] 12 of 12 ✔
[37/71658d] process > NFCORE_MAG:MAG:BUSCO_QC:BUSCO_PLOT (SPAdes-test_minigut_sample2)                                           [100%] 4 of 4 ✔
[0c/278447] process > NFCORE_MAG:MAG:BUSCO_QC:BUSCO_SUMMARY                                                                      [100%] 1 of 1 ✔
[53/78ef7c] process > NFCORE_MAG:MAG:QUAST_BINS (SPAdes-test_minigut)                                                            [100%] 4 of 4 ✔
[cf/7644bd] process > NFCORE_MAG:MAG:QUAST_BINS_SUMMARY                                                                          [100%] 1 of 1 ✔
[-        ] process > NFCORE_MAG:MAG:CAT                                                                                         -
[7c/a173a4] process > NFCORE_MAG:MAG:BIN_SUMMARY (1)                                                                             [100%] 1 of 1 ✔
[c1/d26d5a] process > NFCORE_MAG:MAG:GET_SOFTWARE_VERSIONS                                                                       [100%] 1 of 1 ✔
[bc/18bb29] process > NFCORE_MAG:MAG:MULTIQC                                                                                     [100%] 1 of 1 ✔
-[nf-core/mag] Pipeline completed successfully-

Completed at: 01-Mar-2022 10:53:10
Duration    : 1m 10s
CPU hours   : 0.3 (50.2% cached)
Succeeded   : 37
Cached      : 41