search

epi2me-labs / wf-metagenomics

Metagenomic classification of long-read sequencing data
Other
58 stars 23 forks source link

Ran out of Memory Even When Using Memory Mapping #101

Closed Pompidor closed 4 months ago

Pompidor commented 4 months ago

Operating System

Ubuntu 22.04

Other Linux

No response

Workflow Version

v2.9.4

Workflow Execution

EPI2ME Desktop (Local)

Other workflow execution

No response

EPI2ME Version

V5.1.10

CLI command run

No response

Workflow Execution - CLI Execution Profile

None

What happened?

I tried to use the nt Database but it didn't work even with the "Enable memory mapping" option enabled.

Relevant log output

N E X T F L O W  ~  version 23.04.2
Launching `/home/XXXXXX/epi2melabs/workflows/epi2me-labs/wf-metagenomics/main.nf` [XXXXXX_complete_l150_Q15] DSL2 - revision: 72fc1ab296
WARN: NEXTFLOW RECURSION IS A PREVIEW FEATURE - SYNTAX AND FUNCTIONALITY CAN CHANGE IN FUTURE RELEASES
WARN: Found unexpected parameters:
* --client_fields: /home/XXXXXX/epi2melabs/instances/wf-metagenomics_01HZF089X28RZXEEJBVTKZFG0T/client_fields.json
- Ignore this warning: params.schema_ignore_params = "client_fields" 
||||||||||   _____ ____ ___ ____  __  __ _____      _       _
||||||||||  | ____|  _ \_ _|___ \|  \/  | ____|    | | __ _| |__  ___
|||||       |  _| | |_) | |  __) | |\/| |  _| _____| |/ _` | '_ \/ __|
|||||       | |___|  __/| | / __/| |  | | |__|_____| | (_| | |_) \__ \
||||||||||  |_____|_|  |___|_____|_|  |_|_____|    |_|\__,_|_.__/|___/
||||||||||  wf-metagenomics v2.9.4
--------------------------------------------------------------------------------
Core Nextflow options
  runName               : XXXXXX
  containerEngine       : docker
  launchDir             : /home/XXXXXX/epi2melabs/instances/wf-metagenomics_01HZF089X28RZXEEJBVTKZFG0T
  workDir               : /home/XXXXXX/epi2melabs/instances/wf-metagenomics_01HZF089X28RZXEEJBVTKZFG0T/work
  projectDir            : /home/XXXXXX/epi2melabs/workflows/epi2me-labs/wf-metagenomics
  userName              : XXXXXX
  profile               : standard
  configFiles           : /home/XXXXXX/epi2melabs/workflows/epi2me-labs/wf-metagenomics/nextflow.config
Input Options
  fastq                 : /XXXXXXXXXXXX/fastq_pass
Reference Options
  store_dir             : /home/XXXXXX/epi2melabs/data
  database              : /media/XXXXXX/28b63571-a315-41a6-9d28-f18235330e85/k2_nt_20231129-server
  database_sets         : [ncbi_16s_18s:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_16s_18s.fna, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ncbi_targeted_loci_kraken2.tar.gz, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s/ref2taxid.targloci.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip], ncbi_16s_18s_28s_ITS:[reference:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS.fna, database:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ncbi_16s_18s_28s_ITS_kraken2.tar.gz, ref2taxid:https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-metagenomics/ncbi_16s_18s_28s_ITS/ref2taxid.ncbi_16s_18s_28s_ITS.tsv, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/taxdmp_2023-01-01.zip], SILVA_138_1:[database:null], Standard-8:[database:https://genome-idx.s3.amazonaws.com/kraken/k2_standard_08gb_20231009.tar.gz, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/new_taxdump_2023-03-01.zip], PlusPF-8:[database:https://genome-idx.s3.amazonaws.com/kraken/k2_pluspf_08gb_20230314.tar.gz, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/new_taxdump_2023-03-01.zip], PlusPFP-8:[database:https://genome-idx.s3.amazonaws.com/kraken/k2_pluspfp_08gb_20230314.tar.gz, taxonomy:https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump_archive/new_taxdump_2023-03-01.zip]]
Kraken2 Options
  kraken2_memory_mapping: true
Report Options
  n_taxa_barplot        : 20
Output Options
  out_dir               : /home/XXXXXX/epi2melabs/instances/wf-metagenomics_01HZF089X28RZXEEJBVTKZFG0T/output
Advanced Options
  min_len               : 150
  min_read_qual         : 15
!! Only displaying parameters that differ from the pipeline defaults !!
--------------------------------------------------------------------------------
If you use epi2me-labs/wf-metagenomics for your analysis please cite:
* The nf-core framework
  https://doi.org/10.1038/s41587-020-0439-x
--------------------------------------------------------------------------------
This is epi2me-labs/wf-metagenomics v2.9.4.
--------------------------------------------------------------------------------
Checking inputs.
Note: Reference/Database are custom.
Note: Memory available to the workflow must be slightly higher than size of the database custom index.
Note: Or consider to use the --kraken2_memory_mapping.
Note: Memory available to the workflow must be slightly higher than size of the database Standard-8 index (8GB) or consider to use --kraken2_memory_mapping
Searching input for [.fastq, .fastq.gz, .fq, .fq.gz] files.
Note: Empty files or those files whose reads have been discarded after filtering based on read length and/or read quality will not appear in the report and will be excluded from subsequent analysis.
Kraken2 pipeline.
Preparing databases.
Using default taxonomy database.
Checking custom kraken2 database exists
Using the bracken dist file within your custom database directory.
[skipping] Stored process > prepare_databases:download_unpack_taxonomy
[ea/2b8e1b] Submitted process > fastcat (15)
[7e/d811d8] Submitted process > fastcat (6)
[48/636d32] Submitted process > prepare_databases:determine_bracken_length
[17/aed026] Submitted process > fastcat (17)
[04/a2fbef] Submitted process > kraken_pipeline:run_common:getParams
[98/668367] Submitted process > kraken_pipeline:run_common:getVersions
[98/36e1e3] Submitted process > fastcat (10)
[c4/8b61f0] Submitted process > fastcat (2)
[82/fdd734] Submitted process > fastcat (14)
[dd/871aa9] Submitted process > fastcat (5)
[47/18dd21] Submitted process > fastcat (13)
[f9/bd1336] Submitted process > fastcat (18)
[e6/3daabf] Submitted process > fastcat (12)
[6a/eed1ec] Submitted process > fastcat (4)
ERROR ~ Consider to use --kraken2_memory_mapping to reduce the use of RAM memory.
 -- Check '/home/XXXXXX/epi2melabs/instances/wf-metagenomics_01HZF089X28RZXEEJBVTKZFG0T/nextflow.log' file for details
ERROR ~ Error executing process > 'kraken_pipeline:run_kraken2 (barcode12)'
Caused by:
  Process requirement exceeds available memory -- req: 178.1 GB; avail: 62.5 GB
Command executed:
  kraken2 --db k2_nt_20231129-server seqs.fastq.gz         --threads 4         --report "barcode12.kraken2.report.txt"         --confidence 0 --memory-mapping > "barcode12.kraken2.assignments.tsv"
Command exit status:
  -
Command output:
  (empty)
Work dir:
  /home/XXXXXX/epi2melabs/instances/wf-metagenomics_01HZF089X28RZXEEJBVTKZFG0T/work/93/05a4edbab6644375037c8135be00bf
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
 -- Check '/home/XXXXXX/epi2melabs/instances/wf-metagenomics_01HZF089X28RZXEEJBVTKZFG0T/nextflow.log' file for details
WARN: Killing running tasks (12)

Application activity log entry

No response

Were you able to successfully run the latest version of the workflow with the demo data?

yes

Other demo data information

No response

nggvs commented 4 months ago

Hi @Pompidor , Thanks for using the workflow! I have still limited the amount of memory required by the process to 25% of the size of the database, but you can overwrite the memory assigned to the process by using a custom nextflow config. In the app you can do it during setting the workflow: go to Nextflow Configuration and paste this in Configuration:

process {
  withName: run_kraken2 {
    memory = 60.GB 
  }
}

Let me know if that works, could be that take more time to finish.

Pompidor commented 4 months ago

Hi @nggvs, Thank you for the advice, it's now working perfectly! Indeed it takes more time but at least it's working.

nggvs commented 4 months ago

Hi @Pompidor Glad to hear that! Could you close the issue in that case? Thank you very much!