Fastqc error

[francesco-ricci]

Hi Vini,

any clue what's this error?

This is the log file:

Memory value 8 MB was outside the allowed range (100 - 10000) at /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/$

This is the error file:

subprocess.CalledProcessError: Command 'set -euo pipefail; fastqc --threads 24 --memory 8 --outdir /tmp/tmpn8b0y_by /fs04/rp24/fra/analyses/deep_sea_corals/raw_read$ [Thu Dec 14 13:25:55 2023] [Thu Dec 14 13:25:55 2023] Error in rule fastqc_raw: jobid: 7 input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6229_S32_R2_001.fastq.gz output: output/qc/fastqc/SF6229_S321-single_unit-R2-raw_fastqc.zip, output/qc/fastqc/SF6229_S321-single_unit-R2-raw.html log: output/logs/qc/fastqc_raw/SF6229_S321-singleunit-R2.log (check log file(s) for error details) conda-env: /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/b3bbc0ad9f1d0f4f0c13735583f21890

Error in rule fastqc_raw: jobid: 2 input: /fs04/rp24/fra/analyses/deep_sea_corals/raw_reads/SF6227_S30_R1_001.fastq.gz output: output/qc/fastqc/SF6227_S301-single_unit-R1-raw_fastqc.zip, output/qc/fastqc/SF6227_S301-single_unit-R1-raw.html log: output/logs/qc/fastqc_raw/SF6227_S301-singleunit-R1.log (check log file(s) for error details) conda-env: /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/config/conda/b3bbc0ad9f1d0f4f0c13735583f21890

RuleException: CalledProcessError in file /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/qc.smk, line 121: Command 'source /home/fricci/rp24/fra/software/miniconda3/bin/activate '/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/$ File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/qc.smk", line 121, in rule_fastqc_raw File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/concurrent/futures/thread.py", line 58, in run RuleException: CalledProcessError in file /home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/qc.smk, line 121: Command 'source /home/fricci/rp24/fra/software/miniconda3/bin/activate '/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/$ File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/workflow/rules/qc.smk", line 121, in rule_fastqc_raw File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/concurrent/futures/thread.py", line 58, in run Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2023-12-14T131254.167617.snakemake.log Traceback (most recent call last): File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/bin/metaphor", line 10, in sys.exit(main()) ^^^^^^ File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/cli.py", line 278, in main args.func(args) File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/cli/execute.py", line 109, in main retcode = run_cmd(cmd) ^^^^^^^^^^^^ File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/utils.py", line 92, in run_cmd retcode = check_call(cmd.split()) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['snakemake', '--snakefile', '/home/fricci/rp24/fra/software/miniconda3/envs/metaphor/lib/python3.11/site-packages/metaphor/wo$

[vinisalazar]

@francesco-ricci the first few lines of your message indicate a memory error with FastQC. Not sure why your fastqc_raw rule would have the mem_mb set to 8 but that value should be much higher. You can manually edit the fastqc_raw rule to have a value in the range shown in the error (100-100000). To find the correct file, run metaphor config show --snakefile, and on the parent directory of that file, you can go to rules/qc.smk and edit the fastqc_raw rule.

The FastQC that Metaphor uses is based on a wrapper from the snakemake-wrappers repository. Honestly it's been giving me trouble ever since I started developing Metaphor so I might get rid of it for good. For now, you can either turn FastQC off through your settings YAML file, or manually edit the memory value like I mentioned above.

Thanks, Vini

[francesco-ricci]

Thanks for your help Vini, but I am not sure where to edit these parameters. When I open rules/qc.smk this is what I can find:

rule fastqc_raw: # qc on raw, unmerged reads input: get_fastqc_input_raw, output: zip="output/qc/fastqc/{sample}-{unit}-{read}-raw_fastqc.zip", html="output/qc/fastqc/{sample}-{unit}-{read}-raw.html", params: "--quiet", wildcard_constraints: sample="|".join(sample_IDs), unit="|".join(unit_names), read="single|R1|R2", log: "output/logs/qc/fastqc_raw/{sample}-{unit}-{read}.log", benchmark: "output/benchmarks/qc/fastqc_raw/{sample}-{unit}-{read}.txt" threads: get_threads_per_task_size("medium") wrapper: get_wrapper("fastqc")

What should I modify?

Thanks Francesco

[vinisalazar]

Hi @francesco-ricci,

you can add a resources: mem_mb directive, as shown here:

https://snakemake-wrappers.readthedocs.io/en/stable/wrappers/fastqc.html

You can also modify the command-line to add a specific resource like this:

metaphor execute -c 24 -y -e "--set-resources fastqc_raw:mem_mb=32000"

Having this said, could you please double-check the max_mb value on your metaphor_settings.yaml file?

Best, Vini

[francesco-ricci]

Thanks Vini, max_mb is set to 650000.

[vinisalazar]

Sounds good, please let me know if the problem persists.

[vinisalazar]

Closing this issue for now but don't hesitate to reopen it if needed.