search

vanheeringen-lab / seq2science

Automated and customizable preprocessing of Next-Generation Sequencing data, including full (sc)ATAC-seq, ChIP-seq, and (sc)RNA-seq workflows. Works equally easy with public as local data.
https://vanheeringen-lab.github.io/seq2science
MIT License
156 stars 27 forks source link

Issue with Chip-seq pipeline: jobid: 27: one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode! #939

Closed Jerolen closed 1 year ago

Jerolen commented 1 year ago

Question Hi There, I am trying to test out the chip-set pipeline using a single sample of paired end data but the workflow keeps failing at a specific point with the following error:

Activating conda environment: ../opt/miniconda3/envs/seq2science/lib/python3.8/site-packages/seq2science/.snakemake/6824310427e127c1e8fdddc2f1df8a12 /bin/bash: line 1: shuf: command not found [Wed Dec 7 15:20:57 2022] Error in rule random_subset_peaks: jobid: 27 output: /Users/jerolen/my_project2/results/qc/computeMatrix_peak/GRCh38-macs2_N20000.bed conda-env: /Users/jerolen/opt/miniconda3/envs/seq2science/lib/python3.8/site-packages/seq2science/.snakemake/6824310427e127c1e8fdddc2f1df8a12 shell:

    shuf -n 20000 /Users/jerolen/my_project2/results/macs2/GRCh38_combinedsummits.bed | bedtools slop -i stdin -g /Users/jerolen/my_project2/genomes/GRCh38/GRCh38.fa.sizes -b 1000 > /Users/jerolen/my_project2/results/qc/computeMatrix_peak/GRCh38-macs2_N20000.bed

    (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

What have I tried I have tried changing the alignment tool from bwa to hisat2 with no luck. Any advice would be greatly appreciated as I am completely out of depth on the coding side of things. config.yaml, samples.tsv and export of terminal attached. Thank you.

config-yaml.txt samples.txt Terminal Saved Output-Jerolen-Seq2SCience-chipseq.txt

Maarten-vd-Sande commented 1 year ago

Thanks for the issue! I think there's actually a problem on "our" side here. Seems like we haven't defined the environment properly and we are making use of the shuf command which isn't by default on all systems (e.g. on a mac)...

Let me take a look!

For now you can turn off the deeptools figures for the QC report as this is purely used for visualization only. In the config.yaml add:

deeptools_qc: false

Maarten-vd-Sande commented 1 year ago

I don't have a mac, so you will have to check for me if this solution fixes it :smile:

This file should already exists:

/opt/miniconda3/envs/seq2science/lib/python3.8/site-packages/seq2science/envs/bedtools.yaml

Can you change it so that it has coreutils added:

name: bedtools
channels:
  - conda-forge
  - bioconda
  - defaults
dependencies:
  - bioconda::bedtools=2.29.2
  - conda-forge::coreutils=9.1
  - conda-forge::conda-ecosystem-user-package-isolation=1.0

And then try seq2science again? Seq2science should automatically detect the environment definition has been changed and will re-install the environment.

Jerolen commented 1 year ago

Thanks Marteen

Really appreciate the feedback, will try this now and get back to you :) Found the file and edited as instructed. Just started the run again.

Have a wonderful evening.

Kind Regards

Jerolen

Jerolen Naidoo PhD Senior Researcher Bioengineering & Integrated Genomics Group (BIGG) CSIR NextGen Health Building 20 Meiring Naude road Brummeria Pretoria 0002

From: Maarten-vd-Sande @.> Sent: Wednesday, 7 December 2022, 16:39 To: vanheeringen-lab/seq2science @.> Cc: Jerolen Naidoo @.>; Author @.> Subject: Re: [vanheeringen-lab/seq2science] Issue with Chip-seq pipeline: jobid: 27: one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode! (Issue #939)

You don't often get email from @.*** Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

I don't have a mac, so you will have to check for me if this solution fixes it 😄

This file should already exists:

/opt/miniconda3/envs/seq2science/lib/python3.8/site-packages/seq2science/envs/bedtools.yaml

Can you change it so that it has coreutils added:

name: bedtools channels:

And then try seq2science again?

— Reply to this email directly, view it on GitHubhttp://pta-smg3.csir.co.za:32224/?dmVyPTEuMDAxJiZhYTE0OTAyYzdmZTE1YWMxMj02MzkwQTU5Q18zNjY5XzEyNjQ5XzEmJjExNTZhOTFhMDE3YzllZD0xMzMzJiZ1cmw9aHR0cHMlM0ElMkYlMkZnaXRodWIlMkVjb20lMkZ2YW5oZWVyaW5nZW4tbGFiJTJGc2VxMnNjaWVuY2UlMkZpc3N1ZXMlMkY5MzklMjNpc3N1ZWNvbW1lbnQtMTM0MTA2MzM5NQ==, or unsubscribehttp://pta-smg3.csir.co.za:32224/?dmVyPTEuMDAxJiZiMjFhOGEyZDdjZWQ0YmM5Mz02MzkwQTU5Q18zNjY5XzEyNjQ5XzEmJmIxYjYwOTFmZTU0ZGRlMT0xMzMzJiZ1cmw9aHR0cHMlM0ElMkYlMkZnaXRodWIlMkVjb20lMkZub3RpZmljYXRpb25zJTJGdW5zdWJzY3JpYmUtYXV0aCUyRkE0VFlHM0VFT1VXQ042SzRHN0ZMNkQzV01DT1JUQU5DTkZTTTZBQUFBQUFTVzIzWFlF. You are receiving this because you authored the thread.Message ID: @.***>

Maarten-vd-Sande commented 1 year ago

Have you had the time to test if this works?

Jerolen commented 1 year ago

Hi Maarten

Thanks for the follow up. I tried running it a a few more times without success. I'm not sure if my system runs out of RAM at the end.

I was running it locally, with 4 cores and 16GB RAM but I noticed that before the crashes it would say available memory at 3GB.

Any advice would be greatly appreciated.

Kind Regards

Jerolen

Jerolen Naidoo PhD Senior Researcher Bioengineering & Integrated Genomics Group (BIGG) CSIR NextGen Health Building 20 Meiring Naude road Brummeria Pretoria 0002

From: Maarten-vd-Sande @.> Sent: Monday, 12 December 2022, 12:18 To: vanheeringen-lab/seq2science @.> Cc: Jerolen Naidoo @.>; Author @.> Subject: Re: [vanheeringen-lab/seq2science] Issue with Chip-seq pipeline: jobid: 27: one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode! (Issue #939)

You don't often get email from @.*** Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

Have you had the time to test if this works?

— Reply to this email directly, view it on GitHubhttp://pta-smg3.csir.co.za:32224/?dmVyPTEuMDAxJiYyMzA1NDQ4Y2MxZTJiNTM3Mj02Mzk2RkZFMV83MTY3MV8xODE2OF8xJiZkMGQ5ODM5ZTE5ZTVlZDA9MTMzMyYmdXJsPWh0dHBzJTNBJTJGJTJGZ2l0aHViJTJFY29tJTJGdmFuaGVlcmluZ2VuLWxhYiUyRnNlcTJzY2llbmNlJTJGaXNzdWVzJTJGOTM5JTIzaXNzdWVjb21tZW50LTEzNDYyMjQ1Mjk=, or unsubscribehttp://pta-smg3.csir.co.za:32224/?dmVyPTEuMDAxJiYyMzFkNGE5NmMwZTFiOTI2Mj02Mzk2RkZFMV83MTY3MV8xODE2OF8xJiY1MTc5NjMzZTFjMDFkYzQ9MTMzMyYmdXJsPWh0dHBzJTNBJTJGJTJGZ2l0aHViJTJFY29tJTJGbm90aWZpY2F0aW9ucyUyRnVuc3Vic2NyaWJlLWF1dGglMkZBNFRZRzNINkZLN09LUUVYM1JOMlY0RFdNMzNWNUFOQ05GU002QUFBQUFBU1cyM1hZRQ==. You are receiving this because you authored the thread.Message ID: @.***>

Maarten-vd-Sande commented 1 year ago

Not sure, 16GB isn't too much indeed for preprocessing.. Which step(s) is the one crashing? Still the random_subset_peaks rule? Most rules make a logfile, perhaps there is a clue about what goes wrong?

https://vanheeringen-lab.github.io/seq2science/content/faq.html#one-of-the-rules-failed-and-the-log-turns-red

Maarten-vd-Sande commented 1 year ago

Your issue(s) should be solved with the latest 0.9.7 release. Let me know if you encounter anything else!!!

Jerolen commented 1 year ago

Thanks Maarten

I just got back from the break so will try again and give you an update. All the best for 2023.

Kind Regards

Jerolen

Jerolen Naidoo, PhD

Senior Researcher Bioengineering & Integrative Genomics Group (BIGG) Next Generation Health Cluster CSIR Building 20 Meiring Naude Road Pretoria

Tel: 012 841 3940 Fax: 012 841 2615 website: https://csir.co.za/

From: Maarten-vd-Sande @.> Sent: Tuesday, 03 January 2023 16:34 To: vanheeringen-lab/seq2science @.> Cc: Jerolen Naidoo @.>; Author @.> Subject: Re: [vanheeringen-lab/seq2science] Issue with Chip-seq pipeline: jobid: 27: one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode! (Issue #939)

You don't often get email from @.*** Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

Your issue(s) should be solved with the latest 0.9.7 release. Let me know if you encounter anything else!!!

— Reply to this email directly, view it on GitHubhttp://pta-smg3.csir.co.za:32224/?dmVyPTEuMDAxJiY3YjJiMGIwMjY1ZWVlOTkwMz02M0I0M0NFRl83NzAzMV8zNzhfMSYmODFiMjdjNGNhMWY4Mzg1PTEzMzMmJnVybD1odHRwcyUzQSUyRiUyRmdpdGh1YiUyRWNvbSUyRnZhbmhlZXJpbmdlbi1sYWIlMkZzZXEyc2NpZW5jZSUyRmlzc3VlcyUyRjkzOSUyM2lzc3VlY29tbWVudC0xMzY5ODM3NTg3, or unsubscribehttp://pta-smg3.csir.co.za:32224/?dmVyPTEuMDAxJiY3YjMzMDUxODY0ZWRlNTgxMz02M0I0M0NFRl83NzAzMV8zNzhfMSYmMDAxMjljZWNhNDFjMDkxPTEzMzMmJnVybD1odHRwcyUzQSUyRiUyRmdpdGh1YiUyRWNvbSUyRm5vdGlmaWNhdGlvbnMlMkZ1bnN1YnNjcmliZS1hdXRoJTJGQTRUWUczRkdNNlFPUUpaNEJLWFNNT1RXUVEyR1hBTkNORlNNNkFBQUFBQVNXMjNYWUU=. You are receiving this because you authored the thread.Message ID: @.***>