Closed lawtyunc closed 4 weeks ago
Exit status 143 typically occurs either when the memory request, or the wallclock request has been exceeded.
Can you provide the number of reads in S3_R1_001.fastq.gz
?
The second thing to check would be to look in the directory that should be printed to the end of the Nextflow log:
For example:
Command executed:
Rscript my_script.R
Command exit status:
143
Command output:
...text....
Command error:
...text...
Work dir:
/scratch/$USER/workdir/eb/33d560d5f3e2ee78d5db43ba072228
In this example, you would navigate to: /scratch/$USER/workdir/eb/33d560d5f3e2ee78d5db43ba072228
and look at the .command.log
file. What (if anything) is appended to the end of that log file? E.g., if you are on slurm you would likely see some epilogue information (memory usage, wall clock usage, etc.). Does that log report memory usage that exceeds the request (200GB).
This step GBRS_BAM2EMASE
is known to be memory intensive. To fix this issue, increase the memory allocation in the modules/gbrs/gbrs_bam2emase.nf
file beyond 200GB.
@lawtyunc were you successful in troubleshooting this issue?
Yes we were able to fix this but now we are running into this docker image issue.
Caused by: Failed to pull singularity image command: singularity pull --name broadinstitute-gatk-4.2.4.1.img.pulling.1722359365671 docker://broadinstitute/gatk:4.2.4.1 > /dev/null status : 143
From: Mike Lloyd @.> Sent: Tuesday, July 30, 2024 3:19 PM To: TheJacksonLaboratory/cs-nf-pipelines @.> Cc: Laws, Tyler @.>; Mention @.> Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)
You don't often get email from @.*** Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification
@lawtyunchttps://github.com/lawtyunc were you successful in troubleshooting this issue?
— Reply to this email directly, view it on GitHubhttps://github.com/TheJacksonLaboratory/cs-nf-pipelines/issues/6#issuecomment-2259042727, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB62MSYGOWQ2XXBKFAVNHXDZO7RKXAVCNFSM6AAAAABK4WO66GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJZGA2DENZSG4. You are receiving this because you were mentioned.Message ID: @.***>
The fact that this fails during the command: singularity pull
step along with status: 143
suggests that you are not providing enough memory to the Nextflow job, or that you do not have enough memory on the node/system you are trying to run the Nextflow command on.
Nextflow builds and caches containers for modules as needed. The build/pull is done by Nextflow, and isn't captured by the specific memory reservations for each compute step. So however much memory you have on the node where nextflow run ...
is being run, is the memory limit provided to singularity pull ...
I would suggest increasing the memory request to your main nextflow run ...
job and see if this issue resolves.
Unfortunately, beyond that recommendation, I am unable to troubleshoot system/HPC specific issues on systems external to Jax. There are too many possible configuration and options to understand the specifics in such cases.
Given that your initial issue is resolved, I will close this ticket, but please feel free to open a new issue if other pipeline related things come up.
Also one question I had about the GBRS pipeline. Based on the outputs do you know what files or references you could use to make a kinship matrix?
From: Mike Lloyd @.> Sent: Tuesday, July 30, 2024 3:47 PM To: TheJacksonLaboratory/cs-nf-pipelines @.> Cc: Laws, Tyler @.>; Mention @.> Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)
You don't often get email from @.*** Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification
The fact that this fails during the command: singularity pull step along with status: 143 suggests that you are not providing enough memory to the Nextflow job, or that you do not have enough memory on the node/system you are trying to run the Nextflow command on.
Nextflow builds and caches containers for modules as needed. The build/pull is done by Nextflow, and isn't captured by the specific memory reservations for each compute step. So however much memory you have on the node where nextflow run ... is being run, is the memory limit provided to singularity pull ...
I would suggest increasing the memory request to your main nextflow run ... job and see if this issue resolves.
Unfortunately, beyond that recommendation, I am unable to troubleshoot system/HPC specific issues on systems external to Jax. There are too many possible configuration and options to understand the specifics in such cases.
Given that your initial issue is resolved, I will close this ticket, but please feel free to open a new issue if other pipeline related things come up.
— Reply to this email directly, view it on GitHubhttps://github.com/TheJacksonLaboratory/cs-nf-pipelines/issues/6#issuecomment-2259090444, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB62MS5AF6DKJQMAWQD2W2LZO7UWFAVCNFSM6AAAAABK4WO66GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJZGA4TANBUGQ. You are receiving this because you were mentioned.Message ID: @.***>
I would suggest the genoprobs object that is output from GBRS, and perhaps the QTL2 package.
Hey Mike, I thank you for the suggestion. That is actually the package that I found and have been looking into as well. I was wondering. Do you know how this would work for the control data they mention as an input or how that would be generated?
From: Mike Lloyd @.> Sent: Monday, August 5, 2024 10:15 AM To: TheJacksonLaboratory/cs-nf-pipelines @.> Cc: Laws, Tyler @.>; Mention @.> Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)
I would suggest the genoprobs object that is output from GBRS, and perhaps the QTL2https://kbroman.org/qtl2/ package.
— Reply to this email directly, view it on GitHubhttps://github.com/TheJacksonLaboratory/cs-nf-pipelines/issues/6#issuecomment-2269186567, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB62MS7G3MCEDZVHFLHJZSTZP6CHVAVCNFSM6AAAAABK4WO66GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRZGE4DMNJWG4. You are receiving this because you were mentioned.Message ID: @.***>
Also when you say Genoprobs object are you talking about the output npz file?
[cid:64ceb246-7c09-4f08-bf7a-23d6b778632b]
From: Laws, Tyler @.> Sent: Monday, August 5, 2024 10:25 AM To: TheJacksonLaboratory/cs-nf-pipelines @.> Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)
Hey Mike, I thank you for the suggestion. That is actually the package that I found and have been looking into as well. I was wondering. Do you know how this would work for the control data they mention as an input or how that would be generated?
From: Mike Lloyd @.> Sent: Monday, August 5, 2024 10:15 AM To: TheJacksonLaboratory/cs-nf-pipelines @.> Cc: Laws, Tyler @.>; Mention @.> Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)
I would suggest the genoprobs object that is output from GBRS, and perhaps the QTL2https://kbroman.org/qtl2/ package.
— Reply to this email directly, view it on GitHubhttps://github.com/TheJacksonLaboratory/cs-nf-pipelines/issues/6#issuecomment-2269186567, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB62MS7G3MCEDZVHFLHJZSTZP6CHVAVCNFSM6AAAAABK4WO66GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRZGE4DMNJWG4. You are receiving this because you were mentioned.Message ID: @.***>
per the wiki: https://github.com/TheJacksonLaboratory/cs-nf-pipelines/wiki/GBRS-Pipeline-ReadMe
*/gbrs/*.interpolated.genoprobs.tsv
: Interpolated genotype probabilities in tab delimited format
Beyond this, I am not able to help with your analysis.
Hey Mike, Another question I had was how did you guys create the "/Mus_musculus.GRCm38.dna.toplevel.primaryChr.contig_list" file for the WGS pipeline. Also would you happen to have this available for GRCm39?
From: Mike Lloyd @.> Sent: Tuesday, August 6, 2024 11:11 AM To: TheJacksonLaboratory/cs-nf-pipelines @.> Cc: Laws, Tyler @.>; Mention @.> Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)
per the wiki: https://github.com/TheJacksonLaboratory/cs-nf-pipelines/wiki/GBRS-Pipeline-ReadMe
/gbrs/.interpolated.genoprobs.tsv: Interpolated genotype probabilities in tab delimited format
Beyond this, I am not able to help with your analysis.
— Reply to this email directly, view it on GitHubhttps://github.com/TheJacksonLaboratory/cs-nf-pipelines/issues/6#issuecomment-2271533665, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB62MS24E4GZPDDQRZHJW43ZQDRSLAVCNFSM6AAAAABK4WO66GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZRGUZTGNRWGU. You are receiving this because you were mentioned.
Hi Jackson Lab, I was running your cs-nf-GBRS pipeline and was were able to have it work over one sample but when we tried to run this over multiple samples, we run into this error.
Caused by: Process
GBRS:RUN_EMASE:GBRS_BAM2EMASE (S3)
terminated with an error exit status (143)Command executed:
gbrs bam2emase -i S3_mapped_R2.bam -m /path/to/output/emase.fullTranscripts.info -h A,B,C,D,E,F,G,H -o S3_mapped_R2.emase.h5
Command exit status: 143 We have also been seeing this error as well with exit status 143
Command executed:
set -o pipefail
zcat S3_R1_001.fastq.gz \ | bowtie -p 8 -q -a --best --strata --sam -v 3 -x /path/to/output/bowtie.transcripts - 2> S3.bowtie_R1.log \ | samtools view -bS - > S3_mapped_R1.bam
Command exit status: 143
Do you know what may be causing this issue?