TheJacksonLaboratory / cs-nf-pipelines

The Jackson Laboratory Computational Sciences Nextflow based analysis pipelines
MIT License
18 stars 10 forks source link

Command exit status: 143 #6

Closed lawtyunc closed 4 weeks ago

lawtyunc commented 1 month ago

Hi Jackson Lab, I was running your cs-nf-GBRS pipeline and was were able to have it work over one sample but when we tried to run this over multiple samples, we run into this error.

Caused by: Process GBRS:RUN_EMASE:GBRS_BAM2EMASE (S3) terminated with an error exit status (143)

Command executed:

gbrs bam2emase -i S3_mapped_R2.bam -m /path/to/output/emase.fullTranscripts.info -h A,B,C,D,E,F,G,H -o S3_mapped_R2.emase.h5

Command exit status: 143 We have also been seeing this error as well with exit status 143

Command executed:

set -o pipefail

zcat S3_R1_001.fastq.gz \ | bowtie -p 8 -q -a --best --strata --sam -v 3 -x /path/to/output/bowtie.transcripts - 2> S3.bowtie_R1.log \ | samtools view -bS - > S3_mapped_R1.bam

Command exit status: 143

Do you know what may be causing this issue?

MikeWLloyd commented 1 month ago

Exit status 143 typically occurs either when the memory request, or the wallclock request has been exceeded.

Can you provide the number of reads in S3_R1_001.fastq.gz?

The second thing to check would be to look in the directory that should be printed to the end of the Nextflow log:

For example:

Command executed:

  Rscript my_script.R 

Command exit status:
  143

Command output:
  ...text....

Command error:
 ...text...

Work dir:
  /scratch/$USER/workdir/eb/33d560d5f3e2ee78d5db43ba072228

In this example, you would navigate to: /scratch/$USER/workdir/eb/33d560d5f3e2ee78d5db43ba072228 and look at the .command.log file. What (if anything) is appended to the end of that log file? E.g., if you are on slurm you would likely see some epilogue information (memory usage, wall clock usage, etc.). Does that log report memory usage that exceeds the request (200GB).

This step GBRS_BAM2EMASE is known to be memory intensive. To fix this issue, increase the memory allocation in the modules/gbrs/gbrs_bam2emase.nf file beyond 200GB.

MikeWLloyd commented 4 weeks ago

@lawtyunc were you successful in troubleshooting this issue?

lawtyunc commented 4 weeks ago

Yes we were able to fix this but now we are running into this docker image issue.

Caused by: Failed to pull singularity image command: singularity pull --name broadinstitute-gatk-4.2.4.1.img.pulling.1722359365671 docker://broadinstitute/gatk:4.2.4.1 > /dev/null status : 143


From: Mike Lloyd @.> Sent: Tuesday, July 30, 2024 3:19 PM To: TheJacksonLaboratory/cs-nf-pipelines @.> Cc: Laws, Tyler @.>; Mention @.> Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)

You don't often get email from @.*** Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

@lawtyunchttps://github.com/lawtyunc were you successful in troubleshooting this issue?

— Reply to this email directly, view it on GitHubhttps://github.com/TheJacksonLaboratory/cs-nf-pipelines/issues/6#issuecomment-2259042727, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB62MSYGOWQ2XXBKFAVNHXDZO7RKXAVCNFSM6AAAAABK4WO66GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJZGA2DENZSG4. You are receiving this because you were mentioned.Message ID: @.***>

MikeWLloyd commented 4 weeks ago

The fact that this fails during the command: singularity pull step along with status: 143 suggests that you are not providing enough memory to the Nextflow job, or that you do not have enough memory on the node/system you are trying to run the Nextflow command on.

Nextflow builds and caches containers for modules as needed. The build/pull is done by Nextflow, and isn't captured by the specific memory reservations for each compute step. So however much memory you have on the node where nextflow run ... is being run, is the memory limit provided to singularity pull ...

I would suggest increasing the memory request to your main nextflow run ... job and see if this issue resolves.

Unfortunately, beyond that recommendation, I am unable to troubleshoot system/HPC specific issues on systems external to Jax. There are too many possible configuration and options to understand the specifics in such cases.

Given that your initial issue is resolved, I will close this ticket, but please feel free to open a new issue if other pipeline related things come up.

lawtyunc commented 3 weeks ago

Also one question I had about the GBRS pipeline. Based on the outputs do you know what files or references you could use to make a kinship matrix?


From: Mike Lloyd @.> Sent: Tuesday, July 30, 2024 3:47 PM To: TheJacksonLaboratory/cs-nf-pipelines @.> Cc: Laws, Tyler @.>; Mention @.> Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)

You don't often get email from @.*** Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

The fact that this fails during the command: singularity pull step along with status: 143 suggests that you are not providing enough memory to the Nextflow job, or that you do not have enough memory on the node/system you are trying to run the Nextflow command on.

Nextflow builds and caches containers for modules as needed. The build/pull is done by Nextflow, and isn't captured by the specific memory reservations for each compute step. So however much memory you have on the node where nextflow run ... is being run, is the memory limit provided to singularity pull ...

I would suggest increasing the memory request to your main nextflow run ... job and see if this issue resolves.

Unfortunately, beyond that recommendation, I am unable to troubleshoot system/HPC specific issues on systems external to Jax. There are too many possible configuration and options to understand the specifics in such cases.

Given that your initial issue is resolved, I will close this ticket, but please feel free to open a new issue if other pipeline related things come up.

— Reply to this email directly, view it on GitHubhttps://github.com/TheJacksonLaboratory/cs-nf-pipelines/issues/6#issuecomment-2259090444, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB62MS5AF6DKJQMAWQD2W2LZO7UWFAVCNFSM6AAAAABK4WO66GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJZGA4TANBUGQ. You are receiving this because you were mentioned.Message ID: @.***>

MikeWLloyd commented 3 weeks ago

I would suggest the genoprobs object that is output from GBRS, and perhaps the QTL2 package.

lawtyunc commented 3 weeks ago

Hey Mike, I thank you for the suggestion. That is actually the package that I found and have been looking into as well. I was wondering. Do you know how this would work for the control data they mention as an input or how that would be generated?


From: Mike Lloyd @.> Sent: Monday, August 5, 2024 10:15 AM To: TheJacksonLaboratory/cs-nf-pipelines @.> Cc: Laws, Tyler @.>; Mention @.> Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)

I would suggest the genoprobs object that is output from GBRS, and perhaps the QTL2https://kbroman.org/qtl2/ package.

— Reply to this email directly, view it on GitHubhttps://github.com/TheJacksonLaboratory/cs-nf-pipelines/issues/6#issuecomment-2269186567, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB62MS7G3MCEDZVHFLHJZSTZP6CHVAVCNFSM6AAAAABK4WO66GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRZGE4DMNJWG4. You are receiving this because you were mentioned.Message ID: @.***>

lawtyunc commented 3 weeks ago

Also when you say Genoprobs object are you talking about the output npz file?

[cid:64ceb246-7c09-4f08-bf7a-23d6b778632b]


From: Laws, Tyler @.> Sent: Monday, August 5, 2024 10:25 AM To: TheJacksonLaboratory/cs-nf-pipelines @.> Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)

Hey Mike, I thank you for the suggestion. That is actually the package that I found and have been looking into as well. I was wondering. Do you know how this would work for the control data they mention as an input or how that would be generated?


From: Mike Lloyd @.> Sent: Monday, August 5, 2024 10:15 AM To: TheJacksonLaboratory/cs-nf-pipelines @.> Cc: Laws, Tyler @.>; Mention @.> Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)

I would suggest the genoprobs object that is output from GBRS, and perhaps the QTL2https://kbroman.org/qtl2/ package.

— Reply to this email directly, view it on GitHubhttps://github.com/TheJacksonLaboratory/cs-nf-pipelines/issues/6#issuecomment-2269186567, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB62MS7G3MCEDZVHFLHJZSTZP6CHVAVCNFSM6AAAAABK4WO66GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRZGE4DMNJWG4. You are receiving this because you were mentioned.Message ID: @.***>

MikeWLloyd commented 3 weeks ago

per the wiki: https://github.com/TheJacksonLaboratory/cs-nf-pipelines/wiki/GBRS-Pipeline-ReadMe

*/gbrs/*.interpolated.genoprobs.tsv: Interpolated genotype probabilities in tab delimited format

Beyond this, I am not able to help with your analysis.

lawtyunc commented 2 weeks ago

Hey Mike, Another question I had was how did you guys create the "/Mus_musculus.GRCm38.dna.toplevel.primaryChr.contig_list" file for the WGS pipeline. Also would you happen to have this available for GRCm39?


From: Mike Lloyd @.> Sent: Tuesday, August 6, 2024 11:11 AM To: TheJacksonLaboratory/cs-nf-pipelines @.> Cc: Laws, Tyler @.>; Mention @.> Subject: Re: [TheJacksonLaboratory/cs-nf-pipelines] Command exit status: 143 (Issue #6)

per the wiki: https://github.com/TheJacksonLaboratory/cs-nf-pipelines/wiki/GBRS-Pipeline-ReadMe

/gbrs/.interpolated.genoprobs.tsv: Interpolated genotype probabilities in tab delimited format

Beyond this, I am not able to help with your analysis.

— Reply to this email directly, view it on GitHubhttps://github.com/TheJacksonLaboratory/cs-nf-pipelines/issues/6#issuecomment-2271533665, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BB62MS24E4GZPDDQRZHJW43ZQDRSLAVCNFSM6AAAAABK4WO66GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZRGUZTGNRWGU. You are receiving this because you were mentioned.