riederd
commented
9 months ago This is a GATK error. Just to be sure, can you try to manually reproduce it by:
cd /QRISdata/Q6373/results/results_1/work/32/4f5f39ab6af08ba9cc90eed3361958
bash .command.runClosed amoschoomy closed 8 months ago
riederd
commented
9 months ago This is a GATK error. Just to be sure, can you try to manually reproduce it by:
cd /QRISdata/Q6373/results/results_1/work/32/4f5f39ab6af08ba9cc90eed3361958
bash .command.run
amoschoomy
commented
9 months ago Hi sorry for the late reply, I've actually rerun it but I ran into the same error but on a different work directory. And I ran what you said and run into ln: failed to create symbolic link 'sample1_0039-scattered.interval_list.vcf.gz': File exists
I've uploaded the work directory for your use
riederd
commented
9 months ago It seems that /QRISdata/Q6373/results/results_1/work/f3/1a75384eb71df85c5cac8ba99d4b6c/sample1_0015-scattered.interval_list-f1r2.tar.gz does not exist.
Can you change into /QRISdata/Q6373/results/results_1/work/f3/1a75384eb71df85c5cac8ba99d4b6c/ and post the contents of the directory as tar.gz.?
amoschoomy
commented
9 months ago Hi, I've checked it and the file existed but nevertheless here are the directory contents work2.tar.gz
Thanks!
riederd
commented
9 months ago this is strange. May I ask you to try:
cd /QRISdata/Q6373/results/results_1/work/ae/e43079d0e13743a5bd94b0f958e0e7
singularity exec -B /QRISdata/Q6373/results/results_1/work -B /home/uqachoo1/nextNEOpi-1.4.0/bin --no-home --containall -H /scratch/temp/5890914 -B /home/uqachoo1/nextNEOpi-1.4.0/assets -B /scratch/temp/5890914 -B /QRISdata/Q6196/resources -B /home/uqachoo1/hlahd.1.7.0 -B /QRISdata/Q6196/resources/databases/iedb:/opt/iedb -B /QRISdata/Q6196/resources/databases/mhcflurry_data:/opt/mhcflurry_data /QRISdata/Q6373/results/results_1/work/singularity/apps-01.i-med.ac.at-images-singularity-nextNEOpi_1.4.0_f894e892.sif /bin/bashand then
gatk LearnReadOrientationModel \
--tmp-dir /scratch/temp/5890914 \
-I sample1_0039-scattered.interval_list-f1r2.tar.gz -I sample1_0003-scattered.interval_list-f1r2.tar.gz -I sample1_0017-scattered.interval_list-f1r2.tar.gz -I sample1_0029-scattered.interval_list-f1r2.tar.gz -I sample1_0025-scattered.interval_list-f1r2.tar.gz -I sample1_0009-scattered.interval_list-f1r2.tar.gz -I sample1_0002-scattered.interval_list-f1r2.tar.gz -I sample1_0027-scattered.interval_list-f1r2.tar.gz -I sample1_0036-scattered.interval_list-f1r2.tar.gz -I sample1_0021-scattered.interval_list-f1r2.tar.gz -I sample1_0028-scattered.interval_list-f1r2.tar.gz -I sample1_0012-scattered.interval_list-f1r2.tar.gz -I sample1_0004-scattered.interval_list-f1r2.tar.gz -I sample1_0033-scattered.interval_list-f1r2.tar.gz -I sample1_0014-scattered.interval_list-f1r2.tar.gz -I sample1_0019-scattered.interval_list-f1r2.tar.gz -I sample1_0011-scattered.interval_list-f1r2.tar.gz -I sample1_0038-scattered.interval_list-f1r2.tar.gz -I sample1_0015-scattered.interval_list-f1r2.tar.gz -I sample1_0005-scattered.interval_list-f1r2.tar.gz -I sample1_0018-scattered.interval_list-f1r2.tar.gz -I sample1_0026-scattered.interval_list-f1r2.tar.gz -I sample1_0035-scattered.interval_list-f1r2.tar.gz -I sample1_0030-scattered.interval_list-f1r2.tar.gz -I sample1_0016-scattered.interval_list-f1r2.tar.gz -I sample1_0001-scattered.interval_list-f1r2.tar.gz -I sample1_0013-scattered.interval_list-f1r2.tar.gz -I sample1_0008-scattered.interval_list-f1r2.tar.gz -I sample1_0022-scattered.interval_list-f1r2.tar.gz -I sample1_0024-scattered.interval_list-f1r2.tar.gz -I sample1_0034-scattered.interval_list-f1r2.tar.gz -I sample1_0006-scattered.interval_list-f1r2.tar.gz -I sample1_0020-scattered.interval_list-f1r2.tar.gz -I sample1_0010-scattered.interval_list-f1r2.tar.gz -I sample1_0031-scattered.interval_list-f1r2.tar.gz -I sample1_0037-scattered.interval_list-f1r2.tar.gz -I sample1_0000-scattered.interval_list-f1r2.tar.gz -I sample1_0007-scattered.interval_list-f1r2.tar.gz -I sample1_0032-scattered.interval_list-f1r2.tar.gz -I sample1_0023-scattered.interval_list-f1r2.tar.gz \
-O sample1_read-orientation-model.tar.gzand
gatk LearnReadOrientationModel \
--tmp-dir /scratch/temp/5890914 \
-I sample1_0014-scattered.interval_list-f1r2.tar.gz \
-O int_0014_read-orientation-model.tar.gz
zktuong
commented
9 months ago hi, i've been experiencing this as well. I ran it manually as you suggested and I chased it down to --containall was stuffing it up in the singularity exec step. removing it manually made it work.
i'm not well versed with nextflow and nextNEOpi so could you tell me where should i disable this??
zktuong
commented
9 months ago ok sorry now i'm getting a better sense of what's happening.
for clarity, instead of the original code: https://github.com/icbi-lab/nextNEOpi/blob/fe7b21cdc0b97aae38195e5f5ac1b9851674f6b1/conf/params.config#L277-L281
The copy of my conf/params.config was the following as i had previous seen this #25
singularity {
enabled = true
autoMounts = true
runOptions = "--no-home --containall" + " -H " + params.singularityTmpMount + " -B " + params.singularityAssetsMount + " -B " + params.singularityTmpMount + " -B " + params.resourcesBaseDir + params.singularityHLAHDmount + " -B " + params.databases.IEDB_dir + ":/opt/iedb" + " -B " + params.databases.MHCFLURRY_dir + ":/opt/mhcflurry_data"
}so it seems that having --containall could make the NeoFuse works but not the gatherMutect2VCFs step.
Is there a more permanent solution? seems a bit tedious to have to run the pipeline twice (without and with --containall) as the workaround?
riederd
commented
9 months ago Hi,
the --containall run option was removed from the default in nextNEOpi see https://github.com/icbi-lab/nextNEOpi/commit/ebfb100f13f0d00070885fca0337072095f29866
If you need it for NeoFuse to work, you may put it into:
e.g.:
withName:Neofuse {
container = 'https://apps-01.i-med.ac.at/images/singularity/NeoFuse_dev_92712e07.sif'
cpus = 10
containerOptions = "--containall"
}Let me know if this works for you.
amoschoomy
commented
9 months ago Hi @riederd I've tried executing it manually and it worked (I had to specify a different $TMPDIR on that manual command as i do not have access to the temp directory at runtime).
I then reexecuted the pipeline with the full stack trace of GATK enabled and now I'm getting the same issue. Apparently I have no space left on my device? Which is weird because I've allocated 5TB on my shared project space and $TMPDIR is setup and allocated automatically by the HPC system at the submission of jobs (which would be sufficient for most of the heavy jobs?) I'm not sure how much more do GATK needs in regards to this or is there a way to reduce the space usage?
Command error:
17:39:44.852 INFO IOUtils - Extracting file: ./sample1_tumor.alt_table
17:39:44.857 INFO IOUtils - Extracting data from archive: file:///scratch/project_mnt/S0142/results/results_1/work/84/389049f48f5f9b757eb2fcc716e0fa/sample1_0028-scattered.interval_list-f1r2.tar.gz
17:39:44.858 INFO IOUtils - Extracting file: ./sample1_tumor.alt_histogram
17:39:44.859 INFO IOUtils - Extracting file: ./sample1_tumor.ref_histogram
17:39:44.859 INFO IOUtils - Extracting file: ./sample1_tumor.alt_table
17:39:44.865 INFO IOUtils - Extracting data from archive: file:///scratch/project_mnt/S0142/results/results_1/work/84/389049f48f5f9b757eb2fcc716e0fa/sample1_0032-scattered.interval_list-f1r2.tar.gz
17:39:44.867 INFO IOUtils - Extracting file: ./sample1_tumor.alt_histogram
17:39:44.867 INFO IOUtils - Extracting file: ./sample1_tumor.ref_histogram
17:39:44.868 INFO IOUtils - Extracting file: ./sample1_tumor.alt_table
17:39:44.869 INFO LearnReadOrientationModel - Shutting down engine
[October 16, 2023 at 5:39:44 PM UTC] org.broadinstitute.hellbender.tools.walkers.readorientation.LearnReadOrientationModel done. Elapsed time: 0.00 minutes.
Runtime.totalMemory()=335544320
***********************************************************************
A USER ERROR has occurred: Could not extract data from: file:///scratch/project_mnt/S0142/results/results_1/work/84/389049f48f5f9b757eb2fcc716e0fa/sample1_0032-scattered.interval_list-f1r2.tar.gz
***********************************************************************
org.broadinstitute.hellbender.exceptions.UserException: Could not extract data from: file:///scratch/project_mnt/S0142/results/results_1/work/84/389049f48f5f9b757eb2fcc716e0fa/sample1_0032-scattered.interval_list-f1r2.tar.gz
at org.broadinstitute.hellbender.utils.io.IOUtils.extractTarGz(IOUtils.java:458)
at org.broadinstitute.hellbender.utils.io.IOUtils.extractTarGz(IOUtils.java:434)
at org.broadinstitute.hellbender.tools.walkers.readorientation.LearnReadOrientationModel.lambda$doWork$1(LearnReadOrientationModel.java:86)
at java.base/java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:104)
at java.base/java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:617)
at org.broadinstitute.hellbender.tools.walkers.readorientation.LearnReadOrientationModel.doWork(LearnReadOrientationModel.java:86)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:149)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:198)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:217)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
at org.broadinstitute.hellbender.Main.main(Main.java:289)
Caused by: java.io.IOException: No space left on device
at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at java.base/sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:62)
at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:132)
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:97)
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:67)
at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:288)
at java.base/java.nio.channels.Channels.writeFullyImpl(Channels.java:74)
at java.base/java.nio.channels.Channels.writeFully(Channels.java:96)
at java.base/java.nio.channels.Channels$1.write(Channels.java:171)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1310)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:978)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1282)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:953)
at org.broadinstitute.hellbender.utils.io.IOUtils.extractFilesFromArchiveStream(IOUtils.java:523)
at org.broadinstitute.hellbender.utils.io.IOUtils.extractTarGz(IOUtils.java:455)
... 11 more
Using GATK jar /opt/gatk/gatk-package-4.4.0.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -Xmx50G -XX:ParallelGCThreads=60 -DGATK_STACKTRACE_ON_USER_EXCEPTION=true -jar /opt/gatk/gatk-package-4.4.0.0-local.jar LearnReadOrientationModel --tmp-dir /scratch/temp/5976135 -I sample1_0034-scattered.interval_list-f1r2.tar.gz -I sample1_0025-scattered.interval_list-f1r2.tar.gz -I sample1_0036-scattered.interval_list-f1r2.tar.gz -I sample1_0001-scattered.interval_list-f1r2.tar.gz -I sample1_0038-scattered.interval_list-f1r2.tar.gz -I sample1_0020-scattered.interval_list-f1r2.tar.gz -I sample1_0023-scattered.interval_list-f1r2.tar.gz -I sample1_0016-scattered.interval_list-f1r2.tar.gz -I sample1_0010-scattered.interval_list-f1r2.tar.gz -I sample1_0022-scattered.interval_list-f1r2.tar.gz -I sample1_0019-scattered.interval_list-f1r2.tar.gz -I sample1_0004-scattered.interval_list-f1r2.tar.gz -I sample1_0028-scattered.interval_list-f1r2.tar.gz -I sample1_0032-scattered.interval_list-f1r2.tar.gz -I sample1_0018-scattered.interval_list-f1r2.tar.gz -I sample1_0011-scattered.interval_list-f1r2.tar.gz -I sample1_0029-scattered.interval_list-f1r2.tar.gz -I sample1_0008-scattered.interval_list-f1r2.tar.gz -I sample1_0015-scattered.interval_list-f1r2.tar.gz -I sample1_0031-scattered.interval_list-f1r2.tar.gz -I sample1_0024-scattered.interval_list-f1r2.tar.gz -I sample1_0035-scattered.interval_list-f1r2.tar.gz -I sample1_0003-scattered.interval_list-f1r2.tar.gz -I sample1_0030-scattered.interval_list-f1r2.tar.gz -I sample1_0000-scattered.interval_list-f1r2.tar.gz -I sample1_0039-scattered.interval_list-f1r2.tar.gz -I sample1_0006-scattered.interval_list-f1r2.tar.gz -I sample1_0033-scattered.interval_list-f1r2.tar.gz -I sample1_0021-scattered.interval_list-f1r2.tar.gz -I sample1_0012-scattered.interval_list-f1r2.tar.gz -I sample1_0007-scattered.interval_list-f1r2.tar.gz -I sample1_0014-scattered.interval_list-f1r2.tar.gz -I sample1_0037-scattered.interval_list-f1r2.tar.gz -I sample1_0026-scattered.interval_list-f1r2.tar.gz -I sample1_0009-scattered.interval_list-f1r2.tar.gz -I sample1_0005-scattered.interval_list-f1r2.tar.gz -I sample1_0013-scattered.interval_list-f1r2.tar.gz -I sample1_0027-scattered.interval_list-f1r2.tar.gz -I sample1_0017-scattered.interval_list-f1r2.tar.gz -I sample1_0002-scattered.interval_list-f1r2.tar.gz -O sample1_read-orientation-model.tar.gzHi, the
--containallrun option was removed from the default innextNEOpisee ebfb100If you need it for
NeoFuseto work, you may put it into:e.g.:
withName:Neofuse { container = 'https://apps-01.i-med.ac.at/images/singularity/NeoFuse_dev_92712e07.sif' cpus = 10 containerOptions = "--containall" }Let me know if this works for you.
thanks! yes this works for me now. Both gatherMutect2VCFs and Neofuse are working now in a single run with the following modifications:
since params.config has includeConfig './process.config' above, so i modified the last parts of params.config to:
singularity {
enabled = true
autoMounts = true
runOptions = process.containerOptions + " -H " + params.singularityTmpMount + " -B " + params.singularityAssetsMount + " -B " + params.singularityTmpMount + " -B " + params.resourcesBaseDir + params.singularityHLAHDmount + " -B " + params.databases.IEDB_dir + ":/opt/iedb" + " -B " + params.databases.MHCFLURRY_dir + ":/opt/mhcflurry_data"
}and process.config to add:
// default singularity run options
containerOptions = "--no-home"
withName:Neofuse {
container = 'https://apps-01.i-med.ac.at/images/singularity/NeoFuse_dev_92712e07.sif'
cpus = 60
containerOptions = "--no-home --containall"
}Just need to see if it runs through the rest ok.
riederd
commented
8 months ago I'll implement these changes, and close the issue. Feel free to reopen if it needs more discussion
Hello again, now the pipeline is getting this error: