gokalpcelik
commented
1 year ago I will also put the topic link for reference. https://gatk.broadinstitute.org/hc/en-us/community/posts/18885002525467-Error-in-running-SplitNCigarReads-Attempt-to-add-record-to-closed-writer
Hi All,
Affected tool(s) or class(es)
gatk SplitNCigarReads
Affected version(s)
Version 4.2.5.0
I'm having a issue with running SplitNCigarReads on my local cluster. When I try to run the following command:
!/bin/bash -l
$ -l h_rt=48:00:0
$ -l mem=12G
$ -l tmpfs=20G
$ -N bam_recalibration_rna
$ -wd /home/regmvcr/Scratch/workspace/JSBF
$ -pe smp 2
$ -t 1-128
shopt -s expand_aliases
cd $TMPDIR
Load dependencies on Myriad
module load java/temurin-8/8u322-b06 module load gatk/4.2.5.0
Parse parameter files for inputs
number=$SGE_TASK_ID
paramfile=/home/regmvcr/Scratch/jobscripts/JSBF/file_names.txt
SAMPLE="
sed -n ${number}p $paramfile | awk '{print $1}'"run SplitNCigarReads
gatk SplitNCigarReads \ -R /home/regmvcr/Scratch/reference/sarek/resources_broad_hg38_v0_Homo_sapiens_assembly38.fasta \ -I "/home/regmvcr/Scratch/workspace/JSBF/star_salmon/"$SAMPLE".markdup.sorted.bam" \ -O "/home/regmvcr/Scratch/workspace/JSBF/SplitNCigarReads/"$SAMPLE"_split.bam"
I get the following output:
GATK: Some GATK tools require conda and associated libraries. To use them run: module load python/miniconda3/4.10.3 source $UCL_CONDA_PATH/etc/profile.d/conda.sh conda activate $GATK_CONDA Using GATK jar /shared/ucl/apps/gatk-bsd/4.2.5.0/gatk-4.2.5.0/gatk-package-4.2.5.0-local.jar defined in environment variable GATK_LOCAL_JAR Running: java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /shar ed/ucl/apps/gatk-bsd/4.2.5.0/gatk-4.2.5.0/gatk-package-4.2.5.0-local.jar SplitNCigarReads -R /home/regmvcr/Scratch/reference/sarek/resources_broad_hg38_v0_Homo_sapiens_assemb ly38.fasta -I /home/regmvcr/Scratch/workspace/JSBF/star_salmon/I3O-MC-JSBF-100-1003.markdup.sorted.bam -O /home/regmvcr/Scratch/workspace/JSBF/SplitNCigarReads/I3O-MC-JSBF-10 0-1003_split.bam 19:40:24.551 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/lustre/shared/ucl/apps/gatk-bsd/4.2.5.0/gatk-4.2.5.0/gatk-package-4.2.5.0-local.jar!/com /intel/gkl/native/libgkl_compression.so Sep 14, 2023 7:40:24 PM shaded.cloud_nio.com.google.auth.oauth2.ComputeEngineCredentials runningOnComputeEngine INFO: Failed to detect whether we are running on Google Compute Engine. 19:40:24.716 INFO SplitNCigarReads - ------------------------------------------------------------ 19:40:24.716 INFO SplitNCigarReads - The Genome Analysis Toolkit (GATK) v4.2.5.0 19:40:24.716 INFO SplitNCigarReads - For support and documentation go to https://software.broadinstitute.org/gatk/ 19:40:24.717 INFO SplitNCigarReads - Executing as regmvcr@node-h00a-012.myriad.ucl.ac.uk on Linux v3.10.0-1160.53.1.el7.x86_64 amd64 19:40:24.717 INFO SplitNCigarReads - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_322-b06 19:40:24.717 INFO SplitNCigarReads - Start Date/Time: September 14, 2023 7:40:24 PM BST 19:40:24.717 INFO SplitNCigarReads - ------------------------------------------------------------ 19:40:24.717 INFO SplitNCigarReads - ------------------------------------------------------------ 19:40:24.718 INFO SplitNCigarReads - HTSJDK Version: 2.24.1 19:40:24.718 INFO SplitNCigarReads - Picard Version: 2.25.4 19:40:24.718 INFO SplitNCigarReads - Built for Spark Version: 2.4.5 19:40:24.718 INFO SplitNCigarReads - HTSJDK Defaults.COMPRESSION_LEVEL : 2 19:40:24.718 INFO SplitNCigarReads - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false 19:40:24.718 INFO SplitNCigarReads - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true 19:40:24.719 INFO SplitNCigarReads - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false 19:40:24.719 INFO SplitNCigarReads - Deflater: IntelDeflater 19:40:24.719 INFO SplitNCigarReads - Inflater: IntelInflater 19:40:24.719 INFO SplitNCigarReads - GCS max retries/reopens: 20 19:40:24.719 INFO SplitNCigarReads - Requester pays: disabled 19:40:24.719 INFO SplitNCigarReads - Initializing engine 19:40:25.434 INFO SplitNCigarReads - Done initializing engine 19:40:25.543 INFO ProgressMeter - Starting traversal 19:40:25.543 INFO ProgressMeter - Current Locus Elapsed Minutes Reads Processed Reads/Minute 19:40:35.584 INFO ProgressMeter - chr1:1755459 0.2 295000 1762948.2 19:40:45.611 INFO ProgressMeter - chr1:11863327 0.3 564000 1686350.7 19:40:55.620 INFO ProgressMeter - chr1:16582638 0.5 1006000 2006849.1 19:41:05.622 INFO ProgressMeter - chr1:21258696 0.7 1275000 1908730.3 ... 20:44:06.519 INFO ProgressMeter - chr22:41867223 63.7 158095000 2482533.3 20:44:16.525 INFO ProgressMeter - chrX:2719670 63.8 158464000 2481828.4 20:44:26.531 INFO ProgressMeter - chrX:20161751 64.0 158966000 2483204.8 20:44:36.543 INFO ProgressMeter - chrX:41343166 64.2 159351000 2482747.3 20:44:46.545 INFO ProgressMeter - chrX:53216185 64.4 159757000 2482625.0 20:44:56.546 INFO ProgressMeter - chrX:72273898 64.5 160125000 2481914.9 20:45:06.573 INFO ProgressMeter - chrX:79200408 64.7 160415000 2479986.0 20:45:16.611 INFO ProgressMeter - chrX:108571833 64.9 160804000 2479586.6 20:45:26.622 INFO ProgressMeter - chrX:130138642 65.0 161157000 2478652.7 20:45:36.719 INFO ProgressMeter - chrX:153782051 65.2 161530000 2477975.9 20:45:47.140 INFO ProgressMeter - chrY:10825619 65.4 162041000 2479210.0 20:45:57.139 INFO ProgressMeter - chrM:2699 65.5 162766000 2483968.3 20:46:07.139 INFO ProgressMeter - chr22_KI270733v1_random:134787 65.7 164454000 2503361.6 20:46:17.167 INFO ProgressMeter - chrUn_GL000220v1:113440 65.9 166153000 2522805.8 20:46:27.276 INFO ProgressMeter - chrUn_GL000220v1:157564 66.0 167306000 2533830.5 20:46:30.851 WARN IntelInflater - Zero Bytes Written : 0 20:46:30.860 INFO SplitNCigarReads - 0 read(s) filtered by: AllowAllReadsReadFilter
20:46:30.863 INFO OverhangFixingManager - Overhang Fixing Manager saved 694454 reads in the first pass 20:46:30.892 INFO SplitNCigarReads - Starting traversal pass 2 20:46:37.284 INFO ProgressMeter - chr1:3404725 66.2 168146000 2540135.4 20:46:48.282 INFO ProgressMeter - chr1:10291091 66.4 168321000 2535757.4 20:46:58.713 INFO ProgressMeter - chr1:16564851 66.6 168634000 2533836.5 20:47:08.746 INFO ProgressMeter - chr1:19113706 66.7 168965000 2532447.8 20:47:18.763 INFO ProgressMeter - chr1:21890055 66.9 169199000 2529625.2 20:47:29.315 INFO ProgressMeter - chr1:25813920 67.1 169452000 2526764.1 20:47:39.314 INFO ProgressMeter - chr1:28508573 67.2 169734000 2524694.6 20:47:49.443 INFO ProgressMeter - chr1:31692007 67.4 170037000 2522866.5 20:47:59.453 INFO ProgressMeter - chr1:36344170 67.6 170275000 2520159.5 20:48:09.458 INFO ProgressMeter - chr1:39447509 67.7 170537000 2517823.3 20:48:19.478 INFO ProgressMeter - chr1:45013600 67.9 170770000 2515062.2 20:48:29.490 INFO ProgressMeter - chr1:52525685 68.1 170995000 2512202.7 20:48:39.501 INFO ProgressMeter - chr1:63549257 68.2 171193000 2508960.8 20:48:49.565 INFO ProgressMeter - chr1:77632876 68.4 171475000 2506931.0 20:48:59.572 INFO ProgressMeter - chr1:89010984 68.6 171781000 2505295.9 20:49:09.869 INFO ProgressMeter - chr1:92837514 68.7 172031000 2502678.0 20:49:20.131 INFO ProgressMeter - chr1:100891510 68.9 172297000 2500326.5 20:49:30.138 INFO ProgressMeter - chr1:110345420 69.1 172598000 2498647.0 20:49:40.160 INFO ProgressMeter - chr1:117442273 69.2 172913000 2497168.8 20:49:50.174 INFO ProgressMeter - chr1:120463528 69.4 173200000 2495299.1 20:50:00.184 INFO ProgressMeter - chr1:120826424 69.6 173559000 2494475.6 20:50:10.196 INFO ProgressMeter - chr1:120838884 69.7 173996000 2494773.2 20:50:20.326 INFO ProgressMeter - chr1:144429847 69.9 174352000 2493840.6 20:50:30.333 INFO ProgressMeter - chr1:145584694 70.1 174825000 2494654.9 20:50:40.338 INFO ProgressMeter - chr1:146077591 70.2 175095000 2492576.7 ... 21:17:56.047 INFO ProgressMeter - chr6:31828873 97.5 222011000 2276839.7 21:18:06.108 INFO ProgressMeter - chr6:31829365 97.7 222402000 2276934.0 21:18:16.447 INFO ProgressMeter - chr6:31829459 97.8 222828000 2277277.9 21:18:26.537 INFO ProgressMeter - chr6:31871533 98.0 223264000 2277819.0 21:18:36.548 INFO ProgressMeter - chr6:32031575 98.2 223609000 2277462.0 21:18:46.550 INFO ProgressMeter - chr6:33694905 98.4 223890000 2276459.3 21:18:56.882 INFO ProgressMeter - chr6:42652593 98.5 224124000 2274855.2 21:19:06.922 INFO ProgressMeter - chr6:46150821 98.7 224356000 2273348.8 21:19:16.925 INFO ProgressMeter - chr6:56607518 98.9 224691000 2272903.7 21:19:26.951 INFO ProgressMeter - chr6:72182472 99.0 224990000 2272087.7 21:19:36.956 INFO ProgressMeter - chr6:73519441 99.2 225334000 2271736.1 21:19:44.136 INFO SplitNCigarReads - Shutting down engine [September 14, 2023 9:19:44 PM BST] org.broadinstitute.hellbender.tools.walkers.rnaseq.SplitNCigarReads done. Elapsed time: 99.33 minutes. Runtime.totalMemory()=5453119488 htsjdk.samtools.util.RuntimeIOException: Attempt to add record to closed writer. at htsjdk.samtools.util.AbstractAsyncWriter.write(AbstractAsyncWriter.java:57) at htsjdk.samtools.AsyncSAMFileWriter.addAlignment(AsyncSAMFileWriter.java:53) at org.broadinstitute.hellbender.utils.read.SAMFileGATKReadWriter.addRead(SAMFileGATKReadWriter.java:21) at org.broadinstitute.hellbender.tools.walkers.rnaseq.OverhangFixingManager.writeReads(OverhangFixingManager.java:358) at org.broadinstitute.hellbender.tools.walkers.rnaseq.OverhangFixingManager.flush(OverhangFixingManager.java:338) at org.broadinstitute.hellbender.tools.walkers.rnaseq.SplitNCigarReads.closeTool(SplitNCigarReads.java:192) at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1091) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:140) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:192) at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:211) at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160) at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203) at org.broadinstitute.hellbender.Main.main(Main.java:289)
This looks similar to issue #8232, but I've not been able to solve this using any of the fixes suggested on that page. I'm using Java version 8, the input BAMs were mapped using STAR via nfcore rnaseq without issue and I have plenty of space on my disk drive, even when specifying a temp directory. It also doesn't matter if I run the tool independently using the command above or as part of a pre-configured pipeline, and I get the same issue with SplitNCigarReads acting as if it has running out of space.
How do I fix this? I need this step to run variant calling on my rnaseq samples (I'm using the GATK best practices pipeline).