hartleys
commented
6 years ago Do NOT use DEXSeqFmt to create the flattened GFF. That produces a differently formatted file intended for use with other tools, not the flattened format used by qorts.
On Fri, Jun 22, 2018, 6:11 AM Chela James notifications@github.com wrote:
Hi,
I am attempting to generate data compatible with DEXSeq, I run this command to create a flattened gff:
java -jar ~/applications/QoRTs/QoRTs.jar makeFlatGff --stranded --DEXSeqFmt --verbose Homo_sapiens.GRCh38.92.gtf HS.GRCh38.92.flattened.gff
which runs succesfully
INPUT_COMMAND(makeFlatGff) INPUT_ARG(infile)=Homo_sapiens.GRCh38.92.gtf INPUT_ARG(outfile)=HS.GRCh38.92.flattened.gff INPUT_ARG(stranded)=true INPUT_ARG(dexseqFmt)=true INPUT_ARG(verbose)=true FlatteningGtf: starting...(2018-06-22 09:19:45) FlatteningGtf: gtf file read complete.(2018-06-22 09:21:36) FlatteningGtf: Splice Junction Map read.(2018-06-22 09:21:40) FlatteningGtf: gene Sets generated.(2018-06-22 09:21:42) FlatteningGtf: Aggregate Sets built. FlatteningGtf: Compiling Aggregate Info . . . (2018-06-22 09:21:42) FlatteningGtf: Finished Compiling Aggregate Info. (2018-06-22 09:21:42) FlatteningGtf: Iterating through the step-vector...(2018-06-22 09:21:42) FlatteningGtf: Adding the aggregate genes themselves...(2018-06-22 09:21:46) FlatteningGtf: Iterating through the splice junctions...(2018-06-22 09:21:47) FlatteningGtf: Sorting the aggregate genes...(2018-06-22 09:21:49) FlatteningGtf: Folding the FlatGtfLine iterator...(2018-06-22 09:21:50) FlatteningGtf: Features Built.(2018-06-22 09:21:50) FlatteningGtf: done.(2018-06-22 09:21:59) Done. (Fri Jun 22 09:21:59 BST 2018)
I then try to generates counts using the same gtf and flattened gff:
java -jar ~/applications/QoRTs/QoRTs.jar QC --runFunctions writeDEXSeq --minMAPQ 60 --maxReadLength 100 --flatgff HS.GRCh38.92.flattened.gff ~/Scratch/work/eli/project1/day_5/REH3-19/replicate_1/bam/REH3-19.replicate_1.sorted.bam Homo_sapiens.GRCh38.92.gtf ./test_qc
and I get the following error: Starting QoRTs v1.3.0 (Compiled Fri Oct 20 11:56:37 EDT 2017) Starting time: (Fri Jun 22 09:22:53 BST 2018) INPUT_COMMAND(QC)
INPUT_ARG(infile)=/home/sejjctj/Scratch/work/eli/project1/day_5/REH3-19/replicate_1/bam/REH3-19.replicate_1.sorted.bam INPUT_ARG(gtffile)=Homo_sapiens.GRCh38.92.gtf INPUT_ARG(outdir)=./test_qc INPUT_ARG(runFunctions)=List(writeDEXSeq) INPUT_ARG(minMAPQ)=60 INPUT_ARG(maxReadLength)=Some(100) INPUT_ARG(flatgfffile)=Some(HS.GRCh38.92.flattened.gff) Created Log File: ./test_qc/QC.bpKPVLzhuVmb.log Warning: run-in-progress file "./test_qc/QC.QORTS_RUNNING" already exists. Is there another QoRTs job running? Starting QC [Time: 2018-06-22 09:22:53] [Mem usage: [42MB / 1009MB]] [Elapsed Time: 00:00:00.0000] QoRTs is Running in paired-end mode. QoRTs is Running in any-sorted mode. NOTE: Function "writeDEXSeq" requires function "JunctionCalcs". Adding "JunctionCalcs" to the active function list... Running functions: JunctionCalcs, writeDEXSeq Checking first 10000 reads. Checking SAM file for formatting errors... WARNING: BAM index file /home/sejjctj/Scratch/work/eli/project1/day_5/REH3-19/replicate_1/bam/REH3-19.replicate_1.sorted.bam.bai is older than BAM /home/sejjctj/Scratch/work/eli/project1/day_5/REH3-19/replicate_1/bam/REH3-19.replicate_1.sorted.bam NOTE: Read length is not consistent. In the first 10000 reads, read length varies from 20 to 76 (param maxReadLength=100) Note that using data that is hard-clipped prior to alignment is NOT recommended, because this makes it difficult (or impossible) to determine the sequencer read-cycle of each nucleotide base. This may obfuscate cycle-specific artifacts, trends, or errors, the detection of which is one of the primary purposes of QoRTs!In addition, hard clipping (whether before or after alignment) removes quality score data, and thus quality score metrics may be misleadingly optimistic. A MUCH preferable method of removing undesired sequence is to replace such sequence with N's, which preserves the quality score and the sequencer cycle information. Sorting Note: Reads are not sorted by name (This is OK). Sorting Note: Reads are sorted by position (This is OK). Done checking first 10000 reads. WARNINGS FOUND! SAMRecord Reader Generated. Read length: 100. [Time: 2018-06-22 09:22:58] [Mem usage: [245MB / 1272MB]] [Elapsed Time: 00:00:04.0938] <====== FATAL ERROR! ======>
Error message: "FATAL ERROR: OutputGtfLine checkFmt: Attribute num not found!
" Stack Trace: java.lang.Thread.getStackTrace(Thread.java:1552) internalUtils.Reporter$.error(Reporter.scala:294) internalUtils.GtfTool$GtfLine.fmtError(GtfTool.scala:135) internalUtils.GtfTool$InputGtfLine.getAttributeOrDie(GtfTool.scala:337) internalUtils.GtfTool$FlatGtfLine.getFeaturePartNumber(GtfTool.scala:359) internalUtils.GtfTool$FlatGtfLine.getFeaturePartNumber$(GtfTool.scala:359)
internalUtils.GtfTool$FlatInputGtfLine.getFeaturePartNumber(GtfTool.scala:404) internalUtils.GtfTool$FlatGtfLine.getFeatureName(GtfTool.scala:360) internalUtils.GtfTool$FlatGtfLine.getFeatureName$(GtfTool.scala:360) internalUtils.GtfTool$FlatInputGtfLine.getFeatureName(GtfTool.scala:404)
qcUtils.qcGtfAnnotationBuilder$.$anonfun$getFlatFeatureList$1(qcGtfAnnotationBuilder.scala:175) qcUtils.qcGtfAnnotationBuilder$$$Lambda$109/1904324159.apply(Unknown Source)
scala.collection.TraversableOnce.$anonfun$foldLeft$1(TraversableOnce.scala:157)
scala.collection.TraversableOnce.$anonfun$foldLeft$1$adapted(TraversableOnce.scala:157) scala.collection.TraversableOnce$$Lambda$26/90320863.apply(Unknown Source) scala.collection.Iterator.foreach(Iterator.scala:929) scala.collection.Iterator.foreach$(Iterator.scala:929) internalUtils.GtfTool$FlatGtfReader.foreach(GtfTool.scala:465) scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:157) scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:155) internalUtils.GtfTool$FlatGtfReader.foldLeft(GtfTool.scala:465)
qcUtils.qcGtfAnnotationBuilder$.qcUtils$qcGtfAnnotationBuilder$$getFlatFeatureList(qcGtfAnnotationBuilder.scala:174)
qcUtils.qcGtfAnnotationBuilder.flatFeatureList$lzycompute(qcGtfAnnotationBuilder.scala:45)
qcUtils.qcGtfAnnotationBuilder.flatFeatureList(qcGtfAnnotationBuilder.scala:45) qcUtils.qcJunctionCounts.(qcJunctionCounts.scala:91) qcUtils.runAllQC$.runOnSeqFile(runAllQC.scala:1217) qcUtils.runAllQC$.run(runAllQC.scala:960) qcUtils.runAllQC$allQC_runner.run(runAllQC.scala:672) runner.runner$.main(runner.scala:97) runner.runner.main(runner.scala) <==========================> Exception in thread "main" java.lang.Exception: FATAL ERROR: OutputGtfLine checkFmt: Attribute num not found!
at internalUtils.Reporter$.error(Reporter.scala:299) at internalUtils.GtfTool$GtfLine.fmtError(GtfTool.scala:135) at internalUtils.GtfTool$InputGtfLine.getAttributeOrDie(GtfTool.scala:337) at internalUtils.GtfTool$FlatGtfLine.getFeaturePartNumber(GtfTool.scala:359) at internalUtils.GtfTool$FlatGtfLine.getFeaturePartNumber$(GtfTool.scala:359) at internalUtils.GtfTool$FlatInputGtfLine.getFeaturePartNumber(GtfTool.scala:404) at internalUtils.GtfTool$FlatGtfLine.getFeatureName(GtfTool.scala:360) at internalUtils.GtfTool$FlatGtfLine.getFeatureName$(GtfTool.scala:360) at internalUtils.GtfTool$FlatInputGtfLine.getFeatureName(GtfTool.scala:404) at qcUtils.qcGtfAnnotationBuilder$.$anonfun$getFlatFeatureList$1(qcGtfAnnotationBuilder.scala:175) at qcUtils.qcGtfAnnotationBuilder$$$Lambda$109/1904324159.apply(Unknown Source) at scala.collection.TraversableOnce.$anonfun$foldLeft$1(TraversableOnce.scala:157) at scala.collection.TraversableOnce.$anonfun$foldLeft$1$adapted(TraversableOnce.scala:157) at scala.collection.TraversableOnce$$Lambda$26/90320863.apply(Unknown Source) at scala.collection.Iterator.foreach(Iterator.scala:929) at scala.collection.Iterator.foreach$(Iterator.scala:929) at internalUtils.GtfTool$FlatGtfReader.foreach(GtfTool.scala:465) at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:157) at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:155) at internalUtils.GtfTool$FlatGtfReader.foldLeft(GtfTool.scala:465) at qcUtils.qcGtfAnnotationBuilder$.qcUtils$qcGtfAnnotationBuilder$$getFlatFeatureList(qcGtfAnnotationBuilder.scala:174) at qcUtils.qcGtfAnnotationBuilder.flatFeatureList$lzycompute(qcGtfAnnotationBuilder.scala:45) at qcUtils.qcGtfAnnotationBuilder.flatFeatureList(qcGtfAnnotationBuilder.scala:45) at qcUtils.qcJunctionCounts.
(qcJunctionCounts.scala:91) at qcUtils.runAllQC$.runOnSeqFile(runAllQC.scala:1217) at qcUtils.runAllQC$.run(runAllQC.scala:960) at qcUtils.runAllQC$allQC_runner.run(runAllQC.scala:672) at runner.runner$.main(runner.scala:97) at runner.runner.main(runner.scala) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hartleys/QoRTs/issues/64, or mute the thread https://github.com/notifications/unsubscribe-auth/ACwu7J5tpYNAsT_1aUt_zkUfbHVGcEFLks5t_MK3gaJpZM4Uzffa .
Hi,
I am attempting to generate data compatible with DEXSeq, I run this command to create a flattened gff:
java -jar ~/applications/QoRTs/QoRTs.jar makeFlatGff --stranded --DEXSeqFmt --verbose Homo_sapiens.GRCh38.92.gtf HS.GRCh38.92.flattened.gffwhich runs succesfully
INPUT_COMMAND(makeFlatGff) INPUT_ARG(infile)=Homo_sapiens.GRCh38.92.gtf INPUT_ARG(outfile)=HS.GRCh38.92.flattened.gff INPUT_ARG(stranded)=true INPUT_ARG(dexseqFmt)=true INPUT_ARG(verbose)=true FlatteningGtf: starting...(2018-06-22 09:19:45) FlatteningGtf: gtf file read complete.(2018-06-22 09:21:36) FlatteningGtf: Splice Junction Map read.(2018-06-22 09:21:40) FlatteningGtf: gene Sets generated.(2018-06-22 09:21:42) FlatteningGtf: Aggregate Sets built. FlatteningGtf: Compiling Aggregate Info . . . (2018-06-22 09:21:42) FlatteningGtf: Finished Compiling Aggregate Info. (2018-06-22 09:21:42) FlatteningGtf: Iterating through the step-vector...(2018-06-22 09:21:42) FlatteningGtf: Adding the aggregate genes themselves...(2018-06-22 09:21:46) FlatteningGtf: Iterating through the splice junctions...(2018-06-22 09:21:47) FlatteningGtf: Sorting the aggregate genes...(2018-06-22 09:21:49) FlatteningGtf: Folding the FlatGtfLine iterator...(2018-06-22 09:21:50) FlatteningGtf: Features Built.(2018-06-22 09:21:50) FlatteningGtf: done.(2018-06-22 09:21:59) Done. (Fri Jun 22 09:21:59 BST 2018)
I then try to generates counts using the same gtf and flattened gff:
java -jar ~/applications/QoRTs/QoRTs.jar QC --runFunctions writeDEXSeq --minMAPQ 60 --maxReadLength 100 --flatgff HS.GRCh38.92.flattened.gff ~/Scratch/work/eli/project1/day_5/REH3-19/replicate_1/bam/REH3-19.replicate_1.sorted.bam Homo_sapiens.GRCh38.92.gtf ./test_qcand I get the following error Starting QoRTs v1.3.0 (Compiled Fri Oct 20 11:56:37 EDT 2017) Starting time: (Fri Jun 22 09:22:53 BST 2018) INPUT_COMMAND(QC) INPUT_ARG(infile)=/home/sejjctj/Scratch/work/eli/project1/day_5/REH3-19/replicate_1/bam/REH3-19.replicate_1.sorted.bam INPUT_ARG(gtffile)=Homo_sapiens.GRCh38.92.gtf INPUT_ARG(outdir)=./test_qc INPUT_ARG(runFunctions)=List(writeDEXSeq) INPUT_ARG(minMAPQ)=60 INPUT_ARG(maxReadLength)=Some(100) INPUT_ARG(flatgfffile)=Some(HS.GRCh38.92.flattened.gff) Created Log File: ./test_qc/QC.bpKPVLzhuVmb.log Warning: run-in-progress file "./test_qc/QC.QORTS_RUNNING" already exists. Is there another QoRTs job running? Starting QC [Time: 2018-06-22 09:22:53] [Mem usage: [42MB / 1009MB]] [Elapsed Time: 00:00:00.0000] QoRTs is Running in paired-end mode. QoRTs is Running in any-sorted mode. NOTE: Function "writeDEXSeq" requires function "JunctionCalcs". Adding "JunctionCalcs" to the active function list... Running functions: JunctionCalcs, writeDEXSeq Checking first 10000 reads. Checking SAM file for formatting errors... WARNING: BAM index file /home/sejjctj/Scratch/work/eli/project1/day_5/REH3-19/replicate_1/bam/REH3-19.replicate_1.sorted.bam.bai is older than BAM /home/sejjctj/Scratch/work/eli/project1/day_5/REH3-19/replicate_1/bam/REH3-19.replicate_1.sorted.bam NOTE: Read length is not consistent. In the first 10000 reads, read length varies from 20 to 76 (param maxReadLength=100) Note that using data that is hard-clipped prior to alignment is NOT recommended, because this makes it difficult (or impossible) to determine the sequencer read-cycle of each nucleotide base. This may obfuscate cycle-specific artifacts, trends, or errors, the detection of which is one of the primary purposes of QoRTs!In addition, hard clipping (whether before or after alignment) removes quality score data, and thus quality score metrics may be misleadingly optimistic. A MUCH preferable method of removing undesired sequence is to replace such sequence with N's, which preserves the quality score and the sequencer cycle information. Sorting Note: Reads are not sorted by name (This is OK). Sorting Note: Reads are sorted by position (This is OK). Done checking first 10000 reads. WARNINGS FOUND! SAMRecord Reader Generated. Read length: 100. [Time: 2018-06-22 09:22:58] [Mem usage: [245MB / 1272MB]] [Elapsed Time: 00:00:04.0938]