search

moiexpositoalonsolab / grenepipe

A flexible, scalable, and reproducible pipeline to automate variant calling from raw sequence reads, with lots of bells and whistles.
http://grene-net.org
GNU General Public License v3.0
94 stars 21 forks source link

Error in picard_collectmultiplemetrics #16

Closed Brent-Saylor-Canopy closed 2 years ago

Brent-Saylor-Canopy commented 3 years ago

Hi,

I have another error for you. As I mentioned in another thread I am attempting to update a run with new samples. So far most things look good, but I am getting an error in the picard_collectmultiplemetrics job.

This is the stdout output:

Traceback (most recent call last):
  File "/mnt/Data1/GBS_data/grenepipe_GBS_and_Skim_vs_91K/.snakemake/scripts/tmpwsgg3p9h.picard-collectmultiplemetrics.py", line 80, in <module>
    "(picard CollectMultipleMetrics "
  File "/opt/miniconda/envs/SNAKEMAKE_env/lib/python3.9/site-packages/snakemake/shell.py", line 263, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '(picard CollectMultipleMetrics I=mapped/G3N-1.sorted.bam O=qc/picard/G3N-1 R=/mnt/Work_dir/Canopy_Genomes/Competed_NRGene_Phased_Assembelies/V2/Assemblies/91K_A.fasta VALIDATION_STRINGENCY=LENIENT METRIC_ACCUMULATION_LEVEL=null METRIC_ACCUMULATION_LEVEL=SAMPLE  PROGRAM=CollectAlignmentSummaryMetrics PROGRAM=CollectBaseDistributionByCycle PROGRAM=CollectInsertSizeMetrics PROGRAM=CollectGcBiasMetrics PROGRAM=CollectQualityYieldMetrics PROGRAM=MeanQualityByCycle PROGRAM=QualityScoreDistribution )  2> logs/picard/multiple_metrics/G3N-1.log' returned non-zero exit status 1.
[Fri Oct 29 15:37:52 2021]
Error in rule picard_collectmultiplemetrics:
    jobid: 1035
    output: qc/picard/G3N-1.alignment_summary_metrics, qc/picard/G3N-1.base_distribution_by_cycle_metrics, qc/picard/G3N-1.base_distribution_by_cycle.pdf, qc/picard/G3N-1.gc_bias.detail_metrics, qc/picard/G3N-1.gc_bias.summary_metrics, qc/picard/G3N-1.gc_bias.pdf, qc/picard/G3N-1.insert_size_metrics, qc/picard/G3N-1.insert_size_histogram.pdf, qc/picard/G3N-1.quality_by_cycle_metrics, qc/picard/G3N-1.quality_by_cycle.pdf, qc/picard/G3N-1.quality_distribution_metrics, qc/picard/G3N-1.quality_distribution.pdf, qc/picard/G3N-1.quality_yield_metrics
    log: logs/picard/multiple_metrics/G3N-1.log (check log file(s) for error message)
    conda-env: /mnt/Data1/GBS_data/grenepipe_GBS_and_Skim_vs_91K/.snakemake/conda/31eebd0163ee2e1f9558bd20aa6edca5

RuleException:
CalledProcessError in line 192 of /mnt/Data1/GBS_data/grenepipe_GBS_and_Skim_vs_91K/rules/qc.smk:
Command 'source /opt/miniconda/envs/SNAKEMAKE_env/bin/activate '/mnt/Data1/GBS_data/grenepipe_GBS_and_Skim_vs_91K/.snakemake/conda/31eebd0163ee2e1f9558bd20aa6edca5'; python /mnt/Data1/GBS_data/grenepipe_GBS_and_Skim_vs_91K/.snakemake/scripts/tmpwsgg3p9h.picard-collectmultiplemetrics.py' returned non-zero exit status 1.
  File "/mnt/Data1/GBS_data/grenepipe_GBS_and_Skim_vs_91K/rules/qc.smk", line 192, in __rule_picard_collectmultiplemetrics
  File "/opt/miniconda/envs/SNAKEMAKE_env/lib/python3.9/concurrent/futures/thread.py", line 52, in run

And this is the picard log for for that file:

INFO    2021-10-29 15:31:53     CollectMultipleMetrics

********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
**********    CollectMultipleMetrics -I mapped/G3N-1.sorted.bam -O qc/picard/G3N-1 -R /mnt/Work_dir/Canopy_Genomes/Competed_NRGene_Phased_Assembelies/V2/Assemblies/91K_A.fasta -VALIDATION_STRINGENCY LENIENT -METRIC_ACCUMULATION_LEVEL null -METRIC_ACCUMULATION_LEVEL SAMPLE -PROGRAM CollectAlignmentSummaryMetrics -PROGRAM CollectBaseDistributionByCycle -PROGRAM CollectInsertSizeMetrics -PROGRAM CollectGcBiasMetrics -PROGRAM CollectQualityYieldMetrics -PROGRAM MeanQualityByCycle -PROGRAM QualityScoreDistribution
**********

15:31:53.825 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/mnt/Data1/GBS_data/grenepipe_GBS_and_Skim_vs_91K/.snakemake/conda/31eebd0163ee2e1f9558bd20aa6edca5/share/picard-2.23.0-0/picard.jar!/com/intel/gkl/native/libgkl_compression.so
[Fri Oct 29 15:31:53 EDT 2021] CollectMultipleMetrics INPUT=mapped/G3N-1.sorted.bam OUTPUT=qc/picard/G3N-1 METRIC_ACCUMULATION_LEVEL=[SAMPLE] PROGRAM=[CollectAlignmentSummaryMetrics, CollectBaseDistributionByCycle, CollectInsertSizeMetrics, MeanQualityByCycle, QualityScoreDistribution, CollectGcBiasMetrics, CollectQualityYieldMetrics] VALIDATION_STRINGENCY=LENIENT REFERENCE_SEQUENCE=/mnt/Work_dir/Canopy_Genomes/Competed_NRGene_Phased_Assembelies/V2/Assemblies/91K_A.fasta    ASSUME_SORTED=true STOP_AFTER=0 INCLUDE_UNPAIRED=false VERBOSITY=INFO QUIET=false COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[Fri Oct 29 15:31:53 EDT 2021] Executing as brent.saylor@corp.canopygrowth.com@ON7GENOM01 on Linux 5.10.0-1050-oem amd64; OpenJDK 64-Bit Server VM 11.0.8-internal+0-adhoc..src; Deflater: Intel; Inflater: Intel; Provider GCS is not available; Picard version: 2.23.0
WARNING 2021-10-29 15:31:53     CollectMultipleMetrics  The CollectBaseDistributionByCycle program does not support a metric accumulation level, but METRIC_ACCUMULATION_LEVEL was overridden in the command line. CollectBaseDistributionByCycle will be run against the entire input.
WARNING 2021-10-29 15:31:53     CollectMultipleMetrics  The MeanQualityByCycle program does not support a metric accumulation level, but METRIC_ACCUMULATION_LEVEL was overridden in the command line. MeanQualityByCycle will be run against the entire input.
WARNING 2021-10-29 15:31:53     CollectMultipleMetrics  The QualityScoreDistribution program does not support a metric accumulation level, but METRIC_ACCUMULATION_LEVEL was overridden in the command line. QualityScoreDistribution will be run against the entire input.
WARNING 2021-10-29 15:31:53     CollectMultipleMetrics  The CollectQualityYieldMetrics program does not support a metric accumulation level, but METRIC_ACCUMULATION_LEVEL was overridden in the command line. CollectQualityYieldMetrics will be run against the entire input.
INFO    2021-10-29 15:33:26     SinglePassSamProgram    Processed     1,000,000 records.  Elapsed time: 00:01:07s.  Time for last 1,000,000:   64s.  Last read position: 10A:82,872,277
INFO    2021-10-29 15:34:17     SinglePassSamProgram    Processed     2,000,000 records.  Elapsed time: 00:01:59s.  Time for last 1,000,000:   51s.  Last read position: 2A:6,536,813
INFO    2021-10-29 15:35:04     SinglePassSamProgram    Processed     3,000,000 records.  Elapsed time: 00:02:45s.  Time for last 1,000,000:   46s.  Last read position: 3A:7,153,965
INFO    2021-10-29 15:35:44     SinglePassSamProgram    Processed     4,000,000 records.  Elapsed time: 00:03:26s.  Time for last 1,000,000:   40s.  Last read position: 3A:81,414,252
INFO    2021-10-29 15:36:29     SinglePassSamProgram    Processed     5,000,000 records.  Elapsed time: 00:04:11s.  Time for last 1,000,000:   45s.  Last read position: 4A:72,038,205
INFO    2021-10-29 15:37:14     SinglePassSamProgram    Processed     6,000,000 records.  Elapsed time: 00:04:55s.  Time for last 1,000,000:   44s.  Last read position: 5A:77,537,799
[Fri Oct 29 15:37:52 EDT 2021] picard.analysis.CollectMultipleMetrics done. Elapsed time: 5.98 minutes.
Runtime.totalMemory()=536870912
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
        at picard.analysis.AlignmentSummaryMetricsCollector$GroupAlignmentSummaryMetricsPerUnitMetricCollector$IndividualAlignmentSummaryMetricsCollector.collectQualityData(AlignmentSummaryMetricsCollector.java:322)
        at picard.analysis.AlignmentSummaryMetricsCollector$GroupAlignmentSummaryMetricsPerUnitMetricCollector$IndividualAlignmentSummaryMetricsCollector.addRecord(AlignmentSummaryMetricsCollector.java:193)
        at picard.analysis.AlignmentSummaryMetricsCollector$GroupAlignmentSummaryMetricsPerUnitMetricCollector.acceptRecord(AlignmentSummaryMetricsCollector.java:120)
        at picard.analysis.AlignmentSummaryMetricsCollector$GroupAlignmentSummaryMetricsPerUnitMetricCollector.acceptRecord(AlignmentSummaryMetricsCollector.java:93)
        at picard.metrics.MultiLevelCollector$Distributor.acceptRecord(MultiLevelCollector.java:164)
        at picard.metrics.MultiLevelCollector.acceptRecord(MultiLevelCollector.java:315)
        at picard.analysis.AlignmentSummaryMetricsCollector.acceptRecord(AlignmentSummaryMetricsCollector.java:89)
        at picard.analysis.CollectAlignmentSummaryMetrics.acceptRead(CollectAlignmentSummaryMetrics.java:146)
        at picard.analysis.SinglePassSamProgram.makeItSo(SinglePassSamProgram.java:158)
        at picard.analysis.CollectMultipleMetrics.doWork(CollectMultipleMetrics.java:598)
        at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:305)
        at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:103)
        at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:113)

I've done a little reading into into the index out of bounds error and it seems to be a problem with the bam file, but I tried rerunning the alignment and it didn't solve the problem.

lczech commented 3 years ago

Hi Brent,

that is near impossible to debug for me from the distance, and it might be in your data, and not in the pipeline. Hard to tell for now, and hence this will require some testing on your end. Here are some things you can try:

Hope that helps, and let me know what you find :-) Lucas

lczech commented 2 years ago

Hi @Brent-Saylor-Canopy, any news on this?

Brent-Saylor-Canopy commented 2 years ago

Hi, I've just turned it off so far

lczech commented 2 years ago

Haha I see, makes sense. Well, as I unfortunately cannot fix this without additional information, I'm going to close the issue for now, but feel free to re-open if you need this fixed at some point :-)