Error in picard_collectmultiplemetrics

[Brent-Saylor-Canopy]

Hi,

I have another error for you. As I mentioned in another thread I am attempting to update a run with new samples. So far most things look good, but I am getting an error in the picard_collectmultiplemetrics job.

This is the stdout output:

Traceback (most recent call last):
  File "/mnt/Data1/GBS_data/grenepipe_GBS_and_Skim_vs_91K/.snakemake/scripts/tmpwsgg3p9h.picard-collectmultiplemetrics.py", line 80, in <module>
    "(picard CollectMultipleMetrics "
  File "/opt/miniconda/envs/SNAKEMAKE_env/lib/python3.9/site-packages/snakemake/shell.py", line 263, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '(picard CollectMultipleMetrics I=mapped/G3N-1.sorted.bam O=qc/picard/G3N-1 R=/mnt/Work_dir/Canopy_Genomes/Competed_NRGene_Phased_Assembelies/V2/Assemblies/91K_A.fasta VALIDATION_STRINGENCY=LENIENT METRIC_ACCUMULATION_LEVEL=null METRIC_ACCUMULATION_LEVEL=SAMPLE  PROGRAM=CollectAlignmentSummaryMetrics PROGRAM=CollectBaseDistributionByCycle PROGRAM=CollectInsertSizeMetrics PROGRAM=CollectGcBiasMetrics PROGRAM=CollectQualityYieldMetrics PROGRAM=MeanQualityByCycle PROGRAM=QualityScoreDistribution )  2> logs/picard/multiple_metrics/G3N-1.log' returned non-zero exit status 1.
[Fri Oct 29 15:37:52 2021]
Error in rule picard_collectmultiplemetrics:
    jobid: 1035
    output: qc/picard/G3N-1.alignment_summary_metrics, qc/picard/G3N-1.base_distribution_by_cycle_metrics, qc/picard/G3N-1.base_distribution_by_cycle.pdf, qc/picard/G3N-1.gc_bias.detail_metrics, qc/picard/G3N-1.gc_bias.summary_metrics, qc/picard/G3N-1.gc_bias.pdf, qc/picard/G3N-1.insert_size_metrics, qc/picard/G3N-1.insert_size_histogram.pdf, qc/picard/G3N-1.quality_by_cycle_metrics, qc/picard/G3N-1.quality_by_cycle.pdf, qc/picard/G3N-1.quality_distribution_metrics, qc/picard/G3N-1.quality_distribution.pdf, qc/picard/G3N-1.quality_yield_metrics
    log: logs/picard/multiple_metrics/G3N-1.log (check log file(s) for error message)
    conda-env: /mnt/Data1/GBS_data/grenepipe_GBS_and_Skim_vs_91K/.snakemake/conda/31eebd0163ee2e1f9558bd20aa6edca5

RuleException:
CalledProcessError in line 192 of /mnt/Data1/GBS_data/grenepipe_GBS_and_Skim_vs_91K/rules/qc.smk:
Command 'source /opt/miniconda/envs/SNAKEMAKE_env/bin/activate '/mnt/Data1/GBS_data/grenepipe_GBS_and_Skim_vs_91K/.snakemake/conda/31eebd0163ee2e1f9558bd20aa6edca5'; python /mnt/Data1/GBS_data/grenepipe_GBS_and_Skim_vs_91K/.snakemake/scripts/tmpwsgg3p9h.picard-collectmultiplemetrics.py' returned non-zero exit status 1.
  File "/mnt/Data1/GBS_data/grenepipe_GBS_and_Skim_vs_91K/rules/qc.smk", line 192, in __rule_picard_collectmultiplemetrics
  File "/opt/miniconda/envs/SNAKEMAKE_env/lib/python3.9/concurrent/futures/thread.py", line 52, in run

And this is the picard log for for that file:

INFO    2021-10-29 15:31:53     CollectMultipleMetrics

********** NOTE: Picard's command line syntax is changing.
**********
********** For more information, please see:
********** https://github.com/broadinstitute/picard/wiki/Command-Line-Syntax-Transition-For-Users-(Pre-Transition)
**********
********** The command line looks like this in the new syntax:
**********
**********    CollectMultipleMetrics -I mapped/G3N-1.sorted.bam -O qc/picard/G3N-1 -R /mnt/Work_dir/Canopy_Genomes/Competed_NRGene_Phased_Assembelies/V2/Assemblies/91K_A.fasta -VALIDATION_STRINGENCY LENIENT -METRIC_ACCUMULATION_LEVEL null -METRIC_ACCUMULATION_LEVEL SAMPLE -PROGRAM CollectAlignmentSummaryMetrics -PROGRAM CollectBaseDistributionByCycle -PROGRAM CollectInsertSizeMetrics -PROGRAM CollectGcBiasMetrics -PROGRAM CollectQualityYieldMetrics -PROGRAM MeanQualityByCycle -PROGRAM QualityScoreDistribution
**********

15:31:53.825 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/mnt/Data1/GBS_data/grenepipe_GBS_and_Skim_vs_91K/.snakemake/conda/31eebd0163ee2e1f9558bd20aa6edca5/share/picard-2.23.0-0/picard.jar!/com/intel/gkl/native/libgkl_compression.so
[Fri Oct 29 15:31:53 EDT 2021] CollectMultipleMetrics INPUT=mapped/G3N-1.sorted.bam OUTPUT=qc/picard/G3N-1 METRIC_ACCUMULATION_LEVEL=[SAMPLE] PROGRAM=[CollectAlignmentSummaryMetrics, CollectBaseDistributionByCycle, CollectInsertSizeMetrics, MeanQualityByCycle, QualityScoreDistribution, CollectGcBiasMetrics, CollectQualityYieldMetrics] VALIDATION_STRINGENCY=LENIENT REFERENCE_SEQUENCE=/mnt/Work_dir/Canopy_Genomes/Competed_NRGene_Phased_Assembelies/V2/Assemblies/91K_A.fasta    ASSUME_SORTED=true STOP_AFTER=0 INCLUDE_UNPAIRED=false VERBOSITY=INFO QUIET=false COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
[Fri Oct 29 15:31:53 EDT 2021] Executing as brent.saylor@corp.canopygrowth.com@ON7GENOM01 on Linux 5.10.0-1050-oem amd64; OpenJDK 64-Bit Server VM 11.0.8-internal+0-adhoc..src; Deflater: Intel; Inflater: Intel; Provider GCS is not available; Picard version: 2.23.0
WARNING 2021-10-29 15:31:53     CollectMultipleMetrics  The CollectBaseDistributionByCycle program does not support a metric accumulation level, but METRIC_ACCUMULATION_LEVEL was overridden in the command line. CollectBaseDistributionByCycle will be run against the entire input.
WARNING 2021-10-29 15:31:53     CollectMultipleMetrics  The MeanQualityByCycle program does not support a metric accumulation level, but METRIC_ACCUMULATION_LEVEL was overridden in the command line. MeanQualityByCycle will be run against the entire input.
WARNING 2021-10-29 15:31:53     CollectMultipleMetrics  The QualityScoreDistribution program does not support a metric accumulation level, but METRIC_ACCUMULATION_LEVEL was overridden in the command line. QualityScoreDistribution will be run against the entire input.
WARNING 2021-10-29 15:31:53     CollectMultipleMetrics  The CollectQualityYieldMetrics program does not support a metric accumulation level, but METRIC_ACCUMULATION_LEVEL was overridden in the command line. CollectQualityYieldMetrics will be run against the entire input.
INFO    2021-10-29 15:33:26     SinglePassSamProgram    Processed     1,000,000 records.  Elapsed time: 00:01:07s.  Time for last 1,000,000:   64s.  Last read position: 10A:82,872,277
INFO    2021-10-29 15:34:17     SinglePassSamProgram    Processed     2,000,000 records.  Elapsed time: 00:01:59s.  Time for last 1,000,000:   51s.  Last read position: 2A:6,536,813
INFO    2021-10-29 15:35:04     SinglePassSamProgram    Processed     3,000,000 records.  Elapsed time: 00:02:45s.  Time for last 1,000,000:   46s.  Last read position: 3A:7,153,965
INFO    2021-10-29 15:35:44     SinglePassSamProgram    Processed     4,000,000 records.  Elapsed time: 00:03:26s.  Time for last 1,000,000:   40s.  Last read position: 3A:81,414,252
INFO    2021-10-29 15:36:29     SinglePassSamProgram    Processed     5,000,000 records.  Elapsed time: 00:04:11s.  Time for last 1,000,000:   45s.  Last read position: 4A:72,038,205
INFO    2021-10-29 15:37:14     SinglePassSamProgram    Processed     6,000,000 records.  Elapsed time: 00:04:55s.  Time for last 1,000,000:   44s.  Last read position: 5A:77,537,799
[Fri Oct 29 15:37:52 EDT 2021] picard.analysis.CollectMultipleMetrics done. Elapsed time: 5.98 minutes.
Runtime.totalMemory()=536870912
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
        at picard.analysis.AlignmentSummaryMetricsCollector$GroupAlignmentSummaryMetricsPerUnitMetricCollector$IndividualAlignmentSummaryMetricsCollector.collectQualityData(AlignmentSummaryMetricsCollector.java:322)
        at picard.analysis.AlignmentSummaryMetricsCollector$GroupAlignmentSummaryMetricsPerUnitMetricCollector$IndividualAlignmentSummaryMetricsCollector.addRecord(AlignmentSummaryMetricsCollector.java:193)
        at picard.analysis.AlignmentSummaryMetricsCollector$GroupAlignmentSummaryMetricsPerUnitMetricCollector.acceptRecord(AlignmentSummaryMetricsCollector.java:120)
        at picard.analysis.AlignmentSummaryMetricsCollector$GroupAlignmentSummaryMetricsPerUnitMetricCollector.acceptRecord(AlignmentSummaryMetricsCollector.java:93)
        at picard.metrics.MultiLevelCollector$Distributor.acceptRecord(MultiLevelCollector.java:164)
        at picard.metrics.MultiLevelCollector.acceptRecord(MultiLevelCollector.java:315)
        at picard.analysis.AlignmentSummaryMetricsCollector.acceptRecord(AlignmentSummaryMetricsCollector.java:89)
        at picard.analysis.CollectAlignmentSummaryMetrics.acceptRead(CollectAlignmentSummaryMetrics.java:146)
        at picard.analysis.SinglePassSamProgram.makeItSo(SinglePassSamProgram.java:158)
        at picard.analysis.CollectMultipleMetrics.doWork(CollectMultipleMetrics.java:598)
        at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:305)
        at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:103)
        at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:113)

I've done a little reading into into the index out of bounds error and it seems to be a problem with the bam file, but I tried rerunning the alignment and it didn't solve the problem.

[lczech]

Hi Brent,

that is near impossible to debug for me from the distance, and it might be in your data, and not in the pipeline. Hard to tell for now, and hence this will require some testing on your end. Here are some things you can try:

• You could check if you can run picard on these input files manually, and see if that works.
• You can try to run some other tools on these bam files (just some command from bamtools for example), to see if they find an error in the files as well.
• You can try to convert the file to sam an back to bam, or something like that, and see if this solves the issue.
• Lasty, you can also deactivate the rule, if you don't need these metrics. Simply comment out these lines.

Hope that helps, and let me know what you find :-) Lucas

[lczech]

Hi @Brent-Saylor-Canopy, any news on this?

[Brent-Saylor-Canopy]

Hi, I've just turned it off so far

[lczech]

Haha I see, makes sense. Well, as I unfortunately cannot fix this without additional information, I'm going to close the issue for now, but feel free to re-open if you need this fixed at some point :-)