Workflow failed: "Bad output 'align_mito...'

[santiago-es]

Describe the bug

My workflow is failing with message "Bad output, 'align_mito.samstat_qc': Bad array access...' when attempting to run the example JSON file with atac.wdl

OS/Platform

OS/Platform: Ubuntu 20.04
conda version: 4.9.0
backend: local
using encode-atac-seq Conda environment

### Caper configuration file

backend=local

local-hash-strat=path+modtime

local-loc-dir=/media/santiago/SESUbuntu/ATACtmp/

cromwell=/home/santiago/.caper/cromwell_jar/cromwell-52.jar womtool=/home/santiago/.caper/womtool_jar/womtool-52.jar

Input JSON file

It's the example JSON...

{ "atac.pipeline_type" : "atac", "atac.genome_tsv" : "https://storage.googleapis.com/encode-pipeline-genome-data/genome_tsv/v3/hg38.tsv", "atac.fastqs_rep1_R1" : [ "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep1/pair1/ENCFF341MYG.subsampled.400.fastq.gz", "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep1/pair1/ENCFF106QGY.subsampled.400.fastq.gz" ], "atac.fastqs_rep1_R2" : [ "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep1/pair2/ENCFF248EJF.subsampled.400.fastq.gz", "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep1/pair2/ENCFF368TYI.subsampled.400.fastq.gz" ], "atac.fastqs_rep2_R1" : [ "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep2/pair1/ENCFF641SFZ.subsampled.400.fastq.gz", "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep2/pair1/ENCFF751XTV.subsampled.400.fastq.gz", "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep2/pair1/ENCFF927LSG.subsampled.400.fastq.gz", "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep2/pair1/ENCFF859BDM.subsampled.400.fastq.gz", "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep2/pair1/ENCFF193RRC.subsampled.400.fastq.gz", "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep2/pair1/ENCFF366DFI.subsampled.400.fastq.gz" ], "atac.fastqs_rep2_R2" : [ "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep2/pair2/ENCFF031ARQ.subsampled.400.fastq.gz", "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep2/pair2/ENCFF590SYZ.subsampled.400.fastq.gz", "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep2/pair2/ENCFF734PEQ.subsampled.400.fastq.gz", "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep2/pair2/ENCFF007USV.subsampled.400.fastq.gz", "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep2/pair2/ENCFF886FSC.subsampled.400.fastq.gz", "https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ/fastq_subsampled/rep2/pair2/ENCFF573UXK.subsampled.400.fastq.gz" ], "atac.paired_end" : true, "atac.auto_detect_adapter" : true, "atac.enable_xcor" : true, "atac.title" : "ENCSR356KRQ (subsampled 1/400)", "atac.description" : "ATAC-seq on primary keratinocytes in day 0.0 of differentiation" }

Here's my cromwell.out file:

2020-11-08 10:48:41,755 INFO - Running with database db.url = jdbc:hsqldb:mem:d7dcf59d-fa00-47b5-b2a6-37bbc7ed43c9;shutdown=false;hsqldb.tx=mvcc 2020-11-08 10:48:46,632 INFO - Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000 2020-11-08 10:48:46,641 INFO - [RenameWorkflowOptionsInMetadata] 100% 2020-11-08 10:48:46,693 INFO - Running with database db.url = jdbc:hsqldb:mem:c7c564e7-0f2e-44e6-82f2-a0d5f7d089a9;shutdown=false;hsqldb.tx=mvcc 2020-11-08 10:48:46,948 INFO - Slf4jLogger started 2020-11-08 10:48:47,047 cromwell-system-akka.dispatchers.engine-dispatcher-6 INFO - Workflow heartbeat configuration: { "cromwellId" : "cromid-043da09", "heartbeatInterval" : "2 minutes", "ttl" : "10 minutes", "failureShutdownDuration" : "5 minutes", "writeBatchSize" : 10000, "writeThreshold" : 10000 } 2020-11-08 10:48:47,079 cromwell-system-akka.dispatchers.service-dispatcher-10 INFO - Metadata summary refreshing every 1 second. 2020-11-08 10:48:47,094 cromwell-system-akka.dispatchers.service-dispatcher-11 INFO - WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds. 2020-11-08 10:48:47,095 cromwell-system-akka.actor.default-dispatcher-4 INFO - KvWriteActor configured to flush with batch size 200 and process rate 5 seconds. 2020-11-08 10:48:47,104 cromwell-system-akka.dispatchers.engine-dispatcher-40 INFO - CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds. 2020-11-08 10:48:47,105 WARN - 'docker.hash-lookup.gcr-api-queries-per-100-seconds' is being deprecated, use 'docker.hash-lookup.gcr.throttle' instead (see reference.conf) 2020-11-08 10:48:47,326 cromwell-system-akka.dispatchers.engine-dispatcher-40 INFO - JobExecutionTokenDispenser - Distribution rate: 1 per 2 seconds. 2020-11-08 10:48:47,341 cromwell-system-akka.dispatchers.engine-dispatcher-6 INFO - SingleWorkflowRunnerActor: Version 52 2020-11-08 10:48:47,344 cromwell-system-akka.dispatchers.engine-dispatcher-6 INFO - SingleWorkflowRunnerActor: Submitting workflow 2020-11-08 10:48:47,374 cromwell-system-akka.dispatchers.api-dispatcher-48 INFO - Unspecified type (Unspecified version) workflow 89b3e421-646a-4664-a06f-63c86e217676 submitted 2020-11-08 10:48:47,387 cromwell-system-akka.dispatchers.engine-dispatcher-46 INFO - SingleWorkflowRunnerActor: Workflow submitted UUID(89b3e421-646a-4664-a06f-63c86e217676) 2020-11-08 10:48:47,392 cromwell-system-akka.dispatchers.engine-dispatcher-31 INFO - 1 new workflows fetched by cromid-043da09: 89b3e421-646a-4664-a06f-63c86e217676 2020-11-08 10:48:47,399 cromwell-system-akka.dispatchers.engine-dispatcher-40 INFO - WorkflowManagerActor Starting workflow UUID(89b3e421-646a-4664-a06f-63c86e217676) 2020-11-08 10:48:47,405 cromwell-system-akka.dispatchers.engine-dispatcher-40 INFO - WorkflowManagerActor Successfully started WorkflowActor-89b3e421-646a-4664-a06f-63c86e217676 2020-11-08 10:48:47,405 cromwell-system-akka.dispatchers.engine-dispatcher-40 INFO - Retrieved 1 workflows from the WorkflowStoreActor 2020-11-08 10:48:47,419 cromwell-system-akka.dispatchers.engine-dispatcher-40 INFO - WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes. 2020-11-08 10:48:47,482 cromwell-system-akka.dispatchers.engine-dispatcher-31 INFO - MaterializeWorkflowDescriptorActor [UUID(89b3e421)]: Parsing workflow as WDL 1.0 2020-11-08 10:48:48,831 cromwell-system-akka.dispatchers.engine-dispatcher-31 INFO - MaterializeWorkflowDescriptorActor [UUID(89b3e421)]: Call-to-Backend assignments: atac.reproducibility_overlap -> Local, atac.macs2_signal_track_pooled -> Local, atac.bam2ta -> Local, atac.align -> Local, atac.jsd -> Local, atac.pool_ta_pr1 -> Local, atac.spr -> Local, atac.idr_pr -> Local, atac.annot_enrich -> Local, atac.gc_bias -> Local, atac.call_peak_ppr2 -> Local, atac.call_peak_pr1 -> Local, atac.overlap_pr -> Local, atac.reproducibility_idr -> Local, atac.tss_enrich -> Local, atac.preseq -> Local, atac.call_peak_pr2 -> Local, atac.idr_ppr -> Local, atac.read_genome_tsv -> Local, atac.pool_ta_pr2 -> Local, atac.qc_report -> Local, atac.macs2_signal_track -> Local, atac.idr -> Local, atac.overlap -> Local, atac.pool_ta -> Local, atac.align_mito -> Local, atac.fraglen_stat_pe -> Local, atac.call_peak_pooled -> Local, atac.xcor -> Local, atac.bam2ta_no_dedup -> Local, atac.count_signal_track -> Local, atac.error_input_data -> Local, atac.pool_blacklist -> Local, atac.filter -> Local, atac.filter_no_dedup -> Local, atac.count_signal_track_pooled -> Local, atac.overlap_ppr -> Local, atac.compare_signal_to_roadmap -> Local, atac.call_peak -> Local, atac.frac_mito -> Local, atac.call_peak_ppr1 -> Local 2020-11-08 10:48:48,954 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,955 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,955 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,956 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,956 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,956 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,956 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,956 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,957 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,957 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,957 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,957 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,957 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,958 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,958 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,958 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,958 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,958 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,958 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,958 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,959 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,959 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,959 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,959 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,959 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,959 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,959 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,959 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,959 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,960 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,960 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,960 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,960 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,960 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,960 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,961 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,961 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,961 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,961 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,961 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:48,961 cromwell-system-akka.dispatchers.backend-dispatcher-59 WARN - Local [UUID(89b3e421)]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. 2020-11-08 10:48:52,342 cromwell-system-akka.dispatchers.engine-dispatcher-90 INFO - Not triggering log of token queue status. Effective log interval = None 2020-11-08 10:48:53,170 cromwell-system-akka.dispatchers.engine-dispatcher-90 INFO - WorkflowExecutionActor-89b3e421-646a-4664-a06f-63c86e217676 [UUID(89b3e421)]: Starting atac.read_genome_tsv 2020-11-08 10:48:53,350 cromwell-system-akka.dispatchers.engine-dispatcher-90 INFO - Assigned new job execution tokens to the following groups: 89b3e421: 1 2020-11-08 10:48:53,434 cromwell-system-akka.dispatchers.engine-dispatcher-84 INFO - 89b3e421-646a-4664-a06f-63c86e217676-EngineJobExecutionActor-atac.read_genome_tsv:NA:1 [UUID(89b3e421)]: Could not copy a suitable cache hit for 89b3e421:atac.read_genome_tsv:-1:1. No copy attempts were made. 2020-11-08 10:48:53,450 cromwell-system-akka.dispatchers.backend-dispatcher-104 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.read_genome_tsv:NA:1]: Unrecognized runtime attribute keys: disks, cpu, time, memory 2020-11-08 10:48:53,481 cromwell-system-akka.dispatchers.backend-dispatcher-104 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.read_genome_tsv:NA:1]: `echo "$(basename /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-read_genome_tsv/inputs/18759998/hg38.local.tsv)" > genome_name

create empty files for all entries

touch ref_fa bowtie2_idx_tar chrsz gensz blacklist blacklist2 touch ref_mito_fa touch bowtie2_mito_idx_tar touch tss tss_enrich # for backward compatibility touch dnase prom enh reg2map reg2map_bed roadmap_meta touch mito_chr_name touch regex_bfilt_peak_chr_name

python <<CODE import os with open('/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-read_genome_tsv/inputs/18759998/hg38.local.tsv','r') as fp: for line in fp: arr = line.strip('\n').split('\t') if arr: key, val = arr with open(key,'w') as fp2: fp2.write(val) CODE 2020-11-08 10:48:53,631 cromwell-system-akka.dispatchers.backend-dispatcher-104 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.read_genome_tsv:NA:1]: executing: if [ -z \"$SINGULARITY_BINDPATH\" ]; then export SINGULARITY_BINDPATH=; fi; \ if [ -z \"$SINGULARITY_CACHEDIR\" ]; then export SINGULARITY_CACHEDIR=; fi; /bin/bash /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-read_genome_tsv/execution/script 2020-11-08 10:48:57,129 cromwell-system-akka.dispatchers.backend-dispatcher-107 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.read_genome_tsv:NA:1]: job id: 564736 2020-11-08 10:48:57,132 cromwell-system-akka.dispatchers.backend-dispatcher-104 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.read_genome_tsv:NA:1]: Status change from - to WaitingForReturnCode 2020-11-08 10:49:06,031 cromwell-system-akka.dispatchers.backend-dispatcher-106 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.read_genome_tsv:NA:1]: Status change from WaitingForReturnCode to Done 2020-11-08 10:49:11,563 cromwell-system-akka.dispatchers.engine-dispatcher-31 INFO - WorkflowExecutionActor-89b3e421-646a-4664-a06f-63c86e217676 [UUID(89b3e421)]: Starting atac.align (2 shards) 2020-11-08 10:49:13,341 cromwell-system-akka.dispatchers.engine-dispatcher-79 INFO - Assigned new job execution tokens to the following groups: 89b3e421: 1 2020-11-08 10:49:13,365 cromwell-system-akka.dispatchers.engine-dispatcher-31 INFO - 89b3e421-646a-4664-a06f-63c86e217676-EngineJobExecutionActor-atac.align:1:1 [UUID(89b3e421)]: Could not copy a suitable cache hit for 89b3e421:atac.align:1:1. No copy attempts were made. 2020-11-08 10:49:13,367 cromwell-system-akka.dispatchers.backend-dispatcher-109 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align:1:1]: Unrecognized runtime attribute keys: preemptible, disks, cpu, time, memory 2020-11-08 10:49:13,391 cromwell-system-akka.dispatchers.backend-dispatcher-109 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align:1:1]:set -e

check if pipeline dependencies can be found

if [[ -z "$(which encode_task_trim_adapter.py 2> /dev/null || true)" ]] then echo -e "\n* Error: pipeline dependencies not found." 1>&2 echo 'Conda users: Did you activate Conda environment (conda activate encode-atac-seq-pipeline)?' 1>&2 echo ' Or did you install Conda and environment correctly (bash scripts/install_conda_env.sh)?' 1>&2 echo 'GCP/AWS/Docker users: Did you add --docker flag to Caper command line arg?' 1>&2 echo 'Singularity users: Did you add --singularity flag to Caper command line arg?' 1>&2 echo -e "\n" 1>&2 exit 3 fi

trim adapter

python3 $(which encode_task_trim_adapter.py) \ /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align/shard-1/execution/write_tsv_47ab74552aefdfec739465609cf0a312.tmp \ \ --adapters /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align/shard-1/execution/write_tsv_d41d8cd98f00b204e9800998ecf8427e.tmp \ --paired-end \ --auto-detect-adapter \ --cutadapt-param ' -e 0.1 -m 5' \ --nth 6

align on trimmed/merged fastqs

if [ 'bowtie2' == 'bowtie2' ]; then python3 $(which encode_task_bowtie2.py) \ /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align/shard-1/inputs/-165196233/ENCFF110MCL.tar.gz \ R1/.fastq.gz \ R2/.fastq.gz \ --paired-end \ --multimapping 4 \ --mem-gb 4.004375195540488 \ --nth 6 fi

python3 $(which encode_task_post_align.py) \ R1/.fastq.gz $(ls .bam) \ --mito-chr-name chrM \ --chrsz /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align/shard-1/inputs/-1634915880/GRCh38_EBV.chrom.sizes.tsv \ --mem-gb 4.004375195540488 \ --nth 6 rm -rf R1 R2 2020-11-08 10:49:13,399 cromwell-system-akka.dispatchers.backend-dispatcher-109 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align:1:1]: executing: if [ -z \"$SINGULARITY_BINDPATH\" ]; then export SINGULARITY_BINDPATH=; fi; \ if [ -z \"$SINGULARITY_CACHEDIR\" ]; then export SINGULARITY_CACHEDIR=; fi; /bin/bash /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align/shard-1/execution/script 2020-11-08 10:49:14,623 cromwell-system-akka.dispatchers.engine-dispatcher-81 INFO - WorkflowExecutionActor-89b3e421-646a-4664-a06f-63c86e217676 [UUID(89b3e421)]: Starting atac.align_mito (2 shards) 2020-11-08 10:49:15,341 cromwell-system-akka.dispatchers.engine-dispatcher-79 INFO - Assigned new job execution tokens to the following groups: 89b3e421: 1 2020-11-08 10:49:15,351 cromwell-system-akka.dispatchers.engine-dispatcher-79 INFO - 89b3e421-646a-4664-a06f-63c86e217676-EngineJobExecutionActor-atac.align:0:1 [UUID(89b3e421)]: Could not copy a suitable cache hit for 89b3e421:atac.align:0:1. No copy attempts were made. 2020-11-08 10:49:15,352 cromwell-system-akka.dispatchers.backend-dispatcher-110 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align:0:1]: Unrecognized runtime attribute keys: preemptible, disks, cpu, time, memory 2020-11-08 10:49:15,362 cromwell-system-akka.dispatchers.backend-dispatcher-110 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align:0:1]:set -e

check if pipeline dependencies can be found

if [[ -z "$(which encode_task_trim_adapter.py 2> /dev/null || true)" ]] then echo -e "\n* Error: pipeline dependencies not found." 1>&2 echo 'Conda users: Did you activate Conda environment (conda activate encode-atac-seq-pipeline)?' 1>&2 echo ' Or did you install Conda and environment correctly (bash scripts/install_conda_env.sh)?' 1>&2 echo 'GCP/AWS/Docker users: Did you add --docker flag to Caper command line arg?' 1>&2 echo 'Singularity users: Did you add --singularity flag to Caper command line arg?' 1>&2 echo -e "\n" 1>&2 exit 3 fi

trim adapter

python3 $(which encode_task_trim_adapter.py) \ /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align/shard-0/execution/write_tsv_45d231d328d782ef793898080f51c1b2.tmp \ \ --adapters /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align/shard-0/execution/write_tsv_d41d8cd98f00b204e9800998ecf8427e.tmp \ --paired-end \ --auto-detect-adapter \ --cutadapt-param ' -e 0.1 -m 5' \ --nth 6

align on trimmed/merged fastqs

if [ 'bowtie2' == 'bowtie2' ]; then python3 $(which encode_task_bowtie2.py) \ /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align/shard-0/inputs/-165196233/ENCFF110MCL.tar.gz \ R1/.fastq.gz \ R2/.fastq.gz \ --paired-end \ --multimapping 4 \ --mem-gb 4.00359172642231 \ --nth 6 fi

python3 $(which encode_task_post_align.py) \ R1/.fastq.gz $(ls .bam) \ --mito-chr-name chrM \ --chrsz /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align/shard-0/inputs/-1634915880/GRCh38_EBV.chrom.sizes.tsv \ --mem-gb 4.00359172642231 \ --nth 6 rm -rf R1 R2 2020-11-08 10:49:15,366 cromwell-system-akka.dispatchers.backend-dispatcher-110 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align:0:1]: executing: if [ -z \"$SINGULARITY_BINDPATH\" ]; then export SINGULARITY_BINDPATH=; fi; \ if [ -z \"$SINGULARITY_CACHEDIR\" ]; then export SINGULARITY_CACHEDIR=; fi; /bin/bash /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align/shard-0/execution/script 2020-11-08 10:49:17,113 cromwell-system-akka.dispatchers.backend-dispatcher-110 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align:0:1]: job id: 564783 2020-11-08 10:49:17,113 cromwell-system-akka.dispatchers.backend-dispatcher-109 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align:1:1]: job id: 564768 2020-11-08 10:49:17,114 cromwell-system-akka.dispatchers.backend-dispatcher-111 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align:0:1]: Status change from - to WaitingForReturnCode 2020-11-08 10:49:17,114 cromwell-system-akka.dispatchers.backend-dispatcher-112 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align:1:1]: Status change from - to WaitingForReturnCode 2020-11-08 10:49:17,341 cromwell-system-akka.dispatchers.engine-dispatcher-68 INFO - Assigned new job execution tokens to the following groups: 89b3e421: 1 2020-11-08 10:49:17,357 cromwell-system-akka.dispatchers.engine-dispatcher-68 INFO - 89b3e421-646a-4664-a06f-63c86e217676-EngineJobExecutionActor-atac.align_mito:0:1 [UUID(89b3e421)]: Could not copy a suitable cache hit for 89b3e421:atac.align_mito:0:1. No copy attempts were made. 2020-11-08 10:49:17,358 cromwell-system-akka.dispatchers.backend-dispatcher-111 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align_mito:0:1]: Unrecognized runtime attribute keys: preemptible, disks, cpu, time, memory 2020-11-08 10:49:17,366 cromwell-system-akka.dispatchers.backend-dispatcher-111 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align_mito:0:1]:set -e

check if pipeline dependencies can be found

if [[ -z "$(which encode_task_trim_adapter.py 2> /dev/null || true)" ]] then echo -e "\n* Error: pipeline dependencies not found." 1>&2 echo 'Conda users: Did you activate Conda environment (conda activate encode-atac-seq-pipeline)?' 1>&2 echo ' Or did you install Conda and environment correctly (bash scripts/install_conda_env.sh)?' 1>&2 echo 'GCP/AWS/Docker users: Did you add --docker flag to Caper command line arg?' 1>&2 echo 'Singularity users: Did you add --singularity flag to Caper command line arg?' 1>&2 echo -e "\n" 1>&2 exit 3 fi

trim adapter

python3 $(which encode_task_trim_adapter.py) \ /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align_mito/shard-0/execution/write_tsv_b22f294e2949523045e8d240bcb6ce31.tmp \ \ --adapters /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align_mito/shard-0/execution/write_tsv_d41d8cd98f00b204e9800998ecf8427e.tmp \ --paired-end \ --auto-detect-adapter \ --cutadapt-param ' -e 0.1 -m 5' \ --nth 6

align on trimmed/merged fastqs

if [ 'bowtie2' == 'bowtie2' ]; then python3 $(which encode_task_bowtie2.py) \ /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align_mito/shard-0/inputs/-347230842/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bowtie2_index.tar.gz \ R1/.fastq.gz \ R2/.fastq.gz \ --paired-end \ --multimapping 4 \ --mem-gb 4.00359172642231 \ --nth 6 fi

python3 $(which encode_task_post_align.py) \ R1/.fastq.gz $(ls .bam) \ --mito-chr-name chrM \ --chrsz /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align_mito/shard-0/inputs/-1634915880/GRCh38_EBV.chrom.sizes.tsv \ --mem-gb 4.00359172642231 \ --nth 6 rm -rf R1 R2 2020-11-08 10:49:17,369 cromwell-system-akka.dispatchers.backend-dispatcher-111 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align_mito:0:1]: executing: if [ -z \"$SINGULARITY_BINDPATH\" ]; then export SINGULARITY_BINDPATH=; fi; \ if [ -z \"$SINGULARITY_CACHEDIR\" ]; then export SINGULARITY_CACHEDIR=; fi; /bin/bash /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align_mito/shard-0/execution/script 2020-11-08 10:49:19,341 cromwell-system-akka.dispatchers.engine-dispatcher-31 INFO - Assigned new job execution tokens to the following groups: 89b3e421: 1 2020-11-08 10:49:19,358 cromwell-system-akka.dispatchers.engine-dispatcher-81 INFO - 89b3e421-646a-4664-a06f-63c86e217676-EngineJobExecutionActor-atac.align_mito:1:1 [UUID(89b3e421)]: Could not copy a suitable cache hit for 89b3e421:atac.align_mito:1:1. No copy attempts were made. 2020-11-08 10:49:19,358 cromwell-system-akka.dispatchers.backend-dispatcher-111 WARN - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align_mito:1:1]: Unrecognized runtime attribute keys: preemptible, disks, cpu, time, memory 2020-11-08 10:49:19,370 cromwell-system-akka.dispatchers.backend-dispatcher-111 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align_mito:1:1]:set -e

check if pipeline dependencies can be found

if [[ -z "$(which encode_task_trim_adapter.py 2> /dev/null || true)" ]] then echo -e "\n* Error: pipeline dependencies not found." 1>&2 echo 'Conda users: Did you activate Conda environment (conda activate encode-atac-seq-pipeline)?' 1>&2 echo ' Or did you install Conda and environment correctly (bash scripts/install_conda_env.sh)?' 1>&2 echo 'GCP/AWS/Docker users: Did you add --docker flag to Caper command line arg?' 1>&2 echo 'Singularity users: Did you add --singularity flag to Caper command line arg?' 1>&2 echo -e "\n" 1>&2 exit 3 fi

trim adapter

python3 $(which encode_task_trim_adapter.py) \ /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align_mito/shard-1/execution/write_tsv_35941950c46da31b7edc49e3c26c7296.tmp \ \ --adapters /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align_mito/shard-1/execution/write_tsv_d41d8cd98f00b204e9800998ecf8427e.tmp \ --paired-end \ --auto-detect-adapter \ --cutadapt-param ' -e 0.1 -m 5' \ --nth 6

align on trimmed/merged fastqs

if [ 'bowtie2' == 'bowtie2' ]; then python3 $(which encode_task_bowtie2.py) \ /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align_mito/shard-1/inputs/-347230842/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bowtie2_index.tar.gz \ R1/.fastq.gz \ R2/.fastq.gz \ --paired-end \ --multimapping 4 \ --mem-gb 4.004375195540488 \ --nth 6 fi

python3 $(which encode_task_post_align.py) \ R1/.fastq.gz $(ls .bam) \ --mito-chr-name chrM \ --chrsz /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align_mito/shard-1/inputs/-1634915880/GRCh38_EBV.chrom.sizes.tsv \ --mem-gb 4.004375195540488 \ --nth 6 rm -rf R1 R2` 2020-11-08 10:49:19,373 cromwell-system-akka.dispatchers.backend-dispatcher-111 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align_mito:1:1]: executing: if [ -z \"$SINGULARITY_BINDPATH\" ]; then export SINGULARITY_BINDPATH=; fi; \ if [ -z \"$SINGULARITY_CACHEDIR\" ]; then export SINGULARITY_CACHEDIR=; fi; /bin/bash /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/89b3e421-646a-4664-a06f-63c86e217676/call-align_mito/shard-1/execution/script 2020-11-08 10:49:22,113 cromwell-system-akka.dispatchers.backend-dispatcher-112 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align_mito:0:1]: job id: 564800 2020-11-08 10:49:22,113 cromwell-system-akka.dispatchers.backend-dispatcher-110 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align_mito:1:1]: job id: 564881 2020-11-08 10:49:22,113 cromwell-system-akka.dispatchers.backend-dispatcher-113 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align_mito:1:1]: Status change from - to WaitingForReturnCode 2020-11-08 10:49:22,114 cromwell-system-akka.dispatchers.backend-dispatcher-108 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align_mito:0:1]: Status change from - to WaitingForReturnCode 2020-11-08 10:51:11,851 cromwell-system-akka.dispatchers.backend-dispatcher-111 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align_mito:0:1]: Status change from WaitingForReturnCode to Done 2020-11-08 10:51:57,401 cromwell-system-akka.dispatchers.backend-dispatcher-111 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align:0:1]: Status change from WaitingForReturnCode to Done 2020-11-08 10:52:01,831 cromwell-system-akka.dispatchers.backend-dispatcher-109 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align_mito:1:1]: Status change from WaitingForReturnCode to Done 2020-11-08 10:53:33,461 cromwell-system-akka.dispatchers.backend-dispatcher-111 INFO - BackgroundConfigAsyncJobExecutionActor [UUID(89b3e421)atac.align:1:1]: Status change from WaitingForReturnCode to Done 2020-11-08 10:53:33,726 cromwell-system-akka.dispatchers.engine-dispatcher-39 INFO - WorkflowManagerActor Workflow 89b3e421-646a-4664-a06f-63c86e217676 failed (during ExecutingWorkflowState): cromwell.backend.standard.StandardAsyncExecutionActor$$anon$2: Failed to evaluate job outputs: Bad output 'align_mito.samstat_qc': Bad array access glob(".samstats.qc")[0]: Array size 0 does not have an index value '0' Bad output 'align_mito.read_len_log': Bad array access glob(".read_length.txt")[0]: Array size 0 does not have an index value '0' Bad output 'align_mito.bai': Bad array access glob(".bai")[0]: Array size 0 does not have an index value '0' Bad output 'align_mito.non_mito_samstat_qc': Bad array access glob("non_mito/.samstats.qc")[0]: Array size 0 does not have an index value '0' Bad output 'align_mito.bam': Bad array access glob("*.bam")[0]: Array size 0 does not have an index value '0' Bad output 'align_mito.read_len': ValueEvaluator[IdentifierLookup]: No suitable input for 'read_len_log' amongst {adapters_R1, auto_detect_adapter, mem_factor, fastqs_R2, multimapping, input_file_size_gb, time_hr, mito_chr_name, adapter, chrsz, tmp_fastqs, fastqs_R1, disk_factor, aligner, disk_gb, cpu, cutadapt_param, paired_end, tmp_adapters, mem_gb, adapters_R2, idx_tar, samtools_mem_gb} at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$handleExecutionSuccess$1(StandardAsyncExecutionActor.scala:916) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85) at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

cromwell.backend.standard.StandardAsyncExecutionActor$$anon$2: Failed to evaluate job outputs: Bad output 'align.samstat_qc': Bad array access glob(".samstats.qc")[0]: Array size 0 does not have an index value '0' Bad output 'align.read_len_log': Bad array access glob(".read_length.txt")[0]: Array size 0 does not have an index value '0' Bad output 'align.bai': Bad array access glob(".bai")[0]: Array size 0 does not have an index value '0' Bad output 'align.non_mito_samstat_qc': Bad array access glob("non_mito/.samstats.qc")[0]: Array size 0 does not have an index value '0' Bad output 'align.bam': Bad array access glob("*.bam")[0]: Array size 0 does not have an index value '0' Bad output 'align.read_len': ValueEvaluator[IdentifierLookup]: No suitable input for 'read_len_log' amongst {adapters_R1, auto_detect_adapter, mem_factor, fastqs_R2, multimapping, input_file_size_gb, time_hr, mito_chr_name, adapter, chrsz, tmp_fastqs, fastqs_R1, disk_factor, aligner, disk_gb, cpu, cutadapt_param, paired_end, tmp_adapters, mem_gb, adapters_R2, idx_tar, samtools_mem_gb} at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$handleExecutionSuccess$1(StandardAsyncExecutionActor.scala:916) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85) at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

cromwell.backend.standard.StandardAsyncExecutionActor$$anon$2: Failed to evaluate job outputs: Bad output 'align_mito.samstat_qc': Bad array access glob(".samstats.qc")[0]: Array size 0 does not have an index value '0' Bad output 'align_mito.read_len_log': Bad array access glob(".read_length.txt")[0]: Array size 0 does not have an index value '0' Bad output 'align_mito.bai': Bad array access glob(".bai")[0]: Array size 0 does not have an index value '0' Bad output 'align_mito.non_mito_samstat_qc': Bad array access glob("non_mito/.samstats.qc")[0]: Array size 0 does not have an index value '0' Bad output 'align_mito.bam': Bad array access glob("*.bam")[0]: Array size 0 does not have an index value '0' Bad output 'align_mito.read_len': ValueEvaluator[IdentifierLookup]: No suitable input for 'read_len_log' amongst {adapters_R1, auto_detect_adapter, mem_factor, fastqs_R2, multimapping, input_file_size_gb, time_hr, mito_chr_name, adapter, chrsz, tmp_fastqs, fastqs_R1, disk_factor, aligner, disk_gb, cpu, cutadapt_param, paired_end, tmp_adapters, mem_gb, adapters_R2, idx_tar, samtools_mem_gb} at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$handleExecutionSuccess$1(StandardAsyncExecutionActor.scala:916) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85) at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

cromwell.backend.standard.StandardAsyncExecutionActor$$anon$2: Failed to evaluate job outputs: Bad output 'align.samstat_qc': Bad array access glob(".samstats.qc")[0]: Array size 0 does not have an index value '0' Bad output 'align.read_len_log': Bad array access glob(".read_length.txt")[0]: Array size 0 does not have an index value '0' Bad output 'align.bai': Bad array access glob(".bai")[0]: Array size 0 does not have an index value '0' Bad output 'align.non_mito_samstat_qc': Bad array access glob("non_mito/.samstats.qc")[0]: Array size 0 does not have an index value '0' Bad output 'align.bam': Bad array access glob("*.bam")[0]: Array size 0 does not have an index value '0' Bad output 'align.read_len': ValueEvaluator[IdentifierLookup]: No suitable input for 'read_len_log' amongst {adapters_R1, auto_detect_adapter, mem_factor, fastqs_R2, multimapping, input_file_size_gb, time_hr, mito_chr_name, adapter, chrsz, tmp_fastqs, fastqs_R1, disk_factor, aligner, disk_gb, cpu, cutadapt_param, paired_end, tmp_adapters, mem_gb, adapters_R2, idx_tar, samtools_mem_gb} at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$handleExecutionSuccess$1(StandardAsyncExecutionActor.scala:916) at scala.util.Success.$anonfun$map$1(Try.scala:255) at scala.util.Success.map(Try.scala:213) at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33) at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:92) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85) at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:92) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

2020-11-08 10:53:33,726 cromwell-system-akka.dispatchers.engine-dispatcher-39 INFO - WorkflowManagerActor WorkflowActor-89b3e421-646a-4664-a06f-63c86e217676 is in a terminal state: WorkflowFailedState 2020-11-08 10:53:55,503 cromwell-system-akka.dispatchers.engine-dispatcher-7 INFO - SingleWorkflowRunnerActor workflow finished with status 'Failed'. 2020-11-08 10:53:57,215 cromwell-system-akka.dispatchers.engine-dispatcher-87 INFO - SingleWorkflowRunnerActor writing metadata to /media/santiago/SESUbuntu/ATACtmp/atac/20201108_101555_230992/metadata.json 2020-11-08 10:53:57,236 INFO - Workflow polling stopped 2020-11-08 10:53:57,242 INFO - 0 workflows released by cromid-043da09 2020-11-08 10:53:57,243 INFO - Shutting down WorkflowStoreActor - Timeout = 5 seconds 2020-11-08 10:53:57,245 INFO - Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds 2020-11-08 10:53:57,246 INFO - Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds 2020-11-08 10:53:57,246 INFO - JobExecutionTokenDispenser stopped 2020-11-08 10:53:57,247 cromwell-system-akka.dispatchers.engine-dispatcher-29 INFO - Aborting all running workflows. 2020-11-08 10:53:57,247 INFO - WorkflowStoreActor stopped 2020-11-08 10:53:57,250 INFO - WorkflowLogCopyRouter stopped 2020-11-08 10:53:57,250 INFO - Shutting down WorkflowManagerActor - Timeout = 3600 seconds 2020-11-08 10:53:57,250 cromwell-system-akka.dispatchers.engine-dispatcher-7 INFO - WorkflowManagerActor All workflows finished 2020-11-08 10:53:57,250 INFO - WorkflowManagerActor stopped 2020-11-08 10:53:57,383 INFO - Connection pools shut down 2020-11-08 10:53:57,385 INFO - Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds 2020-11-08 10:53:57,385 INFO - Shutting down JobStoreActor - Timeout = 1800 seconds 2020-11-08 10:53:57,386 INFO - Shutting down CallCacheWriteActor - Timeout = 1800 seconds 2020-11-08 10:53:57,386 INFO - Shutting down ServiceRegistryActor - Timeout = 1800 seconds 2020-11-08 10:53:57,386 INFO - Shutting down DockerHashActor - Timeout = 1800 seconds 2020-11-08 10:53:57,386 INFO - Shutting down IoProxy - Timeout = 1800 seconds

[leepc12]

Looks like align tasks failed due to unknow reasons. Can you upload caper's log too?

[santiago-es]

I'm not entirely sure what you mean by caper's log but I piped the caper run output to a file and here's what it says, please clarify if you mean a different file and thanks for the help:

• Started troubleshooting workflow: id=13d7b8ec-eaee-402e-b36a-4fd42655f46a, status=Failed
• Found failures JSON object. [ { "message": "Workflow failed", "causedBy": [ { "message": "Failed to evaluate job outputs", "causedBy": [ { "causedBy": [], "message": "Bad output 'align_mito.samstat_qc': Bad array access glob(\".samstats.qc\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align_mito.read_len_log': Bad array access glob(\".read_length.txt\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align_mito.bai': Bad array access glob(\".bai\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align_mito.non_mito_samstat_qc': Bad array access glob(\"non_mito/.samstats.qc\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align_mito.bam': Bad array access glob(\".bam\")[0]: Array size 0 does not have an index value '0'" }, { "message": "Bad output 'align_mito.read_len': ValueEvaluator[IdentifierLookup]: No suitable input for 'read_len_log' amongst {adapters_R1, auto_detect_adapter, mem_factor, fastqs_R2, multimapping, input_file_size_gb, time_hr, mito_chr_name, adapter, chrsz, tmp_fastqs, fastqs_R1, disk_factor, aligner, disk_gb, cpu, cutadapt_param, paired_end, tmp_adapters, mem_gb, adapters_R2, idx_tar, samtools_mem_gb}", "causedBy": [] } ] }, { "message": "Failed to evaluate job outputs", "causedBy": [ { "causedBy": [], "message": "Bad output 'align_mito.samstat_qc': Bad array access glob(\".samstats.qc\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align_mito.read_len_log': Bad array access glob(\".read_length.txt\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align_mito.bai': Bad array access glob(\".bai\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align_mito.non_mito_samstat_qc': Bad array access glob(\"non_mito/.samstats.qc\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align_mito.bam': Bad array access glob(\".bam\")[0]: Array size 0 does not have an index value '0'" }, { "message": "Bad output 'align_mito.read_len': ValueEvaluator[IdentifierLookup]: No suitable input for 'read_len_log' amongst {adapters_R1, auto_detect_adapter, mem_factor, fastqs_R2, multimapping, input_file_size_gb, time_hr, mito_chr_name, adapter, chrsz, tmp_fastqs, fastqs_R1, disk_factor, aligner, disk_gb, cpu, cutadapt_param, paired_end, tmp_adapters, mem_gb, adapters_R2, idx_tar, samtools_mem_gb}", "causedBy": [] } ] }, { "causedBy": [ { "causedBy": [], "message": "Bad output 'align.samstat_qc': Bad array access glob(\".samstats.qc\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align.read_len_log': Bad array access glob(\".read_length.txt\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align.bai': Bad array access glob(\".bai\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align.non_mito_samstat_qc': Bad array access glob(\"non_mito/.samstats.qc\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align.bam': Bad array access glob(\".bam\")[0]: Array size 0 does not have an index value '0'" }, { "message": "Bad output 'align.read_len': ValueEvaluator[IdentifierLookup]: No suitable input for 'read_len_log' amongst {adapters_R1, auto_detect_adapter, mem_factor, fastqs_R2, multimapping, input_file_size_gb, time_hr, mito_chr_name, adapter, chrsz, tmp_fastqs, fastqs_R1, disk_factor, aligner, disk_gb, cpu, cutadapt_param, paired_end, tmp_adapters, mem_gb, adapters_R2, idx_tar, samtools_mem_gb}", "causedBy": [] } ], "message": "Failed to evaluate job outputs" }, { "message": "Failed to evaluate job outputs", "causedBy": [ { "causedBy": [], "message": "Bad output 'align.samstat_qc': Bad array access glob(\".samstats.qc\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align.read_len_log': Bad array access glob(\".read_length.txt\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align.bai': Bad array access glob(\".bai\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align.non_mito_samstat_qc': Bad array access glob(\"non_mito/.samstats.qc\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align.bam': Bad array access glob(\".bam\")[0]: Array size 0 does not have an index value '0'" }, { "causedBy": [], "message": "Bad output 'align.read_len': ValueEvaluator[IdentifierLookup]: No suitable input for 'read_len_log' amongst {adapters_R1, auto_detect_adapter, mem_factor, fastqs_R2, multimapping, input_file_size_gb, time_hr, mito_chr_name, adapter, chrsz, tmp_fastqs, fastqs_R1, disk_factor, aligner, disk_gb, cpu, cutadapt_param, paired_end, tmp_adapters, mem_gb, adapters_R2, idx_tar, samtools_mem_gb}" } ] } ] } ]
• Recursively finding failures in calls (tasks)...

==== NAME=atac.align_mito, STATUS=Failed, PARENT= SHARD_IDX=0, RC=None, JOB_ID=6800 START=2020-11-09T18:40:32.645Z, END=2020-11-09T18:42:32.135Z STDOUT=/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align_mito/shard-0/execution/stdout STDERR=/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align_mito/shard-0/execution/stderr

==== NAME=atac.align_mito, STATUS=Failed, PARENT= SHARD_IDX=1, RC=None, JOB_ID=6835 START=2020-11-09T18:40:34.647Z, END=2020-11-09T18:43:00.770Z STDOUT=/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align_mito/shard-1/execution/stdout STDERR=/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align_mito/shard-1/execution/stderr

==== NAME=atac.align, STATUS=Failed, PARENT= SHARD_IDX=0, RC=None, JOB_ID=6766 START=2020-11-09T18:40:28.658Z, END=2020-11-09T18:43:43.461Z STDOUT=/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/stdout STDERR=/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/stderr

==== NAME=atac.align, STATUS=Failed, PARENT= SHARD_IDX=1, RC=None, JOB_ID=6781 START=2020-11-09T18:40:30.650Z, END=2020-11-09T18:43:40.400Z STDOUT=/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-1/execution/stdout STDERR=/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-1/execution/stderr

[leepc12]

Please post output of this command

$ cat /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/stdout
$ ls -l /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution 

[santiago-es]

I dont have stdout but I do have stdout.background, and cat returns: 6766

output of ls -l

-rwxr-xr-x 1 santiago santiago 2 Nov 9 10:42 ENCFF341MYG.subsampled.400.trim.merged.read_length.txt -rwxr-xr-x 1 santiago santiago 62061324 Nov 9 10:42 ENCFF341MYG.subsampled.400.trim.merged.srt.bam -rwxr-xr-x 1 santiago santiago 4796840 Nov 9 10:42 ENCFF341MYG.subsampled.400.trim.merged.srt.bam.bai -rwxr-xr-x 1 santiago santiago 383 Nov 9 10:42 ENCFF341MYG.subsampled.400.trim.merged.srt.samstats.qc drwxr-xr-x 2 santiago santiago 131072 Nov 9 10:43 glob-3bcbe4e7489c90f75e0523ac6f3a9385 -rwxr-xr-x 1 santiago santiago 0 Nov 9 10:43 glob-3bcbe4e7489c90f75e0523ac6f3a9385.list drwxr-xr-x 2 santiago santiago 131072 Nov 9 10:43 glob-6efbc60cb1e0959bab4e467327a9416c -rwxr-xr-x 1 santiago santiago 0 Nov 9 10:43 glob-6efbc60cb1e0959bab4e467327a9416c.list drwxr-xr-x 2 santiago santiago 131072 Nov 9 10:43 glob-773fb92850749a2b4a829cf3c8c4de27 -rwxr-xr-x 1 santiago santiago 0 Nov 9 10:43 glob-773fb92850749a2b4a829cf3c8c4de27.list drwxr-xr-x 2 santiago santiago 131072 Nov 9 10:43 glob-7b38d9959cf6f3deb83ac2bd156d8317 -rwxr-xr-x 1 santiago santiago 0 Nov 9 10:43 glob-7b38d9959cf6f3deb83ac2bd156d8317.list drwxr-xr-x 2 santiago santiago 131072 Nov 9 10:43 glob-bc1afa799665df5c7d6afd70d2ae2cb4 -rwxr-xr-x 1 santiago santiago 0 Nov 9 10:43 glob-bc1afa799665df5c7d6afd70d2ae2cb4.list drwxr-xr-x 2 santiago santiago 131072 Nov 9 10:42 non_mito -rwxr-xr-x 1 santiago santiago 2 Nov 9 10:42 rc -rwxr-xr-x 1 santiago santiago 14790 Nov 9 10:40 script -rwxr-xr-x 1 santiago santiago 415 Nov 9 10:40 script.background -rwxr-xr-x 1 santiago santiago 322 Nov 9 10:40 script.submit -rwxr-xr-x 1 santiago santiago 3950 Nov 9 10:43 stderr.background -rwxr-xr-x 1 santiago santiago 5 Nov 9 10:40 stdout.background -rwxr-xr-x 1 santiago santiago 733 Nov 9 10:40 write_tsv_b105dd0a3b054e9d4f2f71a9702e1814.tmp -rwxr-xr-x 1 santiago santiago 0 Nov 9 10:40 write_tsv_d41d8cd98f00b204e9800998ecf8427e.tmp

[leepc12]

Can you post contents of *.background files on that execution directory?

[santiago-es]

script.background

!/bin/bash

'/bin/bash' '/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/script.submit' < /dev/null || { rc=$?; if [ "$rc" -gt "128" ]; then echo $rc; else echo -1; fi } > /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/rc & echo $!

stderr.background

/bin/bash: /home/santiago/anaconda3/envs/encode-atac-seq-pipeline/lib/libtinfo.so.6: no version information available (required by /bin/bash) /bin/bash: /home/santiago/anaconda3/envs/encode-atac-seq-pipeline/lib/libtinfo.so.6: no version information available (required by /bin/bash) /bin/bash: /home/santiago/anaconda3/envs/encode-atac-seq-pipeline/lib/libtinfo.so.6: no version information available (required by /bin/bash) mkfifo: cannot create fifo '/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/tmp.da09f78a/out.6768': Operation not permitted mkfifo: cannot create fifo '/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/tmp.da09f78a/err.6768': Operation not permitted /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/script: line 16: /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/tmp.da09f78a/out.6768: No such file or directory /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/script: line 17: /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/tmp.da09f78a/err.6768: No such file or directory ln: failed to create hard link '/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/glob-7b38d9959cf6f3deb83ac2bd156d8317/ENCFF341MYG.subsampled.400.trim.merged.srt.samstats.qc' => '/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/ENCFF341MYG.subsampled.400.trim.merged.srt.samstats.qc': Operation not permitted ln: failed to create hard link '/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/glob-773fb92850749a2b4a829cf3c8c4de27/ENCFF341MYG.subsampled.400.trim.merged.read_length.txt' => '/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/ENCFF341MYG.subsampled.400.trim.merged.read_length.txt': Operation not permitted ln: failed to create hard link '/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/glob-6efbc60cb1e0959bab4e467327a9416c/ENCFF341MYG.subsampled.400.trim.merged.srt.bam.bai' => '/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/ENCFF341MYG.subsampled.400.trim.merged.srt.bam.bai': Operation not permitted ln: failed to create hard link '/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/glob-bc1afa799665df5c7d6afd70d2ae2cb4/ENCFF341MYG.subsampled.400.trim.merged.srt.no_chrM.samstats.qc' => '/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/non_mito/ENCFF341MYG.subsampled.400.trim.merged.srt.no_chrM.samstats.qc': Operation not permitted ln: failed to create hard link '/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/glob-3bcbe4e7489c90f75e0523ac6f3a9385/ENCFF341MYG.subsampled.400.trim.merged.srt.bam' => '/media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls/atac/13d7b8ec-eaee-402e-b36a-4fd42655f46a/call-align/shard-0/execution/ENCFF341MYG.subsampled.400.trim.merged.srt.bam': Operation not permitted

stdout.background

6766

[leepc12]

Do you have write permission on the output directory /media/santiago/SESUbuntu/201029_Artandi/Data/Intensities/BaseCalls (defined with caper's out-dir in conf file or --out-dir in cmd line args)?

[santiago-es]

I definitely should... but if you think thats the issue should I try changing the outdir to a different drive?

[santiago-es]

so I've double checked and I do: (ls -al outside .../BaseCalls

drwxr-xr-x 12 santiago santiago 131072 Nov 9 14:28 BaseCalls

[leepc12]

Does your system allow soft-linking (ln -s) and hard-linking (ln) on that directory?

[santiago-es]

hmm apparently it does not!

I tried echo 'this is a test' > foo and ln foo bar and received ln: failed ....operation not permitted

[santiago-es]

so as it turns out, this was failing because of the above issue. I changed the output directory to another drive and that solved the issue. Thanks for your help!