search

ENCODE-DCC / chip-seq-pipeline2

ENCODE ChIP-seq pipeline
MIT License
234 stars 123 forks source link

Pipeline gets stuck on first stage (read_genome_tsv) #215

Closed kdemuren closed 3 years ago

kdemuren commented 3 years ago

I've run the pipeline successfully many times before (last time end of January), but now it keeps getting stuck for hours/days after reading in the genome data. Running on a SLURM system using the included Conda environment.

JSON:

{
    "chip.genome_tsv" : "/net/bmc-pub14/data/boyer/users/kdemuren/chip-seq-pipeline2/genome/mm9/mm9.tsv",
    "chip.paired_end" : false,
    "chip.ctl_paired_end" : false,

    "chip.fastqs_rep1_R1" : ["/net/bmc-pub15/data/bmc/public/Boyer/210105Boy/D21-160-4854L/210105Boy_D21-160_NA_sequence.fastq"],
    "chip.fastqs_rep2_R1" : ["/net/bmc-pub14/data/boyer/users/kdemuren/SeqFiles/201005Boy/D20-4957-4668G/201005Boy_D20-4957_NA_sequence.fastq"],

    "chip.ctl_fastqs_rep1_R1" : ["/net/bmc-pub15/data/bmc/public/Boyer/210105Boy/D21-165-4854L/210105Boy_D21-165_NA_sequence.fastq"],

    "chip.title" : "CP TEAD1-HA DMSO",
    "chip.description" : "CP TEAD1-HA DMSO - Tead1-FKBP",

    "chip.pipeline_type" : "tf",
    "chip.aligner" : "bowtie2",

    "chip.align_only" : false,
    "chip.true_rep_only" : false,

    "chip.always_use_pooled_ctl" : true
}

Cromwell output:

2021-03-01 19:01:26,103  INFO  - Running with database db.url = jdbc:hsqldb:mem:7bb53a68-3842-4a84-bedc-25fb4107ab58;shutdown=false;hsqldb.tx=mvcc
2021-03-01 19:01:34,948  INFO  - Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
2021-03-01 19:01:34,962  INFO  - [RenameWorkflowOptionsInMetadata] 100%
2021-03-01 19:01:35,056  INFO  - Running with database db.url = jdbc:hsqldb:mem:05d392ec-9efb-464a-9623-ff024eb6d175;shutdown=false;hsqldb.tx=mvcc
2021-03-01 19:01:35,482  INFO  - Slf4jLogger started
2021-03-01 19:01:35,651 cromwell-system-akka.dispatchers.engine-dispatcher-5 INFO  - Workflow heartbeat configuration:
{
  "cromwellId" : "cromid-65201fc",
  "heartbeatInterval" : "2 minutes",
  "ttl" : "10 minutes",
  "failureShutdownDuration" : "5 minutes",
  "writeBatchSize" : 10000,
  "writeThreshold" : 10000
}
2021-03-01 19:01:35,689 cromwell-system-akka.dispatchers.service-dispatcher-12 INFO  - Metadata summary refreshing every 1 second.
2021-03-01 19:01:35,705 cromwell-system-akka.dispatchers.service-dispatcher-8 INFO  - WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
2021-03-01 19:01:35,706 cromwell-system-akka.actor.default-dispatcher-3 INFO  - KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
2021-03-01 19:01:35,727 cromwell-system-akka.dispatchers.engine-dispatcher-56 INFO  - CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
2021-03-01 19:01:35,728  WARN  - 'docker.hash-lookup.gcr-api-queries-per-100-seconds' is being deprecated, use 'docker.hash-lookup.gcr.throttle' instead (see reference.conf)
2021-03-01 19:01:36,022 cromwell-system-akka.dispatchers.engine-dispatcher-56 INFO  - JobExecutionTokenDispenser - Distribution rate: 1 per 2 seconds.
2021-03-01 19:01:36,048 cromwell-system-akka.dispatchers.engine-dispatcher-5 INFO  - SingleWorkflowRunnerActor: Version 52
2021-03-01 19:01:36,053 cromwell-system-akka.dispatchers.engine-dispatcher-5 INFO  - SingleWorkflowRunnerActor: Submitting workflow
2021-03-01 19:01:36,108 cromwell-system-akka.dispatchers.api-dispatcher-62 INFO  - Unspecified type (Unspecified version) workflow 1eba32a4-5bb0-431c-846b-297ab6bd676d submitted
2021-03-01 19:01:36,128 cromwell-system-akka.dispatchers.engine-dispatcher-60 INFO  - SingleWorkflowRunnerActor: Workflow submitted UUID(1eba32a4-5bb0-431c-846b-297ab6bd676d)
2021-03-01 19:01:36,132 cromwell-system-akka.dispatchers.engine-dispatcher-56 INFO  - 1 new workflows fetched by cromid-65201fc: 1eba32a4-5bb0-431c-846b-297ab6bd676d
2021-03-01 19:01:36,141 cromwell-system-akka.dispatchers.engine-dispatcher-60 INFO  - WorkflowManagerActor Starting workflow UUID(1eba32a4-5bb0-431c-846b-297ab6bd676d)
2021-03-01 19:01:36,148 cromwell-system-akka.dispatchers.engine-dispatcher-60 INFO  - WorkflowManagerActor Successfully started WorkflowActor-1eba32a4-5bb0-431c-846b-297ab6bd676d
2021-03-01 19:01:36,148 cromwell-system-akka.dispatchers.engine-dispatcher-60 INFO  - Retrieved 1 workflows from the WorkflowStoreActor
2021-03-01 19:01:36,166 cromwell-system-akka.dispatchers.engine-dispatcher-56 INFO  - WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
2021-03-01 19:01:36,251 cromwell-system-akka.dispatchers.engine-dispatcher-60 INFO  - MaterializeWorkflowDescriptorActor [UUID(1eba32a4)]: Parsing workflow as WDL 1.0
2021-03-01 19:01:38,452 cromwell-system-akka.dispatchers.engine-dispatcher-60 INFO  - MaterializeWorkflowDescriptorActor [UUID(1eba32a4)]: Call-to-Backend assignments: chip.jsd -> slurm, chip.spr -> slurm, chip.error_use_bwa_mem_for_non_bwa -> slurm, chip.error_input_data -> slurm, chip.filter_no_dedup -> slurm, chip.overlap_pr -> slurm, chip.filter_ctl -> slurm, chip.subsample_ctl -> slurm, chip.idr_ppr -> slurm, chip.pool_ta_pr1 -> slurm, chip.reproducibility_idr -> slurm, chip.xcor -> slurm, chip.pool_ta_ctl -> slurm, chip.overlap_ppr -> slurm, chip.call_peak_pooled -> slurm, chip.call_peak -> slurm, chip.filter -> slurm, chip.error_ctl_fastq_input_required_for_control_mode -> slurm, chip.pool_ta_pr2 -> slurm, chip.error_control_required -> slurm, chip.qc_report -> slurm, chip.filter_R1 -> slurm, chip.choose_ctl -> slurm, chip.idr_pr -> slurm, chip.call_peak_pr1 -> slurm, chip.macs2_signal_track_pooled -> slurm, chip.bam2ta_no_dedup_R1 -> slurm, chip.fraglen_mean -> slurm, chip.error_ctl_input_defined_in_control_mode -> slurm, chip.error_subsample_pooled_control_with_mixed_endedness -> slurm, chip.overlap -> slurm, chip.error_custom_aligner -> slurm, chip.bam2ta -> slurm, chip.align -> slurm, chip.idr -> slurm, chip.align_R1 -> slurm, chip.call_peak_ppr2 -> slurm, chip.subsample_ctl_pooled -> slurm, chip.pool_ta -> slurm, chip.reproducibility_overlap -> slurm, chip.read_genome_tsv -> slurm, chip.macs2_signal_track -> slurm, chip.align_ctl -> slurm, chip.error_wrong_aligner -> slurm, chip.gc_bias -> slurm, chip.call_peak_pr2 -> slurm, chip.count_signal_track_pooled -> slurm, chip.bam2ta_no_dedup -> slurm, chip.pool_blacklist -> slurm, chip.call_peak_ppr1 -> slurm, chip.bam2ta_ctl -> slurm, chip.count_signal_track -> slurm
2021-03-01 19:01:38,674 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,675 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,676 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,676 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,676 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,676 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,677 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,677 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,677 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,677 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,677 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,678 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,678 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,678 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,678 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,678 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,679 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,679 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,679 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,679 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,679 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,679 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,680 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,680 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,680 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,680 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,680 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,680 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,681 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,681 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,681 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,681 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,681 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,681 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,682 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,682 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,682 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,682 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,682 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,682 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,683 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,683 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,683 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,683 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,683 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,683 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,684 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,684 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,684 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,684 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,685 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:38,685 cromwell-system-akka.dispatchers.backend-dispatcher-99 WARN  - slurm [UUID(1eba32a4)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-03-01 19:01:41,036 cromwell-system-akka.dispatchers.engine-dispatcher-5 INFO  - Not triggering log of token queue status. Effective log interval = None
2021-03-01 19:01:42,996 cromwell-system-akka.dispatchers.engine-dispatcher-43 INFO  - WorkflowExecutionActor-1eba32a4-5bb0-431c-846b-297ab6bd676d [UUID(1eba32a4)]: Starting chip.read_genome_tsv
2021-03-01 19:01:44,045 cromwell-system-akka.dispatchers.engine-dispatcher-43 INFO  - Assigned new job execution tokens to the following groups: 1eba32a4: 1
2021-03-01 19:01:44,204 cromwell-system-akka.dispatchers.engine-dispatcher-127 INFO  - 1eba32a4-5bb0-431c-846b-297ab6bd676d-EngineJobExecutionActor-chip.read_genome_tsv:NA:1 [UUID(1eba32a4)]: Could not copy a suitable cache hit for 1eba32a4:chip.read_genome_tsv:-1:1. No copy attempts were made.
2021-03-01 19:01:44,229 cromwell-system-akka.dispatchers.backend-dispatcher-138 WARN  - BackgroundConfigAsyncJobExecutionActor [UUID(1eba32a4)chip.read_genome_tsv:NA:1]: Unrecognized runtime attribute keys: disks
2021-03-01 19:01:44,317 cromwell-system-akka.dispatchers.backend-dispatcher-138 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(1eba32a4)chip.read_genome_tsv:NA:1]: `echo "$(basename /net/bmc-pub14/data/boyer/users/kdemuren/output-chip/CP_TEAD1HA_DMSO/chip/1eba32a4-5bb0-431c-846b-297ab6bd676d/call-read_genome_tsv/inputs/-1769241308/mm9.tsv)" > genome_name
# create empty files for all entries
touch ref_fa bowtie2_idx_tar bwa_idx_tar chrsz gensz blacklist blacklist2
touch mito_chr_name
touch regex_bfilt_peak_chr_name

python <<CODE
import os
with open('/net/bmc-pub14/data/boyer/users/kdemuren/output-chip/CP_TEAD1HA_DMSO/chip/1eba32a4-5bb0-431c-846b-297ab6bd676d/call-read_genome_tsv/inputs/-1769241308/mm9.tsv','r') as fp:
    for line in fp:
        arr = line.strip('\n').split('\t')
        if arr:
            key, val = arr
            with open(key,'w') as fp2:
                fp2.write(val)
CODE`
2021-03-01 19:01:44,518 cromwell-system-akka.dispatchers.backend-dispatcher-138 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(1eba32a4)chip.read_genome_tsv:NA:1]: executing: if [ -z \"$SINGULARITY_BINDPATH\" ]; then export SINGULARITY_BINDPATH=; fi; \
if [ -z \"$SINGULARITY_CACHEDIR\" ]; then export SINGULARITY_CACHEDIR=; fi;

ITER=0
until [ $ITER -ge 3 ]; do
    sbatch \
        --export=ALL \
        -J cromwell_1eba32a4_read_genome_tsv \
        -D /net/bmc-pub14/data/boyer/users/kdemuren/output-chip/CP_TEAD1HA_DMSO/chip/1eba32a4-5bb0-431c-846b-297ab6bd676d/call-read_genome_tsv \
        -o /net/bmc-pub14/data/boyer/users/kdemuren/output-chip/CP_TEAD1HA_DMSO/chip/1eba32a4-5bb0-431c-846b-297ab6bd676d/call-read_genome_tsv/execution/stdout \
        -e /net/bmc-pub14/data/boyer/users/kdemuren/output-chip/CP_TEAD1HA_DMSO/chip/1eba32a4-5bb0-431c-846b-297ab6bd676d/call-read_genome_tsv/execution/stderr \
        -t 60 \
        -n 1 \
        --ntasks-per-node=1 \
        --cpus-per-task=1 \
        --mem=2048 \
        -p normal \
         \
         \
         \
        --wrap "/bin/bash /net/bmc-pub14/data/boyer/users/kdemuren/output-chip/CP_TEAD1HA_DMSO/chip/1eba32a4-5bb0-431c-846b-297ab6bd676d/call-read_genome_tsv/execution/script" \
        && break
    ITER=$[$ITER+1]
    sleep 30
done
2021-03-01 19:01:45,732 cromwell-system-akka.dispatchers.backend-dispatcher-138 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(1eba32a4)chip.read_genome_tsv:NA:1]: job id: 21301
2021-03-01 19:01:45,737 cromwell-system-akka.dispatchers.backend-dispatcher-140 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(1eba32a4)chip.read_genome_tsv:NA:1]: Status change from - to WaitingForReturnCode
kdemuren commented 3 years ago

Re-re-installed the conda environment rather than just updating it and it fixed the problem, so closing this issue.