search

ENCODE-DCC / atac-seq-pipeline

ENCODE ATAC-seq pipeline
MIT License
384 stars 172 forks source link

Pipeline failure at atac.align_mito #343

Open LaraBM opened 2 years ago

LaraBM commented 2 years ago

Hi there,

I have a problem running the pipeline which I have set up on a SLURM HPC. It had worked previously and I used it regularly for the best part of a year. However it stopped working after a job timed out back in August and so I decided to re-install the pipeline/pipeline environment from scratch. Since then, I have not able to get it running like before as it always stops at this align step. After installing the pipeline's environment I also installed croo and caper in the environment, as instructed, and I configured the caper config file too. With regards to the caper configuration, I spoke to our HPC team and they said that I do not need to state partition or account as that would be stated when I submit the jobs. With regards to general install, I didn't use mamba to avoid the keys/disks error but I still get it. I have also looked at other issues raised here in the forum that were similar and have seen that the suggestion is to uninstall and re-install the environment again which I have done as well. Could I get some help on how to fix this please?

I have the input JSON and the output below. This is a sample I did successfully run a year ago, but now fails on the new pipeline.

Many thanks in advance, Lara

OS/Module Versions

SLURM cluster Conda: conda v4.9.2 Pipeline: v1.9.2 Caper: caper v1.6.3

Caper Configuration File

backend=slurm

# define one of the followings (or both) according to your
# cluster's SLURM configuration.
slurm-partition=
slurm-account=

# Hashing strategy for call-caching (3 choices)
# This parameter is for local (local/slurm/sge/pbs) backend only.
# This is important for call-caching,
# which means re-using outputs from previous/failed workflows.
# Cache will miss if different strategy is used.
# "file" method has been default for all old versions of Caper<1.0.
# "path+modtime" is a new default for Caper>=1.0,
#   file: use md5sum hash (slow).
#   path: use path.
#   path+modtime: use path and modification time.
local-hash-strat=path+modtime

# Local directory for localized files and Cromwell's intermediate files
# If not defined, Caper will make .caper_tmp/ on local-out-dir or CWD.
# /tmp is not recommended here since Caper store all localized data files
# on this directory (e.g. input FASTQs defined as URLs in input JSON).
local-loc-dir=

cromwell=/home/lem2g17/.caper/cromwell_jar/cromwell-59.jar
womtool=/home/lem2g17/.caper/womtool_jar/womtool-59.jar

Input JSON File

{
        "atac.paired_end": true,
        "atac.auto_detect_adapter": true,
        "atac.enable_xcor": true,
        "atac.title": "S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301",
        "atac.pipeline_type": "atac",
        "atac.description": "ATAC on healthy memory B cells (Beekman et al)",
        "atac.genome_tsv": "/scratch/lem2g17/atac-seq-pipeline/GENOME/hg38.tsv",
        "atac.fastqs_rep1_R1": [
                "/scratch/lem2g17/atac-seq-pipeline/fastqs/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301.read1.fq"
        ],
        "atac.fastqs_rep1_R2": [
                "/scratch/lem2g17/atac-seq-pipeline/fastqs/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301.read2.fq"
        ]
}

SLURM-out Output

2021-10-20 14:31:20,891|caper.caper_base|INFO| Creating a timestamped temporary directory. /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/.caper_tmp/atac/20211020_143117_234628
2021-10-20 14:31:20,891|caper.caper_runner|INFO| Localizing files on work_dir. /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/.caper_tmp/atac/20211020_143117_234628
2021-10-20 14:32:17,961|caper.cromwell|INFO| Validating WDL/inputs/imports with Womtool...
2021-10-20 14:34:37,528|caper.cromwell|INFO| Passed Womtool validation.
2021-10-20 14:34:37,529|caper.caper_runner|INFO| launching run: wdl=/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac.wdl, inputs=/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/caper_instructions.json, backend_conf=/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/.caper_tmp/atac/20211020_143117_234628/backend.conf
2021-10-20 14:37:51,418|caper.cromwell_workflow_monitor|INFO| Workflow: id=9850fe74-b790-44fb-a527-982b7d252fac, status=Submitted
2021-10-20 14:37:51,554|caper.cromwell_workflow_monitor|INFO| Workflow: id=9850fe74-b790-44fb-a527-982b7d252fac, status=Running
2021-10-20 14:39:50,205|caper.cromwell_workflow_monitor|INFO| Task: id=9850fe74-b790-44fb-a527-982b7d252fac, task=atac.read_genome_tsv:-1, retry=0, status=Started, job_id=289033
2021-10-20 14:39:50,217|caper.cromwell_workflow_monitor|INFO| Task: id=9850fe74-b790-44fb-a527-982b7d252fac, task=atac.read_genome_tsv:-1, retry=0, status=WaitingForReturnCode
2021-10-20 14:41:13,350|caper.cromwell_workflow_monitor|INFO| Task: id=9850fe74-b790-44fb-a527-982b7d252fac, task=atac.read_genome_tsv:-1, retry=0, status=Done
2021-10-20 14:45:30,165|caper.cromwell_workflow_monitor|INFO| Task: id=9850fe74-b790-44fb-a527-982b7d252fac, task=atac.align_mito:0, retry=0, status=Started, job_id=289622
2021-10-20 14:45:30,165|caper.cromwell_workflow_monitor|INFO| Task: id=9850fe74-b790-44fb-a527-982b7d252fac, task=atac.align:0, retry=0, status=Started, job_id=289620
2021-10-20 14:45:30,172|caper.cromwell_workflow_monitor|INFO| Task: id=9850fe74-b790-44fb-a527-982b7d252fac, task=atac.align_mito:0, retry=0, status=WaitingForReturnCode
2021-10-20 14:45:30,174|caper.cromwell_workflow_monitor|INFO| Task: id=9850fe74-b790-44fb-a527-982b7d252fac, task=atac.align:0, retry=0, status=WaitingForReturnCode
2021-10-20 15:05:18,753|caper.cromwell_workflow_monitor|INFO| Task: id=9850fe74-b790-44fb-a527-982b7d252fac, task=atac.align_mito:0, retry=0, status=Done
2021-10-20 15:06:24,123|caper.cromwell_workflow_monitor|INFO| Task: id=9850fe74-b790-44fb-a527-982b7d252fac, task=atac.align:0, retry=0, status=Done
2021-10-20 15:07:50,151|caper.cromwell_workflow_monitor|INFO| Task: id=9850fe74-b790-44fb-a527-982b7d252fac, task=atac.align_mito:0, retry=1, status=Started, job_id=292425
2021-10-20 15:07:50,175|caper.cromwell_workflow_monitor|INFO| Task: id=9850fe74-b790-44fb-a527-982b7d252fac, task=atac.align_mito:0, retry=1, status=WaitingForReturnCode
2021-10-20 15:09:00,150|caper.cromwell_workflow_monitor|INFO| Task: id=9850fe74-b790-44fb-a527-982b7d252fac, task=atac.align:0, retry=1, status=Started, job_id=292597
2021-10-20 15:09:00,153|caper.cromwell_workflow_monitor|INFO| Task: id=9850fe74-b790-44fb-a527-982b7d252fac, task=atac.align:0, retry=1, status=WaitingForReturnCode
2021-10-20 15:56:08,979|caper.cromwell_workflow_monitor|INFO| Task: id=9850fe74-b790-44fb-a527-982b7d252fac, task=atac.align_mito:0, retry=1, status=Done
2021-10-20 15:56:17,693|caper.cromwell_workflow_monitor|INFO| Task: id=9850fe74-b790-44fb-a527-982b7d252fac, task=atac.align:0, retry=1, status=Done
2021-10-20 15:56:28,728|caper.cromwell_workflow_monitor|INFO| Workflow: id=9850fe74-b790-44fb-a527-982b7d252fac, status=Failed
2021-10-20 15:57:05,079|caper.cromwell_metadata|INFO| Wrote metadata file. /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/metadata.json
2021-10-20 15:57:05,079|caper.cromwell|INFO| Workflow failed. Auto-troubleshooting...
2021-10-20 15:57:15,499|caper.nb_subproc_thread|ERROR| Cromwell failed. returncode=1
2021-10-20 15:57:15,499|caper.cli|ERROR| Check stdout in /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/cromwell.out
* Started troubleshooting workflow: id=9850fe74-b790-44fb-a527-982b7d252fac, status=Failed
* Found failures JSON object.
[
    {
        "causedBy": [
            {
                "message": "Job atac.align_mito:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.",
                "causedBy": []
            },
            {
                "causedBy": [],
                "message": "Job atac.align:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details."
            }
        ],
        "message": "Workflow failed"
    }
]
* Recursively finding failures in calls (tasks)...

==== NAME=atac.align_mito, STATUS=RetryableFailure, PARENT=
SHARD_IDX=0, RC=1, JOB_ID=289622
START=2021-10-20T13:42:59.049Z, END=2021-10-20T14:05:25.163Z
STDOUT=/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align_mito/shard-0/execution/stdout
STDERR=/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align_mito/shard-0/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 192, in <module>
    main()
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 169, in main
    args.out_dir)
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 102, in bowtie2_pe
    tmp_bam=tmp_bam,
  File "/mainfs/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_lib_common.py", line 359, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=172874, PGID=172874, RC=127, DURATION_SEC=1.2
STDERR=/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/bowtie2-align-s: error while loading shared libraries: libtbb.so.2: cannot open shared object file: No such file or directory
(ERR): Description of arguments failed!
Exiting now ...
STDOUT=
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align_mito/shard-0/execution/*.samstats.qc': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align_mito/shard-0/execution/*.read_length.txt': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align_mito/shard-0/execution/*.bai': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align_mito/shard-0/execution/non_mito/*.samstats.qc': No such file or directory

==== NAME=atac.align_mito, STATUS=Failed, PARENT=
SHARD_IDX=0, RC=1, JOB_ID=292425
START=2021-10-20T14:05:32.363Z, END=2021-10-20T14:56:18.217Z
STDOUT=/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align_mito/shard-0/attempt-2/execution/stdout
STDERR=/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align_mito/shard-0/attempt-2/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 192, in <module>
    main()
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 169, in main
    args.out_dir)
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 102, in bowtie2_pe
    tmp_bam=tmp_bam,
  File "/mainfs/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_lib_common.py", line 359, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=2212, PGID=2212, RC=127, DURATION_SEC=152.1
STDERR=/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/bowtie2-align-s: error while loading shared libraries: libtbb.so.2: cannot open shared object file: No such file or directory
(ERR): Description of arguments failed!
Exiting now ...
STDOUT=
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align_mito/shard-0/attempt-2/execution/*.samstats.qc': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align_mito/shard-0/attempt-2/execution/*.read_length.txt': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align_mito/shard-0/attempt-2/execution/*.bai': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align_mito/shard-0/attempt-2/execution/non_mito/*.samstats.qc': No such file or directory

==== NAME=atac.align, STATUS=RetryableFailure, PARENT=
SHARD_IDX=0, RC=1, JOB_ID=289620
START=2021-10-20T13:42:58.179Z, END=2021-10-20T14:06:30.160Z
STDOUT=/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align/shard-0/execution/stdout
STDERR=/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align/shard-0/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 192, in <module>
    main()
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 169, in main
    args.out_dir)
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 102, in bowtie2_pe
    tmp_bam=tmp_bam,
  File "/mainfs/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_lib_common.py", line 359, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=173046, PGID=173046, RC=127, DURATION_SEC=0.1
STDERR=/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/bowtie2-align-s: error while loading shared libraries: libtbb.so.2: cannot open shared object file: No such file or directory
(ERR): Description of arguments failed!
Exiting now ...
STDOUT=
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align/shard-0/execution/*.samstats.qc': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align/shard-0/execution/*.read_length.txt': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align/shard-0/execution/*.bai': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align/shard-0/execution/non_mito/*.samstats.qc': No such file or directory

==== NAME=atac.align, STATUS=Failed, PARENT=
SHARD_IDX=0, RC=1, JOB_ID=292597
START=2021-10-20T14:06:37.701Z, END=2021-10-20T14:56:19.522Z
STDOUT=/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align/shard-0/attempt-2/execution/stdout
STDERR=/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align/shard-0/attempt-2/execution/stderr
STDERR_CONTENTS=
Traceback (most recent call last):
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 192, in <module>
    main()
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 169, in main
    args.out_dir)
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 102, in bowtie2_pe
    tmp_bam=tmp_bam,
  File "/mainfs/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_lib_common.py", line 359, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=2329, PGID=2329, RC=127, DURATION_SEC=124.1
STDERR=/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/bowtie2-align-s: error while loading shared libraries: libtbb.so.2: cannot open shared object file: No such file or directory
(ERR): Description of arguments failed!
Exiting now ...
STDOUT=
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align/shard-0/attempt-2/execution/*.samstats.qc': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align/shard-0/attempt-2/execution/*.read_length.txt': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align/shard-0/attempt-2/execution/*.bai': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/9850fe74-b790-44fb-a527-982b7d252fac/call-align/shard-0/attempt-2/execution/non_mito/*.samstats.qc': No such file or directory

Cromwell.out Output

2021-10-20 20:20:22,280  INFO  - Running with database db.url = jdbc:hsqldb:mem:6f42d304-9869-4336-b1fc-be0c18ac913a;shutdown=false;hsqldb.tx=mvcc
2021-10-20 20:20:45,871  INFO  - Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
2021-10-20 20:20:45,955  INFO  - [RenameWorkflowOptionsInMetadata] 100%
2021-10-20 20:20:46,676  INFO  - Running with database db.url = jdbc:hsqldb:mem:4c95467b-03f9-4003-839b-7b25bfe0af6e;shutdown=false;hsqldb.tx=mvcc
2021-10-20 20:20:49,213  INFO  - Slf4jLogger started
2021-10-20 20:20:49,846 cromwell-system-akka.dispatchers.engine-dispatcher-8 INFO  - Workflow heartbeat configuration:
{
  "cromwellId" : "cromid-1af6c31",
  "heartbeatInterval" : "2 minutes",
  "ttl" : "10 minutes",
  "failureShutdownDuration" : "5 minutes",
  "writeBatchSize" : 10000,
  "writeThreshold" : 10000
}
2021-10-20 20:20:50,112 cromwell-system-akka.dispatchers.service-dispatcher-13 INFO  - WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
2021-10-20 20:20:50,203 cromwell-system-akka.actor.default-dispatcher-3 INFO  - KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
2021-10-20 20:20:50,590  WARN  - 'docker.hash-lookup.gcr-api-queries-per-100-seconds' is being deprecated, use 'docker.hash-lookup.gcr.throttle' instead (see reference.conf)
2021-10-20 20:20:50,602 cromwell-system-akka.dispatchers.service-dispatcher-16 INFO  - Metadata summary refreshing every 1 second.
2021-10-20 20:20:50,717 cromwell-system-akka.dispatchers.engine-dispatcher-33 INFO  - CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
2021-10-20 20:20:50,901 cromwell-system-akka.dispatchers.engine-dispatcher-56 INFO  - JobExecutionTokenDispenser - Distribution rate: 1 per 2 seconds.
2021-10-20 20:20:51,252 cromwell-system-akka.dispatchers.engine-dispatcher-8 INFO  - SingleWorkflowRunnerActor: Version 59
2021-10-20 20:20:51,295 cromwell-system-akka.dispatchers.engine-dispatcher-8 INFO  - SingleWorkflowRunnerActor: Submitting workflow
2021-10-20 20:20:51,559 cromwell-system-akka.dispatchers.api-dispatcher-57 INFO  - Unspecified type (Unspecified version) workflow 35e483c9-d59f-4446-ae7c-ce049b969e9a submitted
2021-10-20 20:20:51,662 cromwell-system-akka.dispatchers.engine-dispatcher-33 INFO  - SingleWorkflowRunnerActor: Workflow submitted UUID(35e483c9-d59f-4446-ae7c-ce049b969e9a)
2021-10-20 20:20:51,686 cromwell-system-akka.dispatchers.engine-dispatcher-32 INFO  - 1 new workflows fetched by cromid-1af6c31: 35e483c9-d59f-4446-ae7c-ce049b969e9a
2021-10-20 20:20:51,707 cromwell-system-akka.dispatchers.engine-dispatcher-33 INFO  - WorkflowManagerActor: Starting workflow UUID(35e483c9-d59f-4446-ae7c-ce049b969e9a)
2021-10-20 20:20:51,722 cromwell-system-akka.dispatchers.engine-dispatcher-33 INFO  - WorkflowManagerActor: Successfully started WorkflowActor-35e483c9-d59f-4446-ae7c-ce049b969e9a
2021-10-20 20:20:51,724 cromwell-system-akka.dispatchers.engine-dispatcher-33 INFO  - Retrieved 1 workflows from the WorkflowStoreActor
2021-10-20 20:20:51,882 cromwell-system-akka.dispatchers.engine-dispatcher-55 INFO  - WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
2021-10-20 20:20:55,918 cromwell-system-akka.dispatchers.engine-dispatcher-51 INFO  - Not triggering log of token queue status. Effective log interval = None
2021-10-20 20:20:58,870 cromwell-system-akka.dispatchers.engine-dispatcher-55 INFO  - MaterializeWorkflowDescriptorActor [UUID(35e483c9)]: Parsing workflow as WDL 1.0
2021-10-20 20:21:04,452 cromwell-system-akka.dispatchers.engine-dispatcher-55 INFO  - MaterializeWorkflowDescriptorActor [UUID(35e483c9)]: Call-to-Backend assignments: atac.annot_enrich -> slurm, atac.reproducibility_overlap -> slurm, atac.call_peak_ppr2 -> slurm, atac.count_signal_track_pooled -> slurm, atac.align -> slurm, atac.call_peak_pr2 -> slurm, atac.spr -> slurm, atac.call_peak -> slurm, atac.pool_ta_pr1 -> slurm, atac.read_genome_tsv -> slurm, atac.overlap_pr -> slurm, atac.count_signal_track -> slurm, atac.preseq -> slurm, atac.qc_report -> slurm, atac.bam2ta -> slurm, atac.call_peak_pr1 -> slurm, atac.align_mito -> slurm, atac.pool_blacklist -> slurm, atac.idr -> slurm, atac.fraglen_stat_pe -> slurm, atac.overlap -> slurm, atac.filter_no_dedup -> slurm, atac.idr_ppr -> slurm, atac.reproducibility_idr -> slurm, atac.macs2_signal_track -> slurm, atac.frac_mito -> slurm, atac.filter -> slurm, atac.bam2ta_no_dedup -> slurm, atac.overlap_ppr -> slurm, atac.error_input_data -> slurm, atac.jsd -> slurm, atac.xcor -> slurm, atac.call_peak_pooled -> slurm, atac.tss_enrich -> slurm, atac.pool_ta_pr2 -> slurm, atac.idr_pr -> slurm, atac.macs2_signal_track_pooled -> slurm, atac.gc_bias -> slurm, atac.call_peak_ppr1 -> slurm, atac.pool_ta -> slurm, atac.compare_signal_to_roadmap -> slurm
2021-10-20 20:21:05,299 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,316 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,318 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,319 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,334 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,335 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,335 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,336 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,337 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,337 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,339 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,349 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,350 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,351 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,351 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,353 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,359 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,361 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,362 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,364 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,364 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,366 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,367 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,369 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,370 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,371 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,373 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,373 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,374 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,375 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,376 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,381 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,383 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,383 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,384 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,385 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,386 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,386 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,387 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [preemptible, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,387 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:05,389 cromwell-system-akka.dispatchers.backend-dispatcher-64 WARN  - slurm [UUID(35e483c9)]: Key/s [disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
2021-10-20 20:21:12,300 cromwell-system-akka.dispatchers.engine-dispatcher-37 INFO  - WorkflowExecutionActor-35e483c9-d59f-4446-ae7c-ce049b969e9a [UUID(35e483c9)]: Starting atac.read_genome_tsv
2021-10-20 20:21:12,962 cromwell-system-akka.dispatchers.engine-dispatcher-37 INFO  - Assigned new job execution tokens to the following groups: 35e483c9: 1
2021-10-20 20:21:13,302 cromwell-system-akka.dispatchers.engine-dispatcher-37 INFO  - 35e483c9-d59f-4446-ae7c-ce049b969e9a-EngineJobExecutionActor-atac.read_genome_tsv:NA:1 [UUID(35e483c9)]: Could not copy a suitable cache hit for 35e483c9:atac.read_genome_tsv:-1:1. No copy attempts were made.
2021-10-20 20:21:13,358 cromwell-system-akka.dispatchers.backend-dispatcher-66 WARN  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.read_genome_tsv:NA:1]: Unrecognized runtime attribute keys: disks
2021-10-20 20:21:40,502 cromwell-system-akka.dispatchers.backend-dispatcher-66 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.read_genome_tsv:NA:1]: `echo "$(basename /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-read_genome_tsv/inputs/-1950825054/hg38.tsv)" > genome_name
# create empty files for all entries
touch ref_fa bowtie2_idx_tar chrsz gensz blacklist blacklist2
touch ref_mito_fa
touch bowtie2_mito_idx_tar
touch tss tss_enrich # for backward compatibility
touch dnase prom enh reg2map reg2map_bed roadmap_meta
touch mito_chr_name
touch regex_bfilt_peak_chr_name

python <<CODE
import os
with open('/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-read_genome_tsv/inputs/-1950825054/hg38.tsv','r') as fp:
    for line in fp:
        arr = line.strip('\n').split('\t')
        if arr:
            key, val = arr
            with open(key,'w') as fp2:
                fp2.write(val)
CODE`
2021-10-20 20:22:45,213 cromwell-system-akka.dispatchers.backend-dispatcher-66 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.read_genome_tsv:NA:1]: executing: if [ -z \"$SINGULARITY_BINDPATH\" ]; then export SINGULARITY_BINDPATH=; fi; \
if [ -z \"$SINGULARITY_CACHEDIR\" ]; then export SINGULARITY_CACHEDIR=; fi;

ITER=0
until [ $ITER -ge 3 ]; do
    sbatch \
        --export=ALL \
        -J cromwell_35e483c9_read_genome_tsv \
        -D /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-read_genome_tsv \
        -o /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-read_genome_tsv/execution/stdout \
        -e /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-read_genome_tsv/execution/stderr \
        -t 60 \
        -n 1 \
        --ntasks-per-node=1 \
        --cpus-per-task=1 \
        --mem=2048 \
         \
         \
         \
         \
        --wrap "/bin/bash /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-read_genome_tsv/execution/script" \
        && break
    ITER=$[$ITER+1]
    sleep 30
done
2021-10-20 20:23:15,229 cromwell-system-akka.dispatchers.backend-dispatcher-69 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.read_genome_tsv:NA:1]: job id: 36047
2021-10-20 20:23:15,233 cromwell-system-akka.dispatchers.backend-dispatcher-69 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.read_genome_tsv:NA:1]: Status change from - to WaitingForReturnCode
2021-10-20 20:35:31,329 cromwell-system-akka.dispatchers.backend-dispatcher-71 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.read_genome_tsv:NA:1]: Status change from WaitingForReturnCode to Done
2021-10-20 20:36:45,816 cromwell-system-akka.dispatchers.engine-dispatcher-8 INFO  - WorkflowExecutionActor-35e483c9-d59f-4446-ae7c-ce049b969e9a [UUID(35e483c9)]: Starting atac.align
2021-10-20 20:36:46,914 cromwell-system-akka.dispatchers.engine-dispatcher-33 INFO  - Assigned new job execution tokens to the following groups: 35e483c9: 1
2021-10-20 20:36:48,876 cromwell-system-akka.dispatchers.engine-dispatcher-48 INFO  - WorkflowExecutionActor-35e483c9-d59f-4446-ae7c-ce049b969e9a [UUID(35e483c9)]: Starting atac.align_mito
2021-10-20 20:36:48,915 cromwell-system-akka.dispatchers.engine-dispatcher-36 INFO  - Assigned new job execution tokens to the following groups: 35e483c9: 1
2021-10-20 20:36:52,423 cromwell-system-akka.dispatchers.engine-dispatcher-34 INFO  - 35e483c9-d59f-4446-ae7c-ce049b969e9a-EngineJobExecutionActor-atac.align_mito:0:1 [UUID(35e483c9)]: Could not copy a suitable cache hit for 35e483c9:atac.align_mito:0:1. No copy attempts were made.
2021-10-20 20:36:52,457 cromwell-system-akka.dispatchers.backend-dispatcher-84 WARN  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align_mito:0:1]: Unrecognized runtime attribute keys: preemptible, disks
2021-10-20 20:36:53,265 cromwell-system-akka.dispatchers.engine-dispatcher-36 INFO  - 35e483c9-d59f-4446-ae7c-ce049b969e9a-EngineJobExecutionActor-atac.align:0:1 [UUID(35e483c9)]: Could not copy a suitable cache hit for 35e483c9:atac.align:0:1. No copy attempts were made.
2021-10-20 20:36:53,285 cromwell-system-akka.dispatchers.backend-dispatcher-86 WARN  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align:0:1]: Unrecognized runtime attribute keys: preemptible, disks
2021-10-20 20:38:26,792 cromwell-system-akka.dispatchers.backend-dispatcher-84 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align_mito:0:1]: `set -e

# check if pipeline dependencies can be found
if [[ -z "$(which encode_task_trim_adapter.py 2> /dev/null || true)" ]]
then
  echo -e "\n* Error: pipeline dependencies not found." 1>&2
  echo 'Conda users: Did you activate Conda environment (conda activate encode-atac-seq-pipeline)?' 1>&2
  echo '    Or did you install Conda and environment correctly (bash scripts/install_conda_env.sh)?' 1>&2
  echo 'GCP/AWS/Docker users: Did you add --docker flag to Caper command line arg?' 1>&2
  echo 'Singularity users: Did you add --singularity flag to Caper command line arg?' 1>&2
  echo -e "\n" 1>&2
  exit 3
fi

# trim adapter
python3 $(which encode_task_trim_adapter.py) \
    /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/execution/write_tsv_5eabc7f92cc4ca00bb32026044b1e1aa.tmp \
     \
    --adapters /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/execution/write_tsv_d41d8cd98f00b204e9800998ecf8427e.tmp \
    --paired-end \
    --auto-detect-adapter \
    --cutadapt-param ' -e 0.1 -m 5' \
    --nth 6

# align on trimmed/merged fastqs
if [ 'bowtie2' == 'bowtie2' ]; then
    python3 $(which encode_task_bowtie2.py) \
        /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/inputs/-73807171/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bowtie2_index.tar.gz \
        R1/*.fastq.gz \
        R2/*.fastq.gz \
        --paired-end \
        --multimapping 4 \
        --mem-gb 4.752078142911196 \
        --nth 6
fi

python3 $(which encode_task_post_align.py) \
    R1/*.fastq.gz $(ls *.bam) \
    --mito-chr-name chrM \
    --chrsz /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/inputs/1263107085/GRCh38_EBV.chrom.sizes.tsv \
    --mem-gb 4.752078142911196 \
    --nth 6
rm -rf R1 R2`
2021-10-20 20:38:30,821 cromwell-system-akka.dispatchers.backend-dispatcher-84 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align_mito:0:1]: executing: if [ -z \"$SINGULARITY_BINDPATH\" ]; then export SINGULARITY_BINDPATH=; fi; \
if [ -z \"$SINGULARITY_CACHEDIR\" ]; then export SINGULARITY_CACHEDIR=; fi;

ITER=0
until [ $ITER -ge 3 ]; do
    sbatch \
        --export=ALL \
        -J cromwell_35e483c9_align_mito \
        -D /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0 \
        -o /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/execution/stdout \
        -e /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/execution/stderr \
        -t 2880 \
        -n 1 \
        --ntasks-per-node=1 \
        --cpus-per-task=6 \
        --mem=6082 \
         \
         \
         \
         \
        --wrap "/bin/bash /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/execution/script" \
        && break
    ITER=$[$ITER+1]
    sleep 30
done
2021-10-20 20:38:41,102 cromwell-system-akka.dispatchers.backend-dispatcher-86 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align:0:1]: `set -e

# check if pipeline dependencies can be found
if [[ -z "$(which encode_task_trim_adapter.py 2> /dev/null || true)" ]]
then
  echo -e "\n* Error: pipeline dependencies not found." 1>&2
  echo 'Conda users: Did you activate Conda environment (conda activate encode-atac-seq-pipeline)?' 1>&2
  echo '    Or did you install Conda and environment correctly (bash scripts/install_conda_env.sh)?' 1>&2
  echo 'GCP/AWS/Docker users: Did you add --docker flag to Caper command line arg?' 1>&2
  echo 'Singularity users: Did you add --singularity flag to Caper command line arg?' 1>&2
  echo -e "\n" 1>&2
  exit 3
fi

# trim adapter
python3 $(which encode_task_trim_adapter.py) \
    /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/execution/write_tsv_9d5bc0e20f8dd68bb4cbfcfcadb9164b.tmp \
     \
    --adapters /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/execution/write_tsv_d41d8cd98f00b204e9800998ecf8427e.tmp \
    --paired-end \
    --auto-detect-adapter \
    --cutadapt-param ' -e 0.1 -m 5' \
    --nth 6

# align on trimmed/merged fastqs
if [ 'bowtie2' == 'bowtie2' ]; then
    python3 $(which encode_task_bowtie2.py) \
        /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/inputs/-73807171/ENCFF110MCL.tar.gz \
        R1/*.fastq.gz \
        R2/*.fastq.gz \
        --paired-end \
        --multimapping 4 \
        --mem-gb 7.5454027318954475 \
        --nth 6
fi

python3 $(which encode_task_post_align.py) \
    R1/*.fastq.gz $(ls *.bam) \
    --mito-chr-name chrM \
    --chrsz /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/inputs/1263107085/GRCh38_EBV.chrom.sizes.tsv \
    --mem-gb 7.5454027318954475 \
    --nth 6
rm -rf R1 R2`
2021-10-20 20:38:44,539 cromwell-system-akka.dispatchers.backend-dispatcher-86 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align:0:1]: executing: if [ -z \"$SINGULARITY_BINDPATH\" ]; then export SINGULARITY_BINDPATH=; fi; \
if [ -z \"$SINGULARITY_CACHEDIR\" ]; then export SINGULARITY_CACHEDIR=; fi;

ITER=0
until [ $ITER -ge 3 ]; do
    sbatch \
        --export=ALL \
        -J cromwell_35e483c9_align \
        -D /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0 \
        -o /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/execution/stdout \
        -e /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/execution/stderr \
        -t 2880 \
        -n 1 \
        --ntasks-per-node=1 \
        --cpus-per-task=6 \
        --mem=9658 \
         \
         \
         \
         \
        --wrap "/bin/bash /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/execution/script" \
        && break
    ITER=$[$ITER+1]
    sleep 30
done
2021-10-20 20:38:55,214 cromwell-system-akka.dispatchers.backend-dispatcher-91 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align_mito:0:1]: job id: 37943
2021-10-20 20:38:55,226 cromwell-system-akka.dispatchers.backend-dispatcher-90 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align_mito:0:1]: Status change from - to WaitingForReturnCode
2021-10-20 20:39:05,214 cromwell-system-akka.dispatchers.backend-dispatcher-86 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align:0:1]: job id: 38032
2021-10-20 20:39:05,218 cromwell-system-akka.dispatchers.backend-dispatcher-89 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align:0:1]: Status change from - to WaitingForReturnCode
2021-10-20 20:57:34,414 cromwell-system-akka.dispatchers.backend-dispatcher-89 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align_mito:0:1]: Status change from WaitingForReturnCode to Done
2021-10-20 20:57:45,237 cromwell-system-akka.dispatchers.engine-dispatcher-56 INFO  - WorkflowExecutionActor-35e483c9-d59f-4446-ae7c-ce049b969e9a [UUID(35e483c9)]: Retrying job execution for atac.align_mito:0:2
2021-10-20 20:57:45,770 cromwell-system-akka.dispatchers.engine-dispatcher-56 INFO  - WorkflowExecutionActor-35e483c9-d59f-4446-ae7c-ce049b969e9a [UUID(35e483c9)]: Starting atac.align_mito
2021-10-20 20:57:46,915 cromwell-system-akka.dispatchers.engine-dispatcher-49 INFO  - Assigned new job execution tokens to the following groups: 35e483c9: 1
2021-10-20 20:57:53,141 cromwell-system-akka.dispatchers.backend-dispatcher-86 WARN  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align_mito:0:2]: Unrecognized runtime attribute keys: preemptible, disks
2021-10-20 20:58:02,659 cromwell-system-akka.dispatchers.backend-dispatcher-88 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align:0:1]: Status change from WaitingForReturnCode to Done
2021-10-20 20:58:05,217 cromwell-system-akka.dispatchers.engine-dispatcher-55 INFO  - WorkflowExecutionActor-35e483c9-d59f-4446-ae7c-ce049b969e9a [UUID(35e483c9)]: Retrying job execution for atac.align:0:2
2021-10-20 20:58:06,125 cromwell-system-akka.dispatchers.engine-dispatcher-36 INFO  - WorkflowExecutionActor-35e483c9-d59f-4446-ae7c-ce049b969e9a [UUID(35e483c9)]: Starting atac.align
2021-10-20 20:58:06,914 cromwell-system-akka.dispatchers.engine-dispatcher-45 INFO  - Assigned new job execution tokens to the following groups: 35e483c9: 1
2021-10-20 20:58:12,185 cromwell-system-akka.dispatchers.backend-dispatcher-83 WARN  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align:0:2]: Unrecognized runtime attribute keys: preemptible, disks
2021-10-20 21:00:10,661 cromwell-system-akka.dispatchers.backend-dispatcher-86 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align_mito:0:2]: `set -e

# check if pipeline dependencies can be found
if [[ -z "$(which encode_task_trim_adapter.py 2> /dev/null || true)" ]]
then
  echo -e "\n* Error: pipeline dependencies not found." 1>&2
  echo 'Conda users: Did you activate Conda environment (conda activate encode-atac-seq-pipeline)?' 1>&2
  echo '    Or did you install Conda and environment correctly (bash scripts/install_conda_env.sh)?' 1>&2
  echo 'GCP/AWS/Docker users: Did you add --docker flag to Caper command line arg?' 1>&2
  echo 'Singularity users: Did you add --singularity flag to Caper command line arg?' 1>&2
  echo -e "\n" 1>&2
  exit 3
fi

# trim adapter
python3 $(which encode_task_trim_adapter.py) \
    /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/attempt-2/execution/write_tsv_96efcb021d4de43902179370cdbc442b.tmp \
     \
    --adapters /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/attempt-2/execution/write_tsv_d41d8cd98f00b204e9800998ecf8427e.tmp \
    --paired-end \
    --auto-detect-adapter \
    --cutadapt-param ' -e 0.1 -m 5' \
    --nth 6

# align on trimmed/merged fastqs
if [ 'bowtie2' == 'bowtie2' ]; then
    python3 $(which encode_task_bowtie2.py) \
        /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/attempt-2/inputs/-73807171/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only_bowtie2_index.tar.gz \
        R1/*.fastq.gz \
        R2/*.fastq.gz \
        --paired-end \
        --multimapping 4 \
        --mem-gb 4.752078142911196 \
        --nth 6
fi

python3 $(which encode_task_post_align.py) \
    R1/*.fastq.gz $(ls *.bam) \
    --mito-chr-name chrM \
    --chrsz /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/attempt-2/inputs/1263107085/GRCh38_EBV.chrom.sizes.tsv \
    --mem-gb 4.752078142911196 \
    --nth 6
rm -rf R1 R2`
2021-10-20 21:00:12,447 cromwell-system-akka.dispatchers.backend-dispatcher-86 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align_mito:0:2]: executing: if [ -z \"$SINGULARITY_BINDPATH\" ]; then export SINGULARITY_BINDPATH=; fi; \
if [ -z \"$SINGULARITY_CACHEDIR\" ]; then export SINGULARITY_CACHEDIR=; fi;

ITER=0
until [ $ITER -ge 3 ]; do
    sbatch \
        --export=ALL \
        -J cromwell_35e483c9_align_mito \
        -D /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/attempt-2 \
        -o /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/attempt-2/execution/stdout \
        -e /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/attempt-2/execution/stderr \
        -t 2880 \
        -n 1 \
        --ntasks-per-node=1 \
        --cpus-per-task=6 \
        --mem=6082 \
         \
         \
         \
         \
        --wrap "/bin/bash /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/attempt-2/execution/script" \
        && break
    ITER=$[$ITER+1]
    sleep 30
done
2021-10-20 21:00:18,606 cromwell-system-akka.dispatchers.backend-dispatcher-83 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align:0:2]: `set -e

# check if pipeline dependencies can be found
if [[ -z "$(which encode_task_trim_adapter.py 2> /dev/null || true)" ]]
then
  echo -e "\n* Error: pipeline dependencies not found." 1>&2
  echo 'Conda users: Did you activate Conda environment (conda activate encode-atac-seq-pipeline)?' 1>&2
  echo '    Or did you install Conda and environment correctly (bash scripts/install_conda_env.sh)?' 1>&2
  echo 'GCP/AWS/Docker users: Did you add --docker flag to Caper command line arg?' 1>&2
  echo 'Singularity users: Did you add --singularity flag to Caper command line arg?' 1>&2
  echo -e "\n" 1>&2
  exit 3
fi

# trim adapter
python3 $(which encode_task_trim_adapter.py) \
    /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/attempt-2/execution/write_tsv_e73b2b8487231e25b6caf0af8a824c6d.tmp \
     \
    --adapters /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/attempt-2/execution/write_tsv_d41d8cd98f00b204e9800998ecf8427e.tmp \
    --paired-end \
    --auto-detect-adapter \
    --cutadapt-param ' -e 0.1 -m 5' \
    --nth 6

# align on trimmed/merged fastqs
if [ 'bowtie2' == 'bowtie2' ]; then
    python3 $(which encode_task_bowtie2.py) \
        /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/attempt-2/inputs/-73807171/ENCFF110MCL.tar.gz \
        R1/*.fastq.gz \
        R2/*.fastq.gz \
        --paired-end \
        --multimapping 4 \
        --mem-gb 7.5454027318954475 \
        --nth 6
fi

python3 $(which encode_task_post_align.py) \
    R1/*.fastq.gz $(ls *.bam) \
    --mito-chr-name chrM \
    --chrsz /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/attempt-2/inputs/1263107085/GRCh38_EBV.chrom.sizes.tsv \
    --mem-gb 7.5454027318954475 \
    --nth 6
rm -rf R1 R2`
2021-10-20 21:00:23,260 cromwell-system-akka.dispatchers.backend-dispatcher-83 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align:0:2]: executing: if [ -z \"$SINGULARITY_BINDPATH\" ]; then export SINGULARITY_BINDPATH=; fi; \
if [ -z \"$SINGULARITY_CACHEDIR\" ]; then export SINGULARITY_CACHEDIR=; fi;

ITER=0
until [ $ITER -ge 3 ]; do
    sbatch \
        --export=ALL \
        -J cromwell_35e483c9_align \
        -D /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/attempt-2 \
        -o /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/attempt-2/execution/stdout \
        -e /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/attempt-2/execution/stderr \
        -t 2880 \
        -n 1 \
        --ntasks-per-node=1 \
        --cpus-per-task=6 \
        --mem=9658 \
         \
         \
         \
         \
        --wrap "/bin/bash /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/attempt-2/execution/script" \
        && break
    ITER=$[$ITER+1]
    sleep 30
done
2021-10-20 21:00:40,214 cromwell-system-akka.dispatchers.backend-dispatcher-102 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align_mito:0:2]: job id: 40999
2021-10-20 21:00:40,236 cromwell-system-akka.dispatchers.backend-dispatcher-101 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align_mito:0:2]: Status change from - to WaitingForReturnCode
2021-10-20 21:00:45,224 cromwell-system-akka.dispatchers.backend-dispatcher-102 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align:0:2]: job id: 41006
2021-10-20 21:00:45,230 cromwell-system-akka.dispatchers.backend-dispatcher-83 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align:0:2]: Status change from - to WaitingForReturnCode
2021-10-20 21:42:34,068 cromwell-system-akka.dispatchers.backend-dispatcher-88 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align_mito:0:2]: Status change from WaitingForReturnCode to Done
2021-10-20 21:47:07,880 cromwell-system-akka.dispatchers.backend-dispatcher-103 INFO  - BackgroundConfigAsyncJobExecutionActor [UUID(35e483c9)atac.align:0:2]: Status change from WaitingForReturnCode to Done
2021-10-20 21:47:18,993 cromwell-system-akka.dispatchers.engine-dispatcher-37 INFO  - WorkflowManagerActor: Workflow 35e483c9-d59f-4446-ae7c-ce049b969e9a failed (during ExecutingWorkflowState): Job atac.align_mito:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
Check the content of stderr for potential additional information: /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/attempt-2/execution/stderr.
 [First 3000 bytes]:Traceback (most recent call last):
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 192, in <module>
    main()
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 169, in main
    args.out_dir)
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 102, in bowtie2_pe
    tmp_bam=tmp_bam,
  File "/mainfs/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_lib_common.py", line 359, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=44855, PGID=44855, RC=127, DURATION_SEC=150.6
STDERR=/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/bowtie2-align-s: error while loading shared libraries: libtbb.so.2: cannot open shared object file: No such file or directory
(ERR): Description of arguments failed!
Exiting now ...
STDOUT=
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/attempt-2/execution/*.samstats.qc': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/attempt-2/execution/*.read_length.txt': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/attempt-2/execution/*.bai': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align_mito/shard-0/attempt-2/execution/non_mito/*.samstats.qc': No such file or directory

Job atac.align:0:2 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
Check the content of stderr for potential additional information: /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/attempt-2/execution/stderr.
 [First 3000 bytes]:Traceback (most recent call last):
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 192, in <module>
    main()
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 169, in main
    args.out_dir)
  File "/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_task_bowtie2.py", line 102, in bowtie2_pe
    tmp_bam=tmp_bam,
  File "/mainfs/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/encode_lib_common.py", line 359, in run_shell_cmd
    raise Exception(err_str)
Exception: PID=44937, PGID=44937, RC=127, DURATION_SEC=111.2
STDERR=/scratch/lem2g17/miniconda3/envs/encode-atac-seq-pipeline/bin/bowtie2-align-s: error while loading shared libraries: libtbb.so.2: cannot open shared object file: No such file or directory
(ERR): Description of arguments failed!
Exiting now ...
STDOUT=
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/attempt-2/execution/*.samstats.qc': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/attempt-2/execution/*.read_length.txt': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/attempt-2/execution/*.bai': No such file or directory
ln: failed to access '/mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/atac/35e483c9-d59f-4446-ae7c-ce049b969e9a/call-align/shard-0/attempt-2/execution/non_mito/*.samstats.qc': No such file or directory

2021-10-20 21:47:18,993 cromwell-system-akka.dispatchers.engine-dispatcher-37 INFO  - WorkflowManagerActor: Workflow actor for 35e483c9-d59f-4446-ae7c-ce049b969e9a completed with status 'Failed'. The workflow will be removed from the workflow store.
2021-10-20 21:47:44,198 cromwell-system-akka.dispatchers.engine-dispatcher-41 INFO  - SingleWorkflowRunnerActor workflow finished with status 'Failed'.
2021-10-20 21:47:49,189 cromwell-system-akka.dispatchers.engine-dispatcher-35 INFO  - SingleWorkflowRunnerActor writing metadata to /mainfs/scratch/lem2g17/atac-seq-pipeline/S01EDE71.ERX1322302.ATAC-seq.dedup.bwa.GRCh38.20160301/.caper_tmp/atac/20211020_201501_386628/metadata.json
2021-10-20 21:47:55,938  INFO  - Workflow polling stopped
2021-10-20 21:47:55,983  INFO  - 0 workflows released by cromid-1af6c31
2021-10-20 21:47:55,994  INFO  - Shutting down WorkflowStoreActor - Timeout = 5 seconds
2021-10-20 21:47:55,996  INFO  - Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
2021-10-20 21:47:56,041 cromwell-system-akka.dispatchers.engine-dispatcher-8 INFO  - Aborting all running workflows.
2021-10-20 21:47:56,044  INFO  - WorkflowStoreActor stopped
2021-10-20 21:47:56,047 cromwell-system-akka.dispatchers.engine-dispatcher-35 WARN  - SingleWorkflowRunnerActor: received unexpected message: QueueWeight(0) of type cromwell.core.actor.BatchActor.QueueWeight in state RequestingMetadata
2021-10-20 21:47:56,053  INFO  - Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
2021-10-20 21:47:56,054  INFO  - WorkflowLogCopyRouter stopped
2021-10-20 21:47:56,054  INFO  - JobExecutionTokenDispenser stopped
2021-10-20 21:47:56,054  INFO  - Shutting down WorkflowManagerActor - Timeout = 3600 seconds
2021-10-20 21:47:56,056 cromwell-system-akka.dispatchers.engine-dispatcher-56 INFO  - WorkflowManagerActor: All workflows finished
2021-10-20 21:47:56,056  INFO  - WorkflowManagerActor stopped
2021-10-20 21:47:56,086 cromwell-system-akka.dispatchers.engine-dispatcher-35 WARN  - SingleWorkflowRunnerActor: received unexpected message: QueueWeight(0) of type cromwell.core.actor.BatchActor.QueueWeight in state RequestingMetadata
2021-10-20 21:47:59,792  INFO  - Connection pools shut down
2021-10-20 21:47:59,799  INFO  - Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
2021-10-20 21:47:59,800  INFO  - Shutting down JobStoreActor - Timeout = 1800 seconds
2021-10-20 21:47:59,802  INFO  - Shutting down CallCacheWriteActor - Timeout = 1800 seconds
2021-10-20 21:47:59,802  INFO  - CallCacheWriteActor Shutting down: 0 queued messages to process
2021-10-20 21:47:59,803  INFO  - Shutting down ServiceRegistryActor - Timeout = 1800 seconds
2021-10-20 21:47:59,803  INFO  - Shutting down DockerHashActor - Timeout = 1800 seconds
2021-10-20 21:47:59,809  INFO  - Shutting down IoProxy - Timeout = 1800 seconds
2021-10-20 21:47:59,817  INFO  - SubWorkflowStoreActor stopped
2021-10-20 21:47:59,817  INFO  - JobStoreActor stopped
2021-10-20 21:47:59,817  INFO  - CallCacheWriteActor stopped
2021-10-20 21:47:59,817  INFO  - IoProxy stopped
2021-10-20 21:47:59,832  INFO  - WriteMetadataActor Shutting down: 0 queued messages to process
2021-10-20 21:47:59,835  INFO  - KvWriteActor Shutting down: 0 queued messages to process
2021-10-20 21:47:59,842  INFO  - ServiceRegistryActor stopped
2021-10-20 21:47:59,878  INFO  - DockerHashActor stopped
2021-10-20 21:47:59,996  INFO  - Database closed
2021-10-20 21:47:59,996  INFO  - Stream materializer shut down
2021-10-20 21:48:00,016  INFO  - WDL HTTP import resolver closed
2021-10-20 21:48:00,238  INFO  - Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
2021-10-20 21:48:00,239  INFO  - Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
2021-10-20 21:48:00,239  INFO  - Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
2021-10-20 21:48:00,239  INFO  - Shutting down connection pool: curAllocated=0 idleQueues.size=0 waitQueue.size=0 maxWaitQueueLimit=256 closed=false
leepc12 commented 2 years ago

That's a known dependency conflict issue of Conda + mamba. I will release new Caper + atac-seq-pipeline soon by end of this week. That will fix all the problems. Until then please try to install pipeline's Conda environment without mamba.

LaraBM commented 2 years ago

Hi,

I didn't use mamba to install the environment though. I just installed dependencies directly I will try the new release and see if that works now.

Many thanks for your help.

Kind regards, Lara

leepc12 commented 2 years ago

Yes, please try with the latest pipeline + latest caper. Both>=2.0.0. Then re-install pipeline's Conda environments.

$ bash scripts/uninstall_conda_env.sh
$ bash scripts/install_conda_env.sh

Also, don't activate pipeline's Conda environment before running pipelines.