Open icen234 opened 1 year ago
Please specify an environment to be used for the pipeline (--singularity
, --docker
or --conda
). They are flags. We recommend to use singularity on HPCs and docker on PCs.
# PCs
$ caper run WDL -i input.json --docker
# HPCs: make sure to configure Caper's config for HPCs first. See README
$ caper hpc submit WDL -i input.json --singularity
Thanks for the fast reply! I specify --conda. This is the command I use: $ caper run atac.wdl -i "/staging/leuven/stg_00072/Ionas/test.json" --conda
I am using PBS/Torque. When I set pbs as the caper backend, I get a different error:
2023-02-20 13:49:59,212|caper.cromwell|INFO| Validating WDL/inputs/imports with Womtool...
2023-02-20 13:50:02,505|caper.nb_subproc_thread|INFO| Subprocess finished successfully.
2023-02-20 13:50:02,506|caper.cromwell|INFO| Passed Womtool validation.
2023-02-20 13:50:02,507|caper.caper_runner|INFO| launching run: wdl=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac.wdl, inputs=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/.caper_tmp/staging/leuven/stg_00072/Ionas/test.local.json, backend_conf=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/.caper_tmp/atac/20230220_134931_302733/backend.conf
2023-02-20 13:50:07,545|caper.cromwell_workflow_monitor|INFO| Workflow: id=dac7a59d-f64c-4170-b227-d75de987ff1b, status=Submitted
2023-02-20 13:50:07,580|caper.cromwell_workflow_monitor|INFO| Workflow: id=dac7a59d-f64c-4170-b227-d75de987ff1b, status=Running
2023-02-20 13:51:46,761|caper.cromwell_workflow_monitor|INFO| Workflow: id=dac7a59d-f64c-4170-b227-d75de987ff1b, status=Failed
2023-02-20 13:52:02,925|caper.cromwell_metadata|INFO| Wrote metadata file. /lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/dac7a59d-f64c-4170-b227-d75de987ff1b/metadata.json
2023-02-20 13:52:02,925|caper.cromwell|INFO| Workflow failed. Auto-troubleshooting...
* Started troubleshooting workflow: id=dac7a59d-f64c-4170-b227-d75de987ff1b, status=Failed
* Found failures JSON object.
[
{
"causedBy": [
{
"causedBy": [
{
"message": "Could not find job ID from stdout file.Check the stderr file for possible errors: /lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/dac7a59d-f64c-4170-b227-d75de987ff1b/call-read_genome_tsv/execution/stderr.submit",
"causedBy": []
}
],
"message": "java.lang.RuntimeException: Could not find job ID from stdout file.Check the stderr file for possible errors: /lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/dac7a59d-f64c-4170-b227-d75de987ff1b/call-read_genome_tsv/execution/stderr.submit"
}
],
"message": "Workflow failed"
}
]
* Recursively finding failures in calls (tasks)...
==== NAME=atac.read_genome_tsv, STATUS=Failed, PARENT=
SHARD_IDX=-1, RC=None, JOB_ID=None
START=2023-02-20T12:50:15.666Z, END=2023-02-20T12:51:46.245Z
STDOUT=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/dac7a59d-f64c-4170-b227-d75de987ff1b/call-read_genome_tsv/execution/stdout
STDERR=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/dac7a59d-f64c-4170-b227-d75de987ff1b/call-read_genome_tsv/execution/stderr
2023-02-20 13:52:02,927|caper.nb_subproc_thread|ERROR| Cromwell failed. returncode=1
2023-02-20 13:52:02,928|caper.cli|ERROR| Check stdout in /lustre1/project/stg_00072/Ionas/atac-seq-pipeline/cromwell.out.5
When I ran the pipeline using this command: caper hpc submit test.wdl -i input.json --conda I get an error asking me to define --leader-job-name
When I run it as caper hpc submit test.wdl -i input.json --conda --leader-job-name test1 I get this error:
2023-02-20 13:55:54,260|caper.hpc|INFO| Running shell command: qsub -l walltime=48:00:00,mem=4gb -V -N CAPER_test1 /user/leuven/345/vsc34531/ncbkzi5h.sh
pbs_connect received error code 15008 ('trqauthd unable to authenticate. User or PID of calling process not found or does not match') from trqauthd
Unable to communicate with tier2-p-moab-2.tier2.hpc.kuleuven.be(10.118.224.19)
Unable to authorize user.
qsub: cannot connect to server tier2-p-moab-2.tier2.hpc.kuleuven.be (errno=15008) trqauthd unable to authenticate. User or PID of calling process not found or does not match
Any help? Thanks!
First of all, you need to run caper init YOUR_CLUSTER_TYPE
to initialize Caper's configuration file for your cluster.
backend=local
doesn't work with caper hpc
subcommands.
# for example of PBS
$ caper init pbs # supported cluster type: slurm, sge, pbs, lsf
Please read Caper README carefully to configure it correctly for your cluster type. Also, we don't recommend Conda. Does your cluster support Singularity?
Describe the bug
I have followed the instructions for installation of the pipeline using conda, the installation was successfully completed but when I run it I get the following error:
Started troubleshooting workflow: id=d4caa269-70bb-4cd7-981a-ccfa4cf0090e, status=Failed Found failures JSON object. [ { "causedBy": [ { "message": "Job atac.align:0:2 exited with return code 3 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.", "causedBy": [] }, { "message": "Job atac.align_mito:0:2 exited with return code 3 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.", "causedBy": [] } ], "message": "Workflow failed" } ]
==== NAME=atac.align_mito, STATUS=RetryableFailure, PARENT= SHARD_IDX=0, RC=3, JOB_ID=3741 START=2023-02-16T19:31:48.580Z, END=2023-02-16T19:32:01.499Z STDOUT=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/stdout STDERR=/lustre1/project/stg_00072/Ionas/atac-seq-pipeline/atac/d4caa269-70bb-4cd7-981a-ccfa4cf0090e/call-align_mito/shard-0/execution/stderr STDERR_CONTENTS=
STDERR_BACKGROUND_CONTENTS=
OS/Platform
Caper configuration file
Paste contents of
~/.caper/default.conf
.Input JSON file
Paste contents of your input JSON file.
Troubleshooting result
If you ran
caper run
without Caper server then Caper automatically runs a troubleshooter for failed workflows. Find troubleshooting result in the bottom of Caper's screen log.If you ran
caper submit
with a running Caper server then first find your workflow ID (1st column) withcaper list
and runcaper debug [WORKFLOW_ID]
.Paste troubleshooting result.