search

ENCODE-DCC / chip-seq-pipeline2

ENCODE ChIP-seq pipeline
MIT License
241 stars 123 forks source link

GCP Caper Server Failed & Stopped #190

Closed kirbyziegler closed 2 years ago

kirbyziegler commented 4 years ago

Even though it was started in screen mode, the caper server I was using lost it's heartbeat and stopped running after about 20 hours with the following error message: Failed to retrieve metadata from Cromwell server

Is it possible to resume the failed GCP workflows that were affected by this?

gcp-out-dir=gs://bucket gcp-call-caching-dup-strat=reference local-loc-dir=/home/user/.caper_tmp

kirbyziegler commented 3 years ago

(encode-chip-seq-pipeline) kz@aquas-chip-pipeline-2:~$ caper list 2020-10-05 00:45:17,799|caper.server_heartbeat|ERROR| Found a heartbeat file but it has been expired (> timeout). ~/.caper/default_server_heartbeat Traceback (most recent call last): File "/home/kz/miniconda3/envs/encode-chip-seq-pipeline/bin/caper", line 13, in main() File "/home/kz/miniconda3/envs/encode-chip-seq-pipeline/lib/python3.7/site-packages/caper/cli.py", line 504, in main client(parsed_args) File "/home/kz/miniconda3/envs/encode-chip-seq-pipeline/lib/python3.7/site-packages/caper/cli.py", line 265, in client subcmd_list(c, args) File "/home/kz/miniconda3/envs/encode-chip-seq-pipeline/lib/python3.7/site-packages/caper/cli.py", line 390, in subcmd_list workflows = caper_client.list(args.wf_id_or_label) File "/home/kz/miniconda3/envs/encode-chip-seq-pipeline/lib/python3.7/site-packages/caper/caper_client.py", line 110, in list return self._cromwell_rest_api.find(workflow_ids, labels) File "/home/kz/miniconda3/envs/encode-chip-seq-pipeline/lib/python3.7/site-packages/caper/cromwell_rest_api.py", line 226, in find CromwellRestAPI.ENDPOINT_WORKFLOWS, params=CromwellRestAPI.PARAMS_WORKFLOWS File "/home/kz/miniconda3/envs/encode-chip-seq-pipeline/lib/python3.7/site-packages/caper/cromwell_rest_api.py", line 299, in __request_get ) from None Exception: Failed to connect to Cromwell server. req=GET, url=http://localhost:8000/api/workflows/v1/query

leepc12 commented 3 years ago

Check if Caper server is running first. Run caper without screen and check its message. Also check Cromwell's output (cromwell.out).

$ cd /opt/caper
$ caper server

Please define correct output bucket path. In your Caper conf.

gcp-out-dir=gs://bucket-name/bucket-dir-path
leepc12 commented 2 years ago

Closing this due to inactivity.