search

StanfordBioinformatics / trellis-mvp-functions

Trellis serverless data management framework for variant calling of VA MVP whole-genome sequencing data.
6 stars 1 forks source link

Cromwell docker pull failed #7

Closed pbilling closed 4 years ago

pbilling commented 4 years ago

From log file:

[2020-01-09 08:55:32,01] [info] WorkflowManagerActor Workflow 6a278540-661c-4330-be81-93c3ebc12e51 failed (during ExecutingWorkflowState): java.lang.Exception: Task germline_single_sample_workflow.MarkDuplicates:NA:2 failed. The job was stopped before the command finished. PAPI error code 2. Execution failed: pulling image: docker pull: generic::unknown: retry budget exhausted (10 attempts): running ["docker" "pull" "google/cloud-sdk:251.0.0-slim"]: exit status 1 (standard error: "Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n")

Seems to be related to this issue: https://github.com/broadinstitute/cromwell/issues/4640

This is where the docker image seems to be defined. I think this is the not image to run the job, but instead to run data transfer operations: https://github.com/broadinstitute/cromwell/blob/6108b0e261386fbbfc8300b11eab4520616ab455/supportedBackends/google/pipelines/v2alpha1/src/main/scala/cromwell/backend/google/pipelines/v2alpha1/GenomicsFactory.scala

pbilling commented 4 years ago

Switching from cromwell:46 to cromwell:47 image since that one specifies to use the gcr.io image instead of the Dockerhub one.

pbilling commented 4 years ago

Spot checking of log files indicates this has resolved the issue.