broadinstitute / cromwell

Scientific workflow engine designed for simplicity & scalability. Trivially transition between one off use cases to massive scale production environments
http://cromwell.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
999 stars 361 forks source link

docker tags that include sha256 hash don't work with call caching #4912

Open notestaff opened 5 years ago

notestaff commented 5 years ago

When the 'docker' runtime attribute of a task is specified with the sha256 hash, as in runtime { docker: "quay.io/broadinstitute/viral-ngs:1.22.0@sha256:be458fe91e176102c9c8e2933c9ff430d899a0ec8f919939cbdb26f2d4c7d4be" } Cromwell reports exceptions such as 2019-04-28 04:40:08,146 cromwell-system-akka.dispatchers.engine-dispatcher-39 WARN - BackendPreparationActor_for_d5800039:assemble_de\ novo_with_deplete.deplete_taxa:-1:1 [UUID(d5800039)]: Docker lookup failed java.lang.Exception: Docker image quay.io/broadinstitute/viral-ngs@sha256:be458fe91e176102c9c8e2933c9ff430d899a0ec8f919939cbdb26f2d4c7\ d4be not found at cromwell.engine.workflow.WorkflowDockerLookupActor.cromwell$engine$workflow$WorkflowDockerLookupActor$$handleLookupFailure(\ WorkflowDockerLookupActor.scala:194) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:86) at cromwell.engine.workflow.WorkflowDockerLookupActor$$anonfun$3.applyOrElse(WorkflowDockerLookupActor.scala:70) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34) at akka.actor.FSM.processEvent(FSM.scala:684) at akka.actor.FSM.processEvent$(FSM.scala:681) at cromwell.engine.workflow.WorkflowDockerLookupActor.akka$actor$LoggingFSM$$super$processEvent(WorkflowDockerLookupActor.scal\ a:37) at akka.actor.LoggingFSM.processEvent(FSM.scala:820) at akka.actor.LoggingFSM.processEvent$(FSM.scala:802) at cromwell.engine.workflow.WorkflowDockerLookupActor.processEvent(WorkflowDockerLookupActor.scala:37) at akka.actor.FSM.akka$actor$FSM$$processMsg(FSM.scala:678) at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:672) at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:34) at cromwell.docker.DockerClientHelper$$anonfun$dockerResponseReceive$1.applyOrElse(DockerClientHelper.scala:16) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)

and disables call caching.

notestaff commented 5 years ago

P.S. the current docs advise including the sha256 hash in the docker runtime spec: https://cromwell.readthedocs.io/en/stable/tutorials/Containers/#image-versions Which makes sense, but unfortunately right now it seems to disable caching (at least with the Local backend).