Cromwell fails to find the docker image when the digest is none (<none>)

[nh13]

I am trying to run Cromwell with docker images that were loaded with docker load. This means that the digests are unavailable (i.e. <none>). Unforutnately, this means that when looking up the image locally (i.e. when the config docker.hash-lookup.method="local" is used), the image is not found. The offending lines of code are: https://github.com/broadinstitute/cromwell/blob/1898d8103a06d160dc721d464862313e78ee7a2c/dockerHashing/src/main/scala/cromwell/docker/local/DockerCliClient.scala#L26 https://github.com/broadinstitute/cromwell/blob/1898d8103a06d160dc721d464862313e78ee7a2c/dockerHashing/src/main/scala/cromwell/docker/local/DockerCliClient.scala#L78-L92

Can we instead use the image ID instead of the digest when using local images?

log output

``` [INFO] [09/16/2019 11:07:14.821] [cromwell-system-akka.dispatchers.engine-dispatcher-40] [akka://cromwell-system/user/SingleWorkflowRunnerActor/JobExecutionTokenDispenser] Not triggering log of token queue status. Effective log interval = None [INFO] [09/16/2019 11:07:14.830] [cromwell-system-akka.dispatchers.engine-dispatcher-76] [akka://cromwell-system/user/SingleWorkflowRunnerActor/JobExecutionTokenDispenser] Assigned new job execution tokens to the following groups: 2b766fe6: 1 [2019-09-16 11:07:16,20] [error] Docker pull failed java.lang.RuntimeException: Error running: docker pull Exit code: 1 Error response from daemon: pull access denied for repository does not exist or may require 'docker login': denied: requested access to the resource is denied at cromwell.docker.local.DockerCliClient.$anonfun$forRun$1(DockerCliClient.scala:58) at scala.util.Try$.apply(Try.scala:213) at cromwell.docker.local.DockerCliClient.forRun(DockerCliClient.scala:50) at cromwell.docker.local.DockerCliClient.pull(DockerCliClient.scala:37) at cromwell.docker.local.DockerCliClient.pull$(DockerCliClient.scala:36) at cromwell.docker.local.DockerCliClient$.pull(DockerCliClient.scala:94) at cromwell.docker.local.DockerCliFlow$.pull(DockerCliFlow.scala:101) at cromwell.docker.local.DockerCliFlow.$anonfun$run$1(DockerCliFlow.scala:35) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:139) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:351) at cats.effect.internals.IORunLoop$RestartCallback.run(IORunLoop.scala:362) at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:70) at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:36) at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:93) at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:93) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85) at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:93) at cats.effect.internals.Trampoline.execute(Trampoline.scala:43) at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:44) at cats.effect.internals.ForwardCancelable.loop$1(ForwardCancelable.scala:46) at cats.effect.internals.ForwardCancelable.$anonfun$cancel$1(ForwardCancelable.scala:52) at cats.effect.internals.ForwardCancelable.$anonfun$cancel$1$adapted(ForwardCancelable.scala:52) at cats.effect.internals.IORunLoop$RestartCallback.start(IORunLoop.scala:337) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:119) at cats.effect.internals.IORunLoop$.start(IORunLoop.scala:34) at cats.effect.IO.unsafeRunAsync(IO.scala:258) at cats.effect.internals.IORace$.onSuccess$1(IORace.scala:40) at cats.effect.internals.IORace$.$anonfun$simple$4(IORace.scala:79) at cats.effect.internals.IORace$.$anonfun$simple$4$adapted(IORace.scala:77) at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:136) at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:351) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:372) at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:312) at cats.effect.internals.IOShift$Tick.run(IOShift.scala:36) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ```

A workaround is setting up a registry to host the images (so we can have digests), but I ask anyway.

[illusional]

Hi @nh13, not from Broad but have you tried turning the docker-digest lookup off with the following in your config:

docker.hash-lookup.enabled = false

[asalimih]

I'm having the same issue, is there any solution other than docker.hash-lookup.enabled = false because it may cause problems for call caching

[gr8ape]

I am also encountering this issue and have yet to find resolution.

[aednichols]

I think call caching can't work on images without a hash/digest anyway, since the hash is taken into account to evaluable caching eligibility.