Using `FinetunerExecutor` in flow throws `BadImageNameError`

[joschkabraun]

Describe the bug Using FinetunerExecutor in the flow throws jina.excepts.BadImageNameError: image: jinahub/13dzxycc:62d05a93c800e2560cf11ffc can not be found local & remote (code snippet & first screenshot) but one can pull it with jina hub pull jinahub+docker://FinetunerExecutor/v0.9.2 (second screenshot):

from jina import Flow
from docarray import Document, DocumentArray

f = Flow() \
    .add(uses='jinahub+docker://FinetunerExecutor/v0.9.2',
         uses_with={
             'artifact': '012',
             'token': 'abc:def'
         },
         )

with f:
    r = f.post('/index', inputs=DocumentArray([Document(text='hi')]))

Describe how you solve it Waiting some time.

---

Environment

- jina 3.6.6
- docarray 0.13.19
- jina-proto 0.1.12
- protobuf 3.19.4
- proto-backend cpp
- grpcio 1.46.3
- pyyaml 6.0
- python 3.9.12
- platform Darwin
- platform-release 21.5.0
- platform-version Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:22 PDT 2022; root:xnu-8020.121.3~4/RELEASE_X86_64
- architecture x86_64
- processor i386
- uid 190070690681122
- session-id 87124ac8-0429-11ed-a432-acde48001122
- uptime 2022-07-15T12:33:12.058626
- ci-vendor (unset)
* JINA_AUTH_TOKEN (unset)
* JINA_DEFAULT_HOST (unset)
* JINA_DEFAULT_TIMEOUT_CTRL (unset)
* JINA_DEFAULT_WORKSPACE_BASE /Users/joschkabraun/.jina/executor-workspace
* JINA_DEPLOYMENT_NAME (unset)
* JINA_DISABLE_UVLOOP (unset)
* JINA_EARLY_STOP (unset)
* JINA_FULL_CLI (unset)
* JINA_GATEWAY_IMAGE (unset)
* JINA_GRPC_RECV_BYTES (unset)
* JINA_GRPC_SEND_BYTES (unset)
* JINA_HUBBLE_REGISTRY (unset)
* JINA_HUB_CACHE_DIR (unset)
* JINA_HUB_NO_IMAGE_REBUILD (unset)
* JINA_HUB_ROOT (unset)
* JINA_LOG_CONFIG (unset)
* JINA_LOG_LEVEL (unset)
* JINA_LOG_NO_COLOR (unset)
* JINA_MP_START_METHOD (unset)
* JINA_RANDOM_PORT_MAX (unset)
* JINA_RANDOM_PORT_MIN (unset)
* JINA_DISABLE_HEALTHCHECK_LOGS (unset)
* JINA_LOCKS_ROOT (unset)

Screenshots

[delgermurun]

Hi @joschkabraun

Image pulling works fine on my machine with above code snippet. Tried with Jina v3.6.6 and Python v3.9.

Did you try it again recently?

Assume you didn't see docker image pulling progress bar (like below), right?

[mapleeit]

There might be a bug in https://github.com/jina-ai/jina/blob/4bafca88a5d8348c46460e7e37f2991def8c57d4/jina/orchestrate/pods/container.py#L96 In comparison, when run jina hub pull, it’s https://github.com/jina-ai/jina/blob/master/jina/hubble/hubio.py#L804 client.images.get() will only detect local images, right? If the image is not in local, we should use client.images.pull() to pull it from docker hub? But the strange thing is that this should be used frequently, why it doesn’t happen before? :thinking_face:

My previous investigation about this issue. FYI @delgermurun

[joschkabraun]

@delgermurun Yes, I saw that progress bar. The second last one (0.7/11.7 kB) never finishes for me. I can imagine this throwing jina.excepts.BadImageNameError: image: jinahub/13dzxycc:62d05a93c800e2560cf11ffc can not be found local & remote.

[joschkabraun]

Updating docker and restarting the computer multiple times resolved the issue. Thank you so much for your help & time @delgermurun & @mapleeit!

[themantalope]

@joschkabraun

I'm running into the same issue. Which docker version are you using now? I'm currently using:

Docker version 23.0.1, build a5ee5b1

jina version

- jina 3.14.1
- docarray 0.21.0
- jcloud 0.2.2
- jina-hubble-sdk 0.33.2
- jina-proto 0.1.13
- protobuf 4.22.0
- proto-backend upb
- grpcio 1.51.1
- pyyaml 6.0
- python 3.11.0
- platform Linux
- platform-release 5.15.0-67-generic
- platform-version #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023
- architecture x86_64
- processor x86_64
- uid 53447172167855
- session-id dce97d5a-c28c-11ed-881e-309c23b618af
- uptime 2023-03-14T17:22:57.460839
- ci-vendor (unset)
- internal False
* JINA_DEFAULT_HOST (unset)
* JINA_DEFAULT_TIMEOUT_CTRL (unset)
* JINA_DEPLOYMENT_NAME (unset)
* JINA_DISABLE_UVLOOP (unset)
* JINA_EARLY_STOP (unset)
* JINA_FULL_CLI (unset)
* JINA_GATEWAY_IMAGE (unset)
* JINA_GRPC_RECV_BYTES (unset)
* JINA_GRPC_SEND_BYTES (unset)
* JINA_HUB_NO_IMAGE_REBUILD (unset)
* JINA_LOG_CONFIG (unset)
* JINA_LOG_LEVEL (unset)
* JINA_LOG_NO_COLOR (unset)
* JINA_MP_START_METHOD (unset)
* JINA_OPTOUT_TELEMETRY (unset)
* JINA_RANDOM_PORT_MAX (unset)
* JINA_RANDOM_PORT_MIN (unset)
* JINA_LOCKS_ROOT (unset)
* JINA_K8S_ACCESS_MODES (unset)
* JINA_K8S_STORAGE_CLASS_NAME (unset)
* JINA_K8S_STORAGE_CAPACITY (unset)
* JINA_STREAMER_ARGS (unset)

[joschkabraun]

I am using 4.16.2 (95914)