search

neuro-inc / neuro-extras

Curated set of MLOps tools to work with the Neu.ro MLOps platform
https://neu-ro.gitbook.io/neuro-extras-reference/cli
Other
4 stars 1 forks source link

[B] Image transfer works only if the transfer was initiated from `src` cluster #294

Closed YevheniiSemendiak closed 3 years ago

YevheniiSemendiak commented 3 years ago

Summary

If one performs neuro-extras image transfer <src> <dst>, being on the dst cluster, the transfer job will fail to retrieve image from src cluster, since docker auth credits will be fetched from dst cluster, but should be fetched from src cluster.

Example

job-5c3e4ddc-8fd9-4347-8c53-abfc517025d7 logs:

E0721 17:15:59.334305       1 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
    For verbose messaging see aws.Config.CredentialsChainVerboseErrors
INFO[0007] Retrieving image manifest registry.neuro-compute.org.neu.ro/jurpasha/mlops_demo_oss_names:v3
INFO[0007] Retrieving image registry.neuro-compute.org.neu.ro/jurpasha/mlops_demo_oss_names:v3
ERRO[0007] Error while retrieving image from cache: registry.neuro-compute.org.neu.ro/jurpasha/mlops_demo_oss_names:v3 GET https://registry.neuro-compute.org.neu.ro/v2/jurpasha/mlops_demo_oss_names/manifests/v3: unsupported status code 401; body: 401: Unauthorized
INFO[0007] Retrieving image manifest registry.neuro-compute.org.neu.ro/jurpasha/mlops_demo_oss_names:v3
INFO[0007] Retrieving image registry.neuro-compute.org.neu.ro/jurpasha/mlops_demo_oss_names:v3
error building image: GET https://registry.neuro-compute.org.neu.ro/v2/jurpasha/mlops_demo_oss_names/manifests/v3: unsupported status code 401; body: 401: Unauthorized

Environment

Mandatory:

neuro-extras package version: 21.7.2 Neuro Platform Client 21.7.9

YevheniiSemendiak commented 3 years ago

The problem is that we are switching the cluster to dst cluster for the entire time of building the image, we should switch it only for the time of credentials generation.

Example of successful run:

ne image transfer image://neuro-compute/yevheniisemendiak/redis:4.0-alpine image://default/yevheniisemendiak/redis:4.0-alpine
INFO: Already on cluster: neuro-compute
INFO: Temporarily switching cluster: neuro-compute -> default         <----------------------------------  
INFO: Building the image image://default/yevheniisemendiak/redis:4.0-alpine
INFO: Using file:///var/folders/2h/crz9sjyd4xx0rlpvbpb56lth0000gn/T/tmpogg1ky1i as the build context
INFO: Uploading file:...

INFO: Submitting a builder job
...
√ Job <id> stopped
INFO: Successfully built image://default/yevheniisemendiak/redis:4.0-alpine
INFO: Switching back cluster: default -> neuro-compute          <----------------------------------