yatai-deployment buildBentoImages failed to check image docker.io

[guyskk]

Hi, I follow the guide https://docs.bentoml.org/projects/yatai/en/latest/installation/yatai_deployment.html#prepare-container-registry and use docker.io for container registry.

For the DOCKER_REGISTRY_SERVER config, I have try docker.io, index.docker.io, index.docker.io/v1, https://index.docker.io/v1 and all not works. I checked docker login index.docker.io success in my local env.

Could you help me?

export DOCKER_REGISTRY_SERVER=docker.io
export DOCKER_REGISTRY_USERNAME=mobenehub
export DOCKER_REGISTRY_PASSWORD=xxx # dockerhub token
export DOCKER_REGISTRY_SECURE=true
export DOCKER_REGISTRY_BENTO_REPOSITORY_NAME=mobenehub/yatai-bentos

The logs of yatai-deployment: kubectl -n yatai-deployment logs -f deploy/yatai-deployment

1.669177552548967e+09   INFO    start cleaning up abandoned runner services {"func": "doCleanUpAbandonedRunnerServices"}
1.6691775525491023e+09  INFO    finished cleaning up abandoned runner services  {"func": "doCleanUpAbandonedRunnerServices"}
1.6691775532939029e+09  INFO    getting yatai client    {"func": "doBuildBentoImages"}
1.6691775532942638e+09  INFO    getting docker registry {"func": "doBuildBentoImages"}
1.6691775532993517e+09  INFO    listing bentos from yatai   {"func": "doBuildBentoImages"}
1.669177553338061e+09   INFO    found 1 bentos need to build image  {"func": "doBuildBentoImages"}
1.6691775533381195e+09  INFO    checking image docker.io/mobenehub/yatai-bentos:yatai.iris_classifier.h43swpdk3s4g5srm exists   {"func": "doBuildBentoImages", "bentoTag": "iris_classifier:h43swpdk3s4g5srm"}
1.6691775541727772e+09  ERROR   buildBentoImages    {"func": "buildBentoImages", "error": "failed to check image docker.io/mobenehub/yatai-bentos:yatai.iris_classifier.h43swpdk3s4g5srm exists for bento iris_classifier:h43swpdk3s4g5srm: get tags for docker image mobenehub/yatai-bentos: invalid character '<' looking for beginning of value", "errorVerbose": "failed to check image docker.io/mobenehub/yatai-bentos:yatai.iris_classifier.h43swpdk3s4g5srm exists for bento iris_classifier:h43swpdk3s4g5srm: get tags for docker image mobenehub/yatai-bentos: invalid character '<' looking for beginning of value\ngithub.com/bentoml/yatai-common/sync/errsgroup.(*Group).Wait\n\t/go/pkg/mod/github.com/bentoml/yatai-common@v0.0.0-20221020025746-4b883e1aedbe/sync/errsgroup/errsgroup.go:70\ngithub.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).doBuildBentoImages\n\t/workspace/controllers/bentodeployment_controller.go:2224\ngithub.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).buildBentoImages\n\t/workspace/controllers/bentodeployment_controller.go:2256\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594"}
github.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).buildBentoImages
    /workspace/controllers/bentodeployment_controller.go:2258

[yetone]

I suspect it is a network issue with your k8s cluster, can you check if your cluster can communicate with docker.io with the following command?

kubectl delete pod test-docker-hub
kubectl run test-docker-hub --rm --tty -i --restart='Never' \
    --namespace default \
    --image yetone/curl \
    --command -- sh -c 'TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:yetone/yatai-bentos:pull" | jq -r ".token"); curl -H "Authorization: Bearer $TOKEN" https://index.docker.io/v2/yetone/yatai-bentos/tags/list'

The expected output is:

[guyskk]

@yetone

$ kubectl delete pod test-docker-hub
$ kubectl run test-docker-hub --rm --tty -i --restart='Never' \
    --namespace default \
    --image yetone/curl \
    --command -- sh -c 'TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:yetone/yatai-bentos:pull" | jq -r ".token"); curl -H "Authorization: Bearer $TOKEN" https://index.docker.io/v2/yetone/yatai-bentos/tags/list'
Error from server (NotFound): pods "test-docker-hub" not found

If you don't see a command prompt, try pressing enter.
warning: couldn't attach to pod/test-docker-hub, falling back to streaming logs: unable to upgrade connection: container test-docker-hub not found in pod test-docker-hub_default
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4409    0  4409    0     0  13159      0 --:--:-- --:--:-- --:--:-- 13161
{"name":"yetone/yatai-bentos","tags":["yatai.default.iris_classifier.tdldlix5agxoqusu"]}
pod "test-docker-hub" deleted

[guyskk]

Hi @yetone , is there any suggestions or updates?

[yetone]

@guyskk Sorry for the late reply, I have already fixed this bug in yatai-deployment, you can upgrade yatai-deployment to v1.0.4 to prevent this bug.

https://github.com/bentoml/yatai-deployment/releases/tag/v1.0.4

[guyskk]

Thank you very much!