[helm] Heating 429 Too Many Requests on orchestrator-repl-job pods

[AronsonDan]

Helm Chart Version

0.293.4

What step the error happened?

Upgrading the Platform or Helm Chart

Relevant information

During the upgrade to the new 0.293.4 helm chart im getting:

Failed to pull image "airbyte/container-orchestrator:0.63.8": failed to pull and unpack image "docker.io/airbyte/container-orchestrator:0.63.8": failed to copy: httpReadSeeker: failed open: unexpected status code https://registry1.docker.io/v2/airbyte/containerorchestrator/manifests/sha256:f2277e78343406ff9adddcb13b79078893e747ec3ef2353b7510c60c0a637e22: 429 Too Many Requests -Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

In my values, i set the following:

    jobs:
        main_container_image_pull_secret: "dockerhub-secret"
  worker:
    image:
      repository: 143653229561.dkr.ecr.us-east-1.amazonaws.com/docker-hub/airbyte/worker
    containerOrchestrator:
      image: 143653229561.dkr.ecr.us-east-1.amazonaws.com/docker-hub/airbyte/container-orchestrator:0.63.8
  workload-launcher:
    image:
      repository: 143653229561.dkr.ecr.us-east-1.amazonaws.com/docker-hub/airbyte/workload-launcher
    containerOrchestrator:
      image: 143653229561.dkr.ecr.us-east-1.amazonaws.com/docker-hub/airbyte/container-orchestrator:0.63.8

Im using ecr pull through cache so i wont have the docker server throttling issue.

When im looking at the airbyte-env config map, I see the following value: CONTAINER_ORCHESTRATOR_IMAGE: 143653229561.dkr.ecr.us-east-1.amazonaws.com/docker-hub/airbyte/container-orchestrator:0.63.8

But when orchestrator-repl-job are starting they appear with the following image: image: airbyte/container-orchestrator:0.63.8

On top of that, when I set main_container_image_pull_secret: "dockerhub-secret" i would have expected the pull to be made using my docker hub credentials which is not happenning:

- name: JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET                                                                                                                  
   value: dockerhub-secret

Would love to get your help on that :-)

Relevant log output

No response

[AronsonDan]

Saw that line of code that might be related: https://github.com/airbytehq/airbyte-platform/blob/8f00c4184f6a90506caf3aea78c61c18ba090fad/airbyte-server/Tiltfile#L32

[marcosmarxm]

You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

[AronsonDan]

@marcosmarxm We had that do to the fact that we had a separate Airbye deployment which caused a lot of docker pulling.

What I would really love is the ability to add a prefix to all of the docker images being pulled by Airbyte as the current solution would not tolerate significant scale. Solving the issue using pull through cache would be significantly better solution.

BTW, i have authenticated as described in:

- name: JOB_KUBE_MAIN_CONTAINER_IMAGE_PULL_SECRET                                                                                                                  
   value: dockerhub-secret

[sivankumar86]

@marcosmarxm I have created a PR to resolve above issue. Could you please take a look on this ? it is a blocker. I have build custom image and applied as a workaround but, it is tedious to maintain during upgrade.

https://github.com/airbytehq/airbyte-platform/pull/335

[sivankumar86]

issue : image secret is not passed to container-orchestrator pod creation API in worker code . solution: get the secret from env and add it to pod creation API.