sarabala1979
commented
1 year ago @vitalyrychkov can you provide more details like failed PodSpec and your env setup? Is there a way to reproduce locally?
Open vitalyrychkov opened 2 years ago
sarabala1979
commented
1 year ago @vitalyrychkov can you provide more details like failed PodSpec and your env setup? Is there a way to reproduce locally?
vitalyrychkov
commented
1 year ago @sarabala1979
Hi, thank you for patience, it took some time as I tried to create a meaningful test case for you. I have created a primitive workflow and a server deployment based on the same image in my private artifactory. The server, which has the imagePullSecret definition in the deployment, started fine in a pod. I did not create the mentioned secret. Then I have submitted the workflow based on the same image :
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: cmdtest-
labels:
workflows.argoproj.io/archive-strategy: "false"
annotations:
workflows.argoproj.io/description: |
This is a test for image command and entrypoint
spec:
entrypoint: cmdtest
templates:
- name: cmdtest
container:
image: 'artifacts.mycorp.net/docker/docserver:latest'In the first run I got the same error, although I did not specify any imagePullSecret in the Workflow:
failed to look-up entrypoint/cmd for image "artifacts.mycorp.net/docker/docserver:latest", you must either explicitly specify the command, or list the image's command in the index: https://argoproj.github.io/argo-workflows/workflow-executors/#emissary-emissary: secrets "app-registry-creds" not foundThen I tried with a non-existent version of my image: image: 'artifacts.mycorp.net/docker/hugo:1.2.3'
In this case Argo receives the corresponding message from the registry:
failed to look-up entrypoint/cmd for image "artifacts.mycorp.net/docker/hugo:1.2.3", you must either explicitly specify the command, or list the image's command in the index: https://argoproj.github.io/argo-workflows/workflow-executors/#emissary-emissary: GET https://artifacts.mycorp.net/v2/docker/hugo/manifests/1.2.3: MANIFEST_UNKNOWN: The named manifest is not known to the registry.; map[manifest:hugo/1.2.3/manifest.json]Reverted back to the latest and submitted:image: 'artifacts.mycorp.net/docker/docserver:latest'
And all of a sudden Argo can download the image and run the container!
I guess it could be a some kind of cached first deployment's settings in Argo or Kubelet ???
I will keep an eye on the issue and try to nail it down as soon as it re-occurs or maybe someone else reports the same. For now I would appreciate if someone checks the code if the manifest pull function uses anything like a credentials cache.
Thank you
anhqqt
commented
1 year ago I met the same issue Image pull error: User "system:serviceaccount:argo-workflows:argo-workflows-controller" cannot get resource "secrets" in API group "" in the namespace "argo-workflows" . From the previous, the workflow could normally run in EKS 1.22
But after I created another EKS 1.23 and install the Argo Workflow helm chart with the same values.yaml file, this issue happened. Even if the workflow crd is in the same namespace as the controller, the controller cannot read the imagePullSecret.
My workaround is to turn off the controller.rbac.create and controller.serviceAccount.create in the helm values file. Manually create the ServiceAccount, ClusterRole (with Get Secret permission), ClusterRoleBinding, and put the SA name into controller.serviceAccount.name. Then the workflow controller pod will use the SA I created and is able to read the imagePullSecret
vitalyrychkov
commented
1 year ago RIght, my workaround was the same, however I did not have to disable *.create values, just added a ClusterRole to read the secret with a specific name in all namespaces and a ClusterRoleBinding to the workflow-controller's SA.
stale[bot]
commented
1 year ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.
stale[bot]
commented
1 year ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.
noamokman
commented
7 months ago Hey, I just added this to the values yaml for the chart
argo-workflows:
controller:
rbac:
secretWhitelist:
- image-pull-secretreplace the name of the secret with any secret you may have. Works for me when pulling from ECR.
Pre-requisites
:latestWhat happened/what you expected to happen?
Kubernetes pulls the same images without using imagePullSecrets if anonymous pull is allowed. Argo executor shall also pull a container image (to check the cmd/args value) without using imagePullSecrets if anonymous pull is enabled.
Version
3.4.1
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
Logs from the workflow controller
Logs from in your workflow's wait container
No wait logs available, seems argo did not come that far.