artifactGC not working with IRSA

tooptoop4 commented 2 years ago

Pre-requisites

[X] I have double-checked my configuration
[X] I can confirm the issues exists when I tested with :latest
[ ] I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

what happened? artifactgc step failed, object still on s3 and workflow did not get archived

what expected? artifactgc step succeed, object removed off s3 and workflow gets archived

Version

v3.4.1

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

              spec:
                serviceAccountName: argo-workflows-server
                artifactGC:
                  strategy: OnWorkflowCompletion
                  serviceAccountName: argo-workflows-workflow-controller
                  podMetadata:
                    annotations:
                      eks.amazonaws.com/role-arn: "arn:aws:iam::redact:role/redact..approle"

logs from the artgc pod:

{"log":"time=\"2022-10-04T10:16:46.328Z\" level=info msg=\"S3 Delete artifact: key: redact/main.log\"\n","stream":"stderr","time":"2022-10-04T10:16:46.32863739Z"}
{"log":"time=\"2022-10-04T10:16:46.328Z\" level=info msg=\"Creating minio client using AWS SDK credentials\"\n","stream":"stderr","time":"2022-10-04T10:16:46.328647872Z"}
{"log":"time=\"2022-10-04T10:16:46.573Z\" level=warning msg=\"Non-transient error: NoCredentialProviders: no valid providers in chain. Deprecated.\\n\\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors\"\n","stream":"stderr","time":"2022-10-04T10:16:46.573355156Z"}

note: for other AWS things to work i set volume/volumemounts/env variables as in https://github.com/argoproj/argo-workflows/discussions/7461 but artifactgc pod doesn't seem to accept that?

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

terrytangyuan commented 2 years ago

Can you check if the env/volume has been added to the pod?

sarabala1979 commented 2 years ago

@tooptoop4 Can you try with a simple POD resource that can access the AWS using IRSA?

juliev0 commented 2 years ago

I don't know a lot about the various ways of configuring IRSA but what what we tried to provide was two methods of access in S3:

annotations - your eks.amazonaws.com/role-arn should have been transferred to the Artifact GC Pod
you can configure ServiceAccount for Artifact GC in the Workflow spec

juliev0 commented 2 years ago

Could you do something like what's shown here?

sarabala1979 commented 2 years ago

This is enhancement. MVP version supports metadata. we can enhance to support env too. @tooptoop4 Do you like to submit PR for this enhancement?

juliev0 commented 2 years ago

For anybody working on this, the idea would be to take the ArtifactGC struct - which can be used both on the WorkflowSpec level and the Artifact level - and add the capability to specify environment variables there.

The way ArtifactGC Pods get created currently, for each unique GC strategy, we need to create one unique Pod per unique set of annotations and labels defined in ArtifactGC struct.

So, the functionality in artifact_gc.go is taking all the artifacts defined and grouping them by annotations/labels. What we need to do is add environment variables to this.

tooptoop4 commented 2 years ago

I guess need volumeMounts/volumes too

juliev0 commented 2 years ago

I guess need volumeMounts/volumes too

yes, you're right