The OauthAccessToken task doesn't work as expected when wrapped within a WorkingDirectory task

relusc commented 6 months ago

Expected Behavior

I am trying to push a docker image to GCP Artifact Registry. Authentication should be done via OAuth access token, I am using the blueprint and this issue as a reference.

The issue is that when generating an access token via OauthAccessToken, the docker push does not work. When using GCloudCLI and running gcloud auth print-access-token, pushing the docker image with this token works perfectly fine.

Actual Behaviour

Both approaches basically do the same, right ? Both create OAuth access tokens. So I would expect that they behave the same.

I would like to be able to use the OauthAccessToken task here, as it's significantly faster than the GCloudCLI task (and less verbose in the flow definition file).

Steps To Reproduce

I am running a setup on Google Kubernetes Engine with Workload Identity Federation, authenticating against GCP services (like e.g. Artifact Registry) is done with the Kubernetes service account assigned to the Kestra deployment (all IAM roles are set, as written above the docker push itself works, but only for one of the two approaches)
Create the example flows below

Environment Information

Kestra Version: 0.16.6
Plugin version: v0.16.1 (plugin-gcp)
Operating System (OS / Docker / Kubernetes): Kubernetes on Google Cloud/GKE
Java Version (If not docker): -

Example flow

Examples do not include Git checkout/clone

Not working example:

tasks:
  - id: main
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
      ...
      - id: git_clone
      ...

      - id: fetch_auth_token
        type: io.kestra.plugin.gcp.auth.OauthAccessToken
        projectId: my-project

      - id: docker_build_push
        type: io.kestra.plugin.docker.Build
        dockerfile: path/to/Dockerfile
        tags:
          - us-east1-docker.pkg.dev/my-project/xyz/xyz:latest
        push: true
        credentials:
          registry: us-east1-docker.pkg.dev
          username: oauth2accesstoken
          password: "{{ outputs.fetch_auth_token.accessToken.tokenValue }}"

Working example:

tasks:
  - id: main
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
      ...
      - id: git_clone
      ...

      - id: "cli"
        type: "io.kestra.plugin.gcp.cli.GCloudCLI"
        projectId: my-project
        docker:
           image: google/cloud-sdk:slim # Base image does not matter, just including it for full example
        commands:
          - gcloud auth print-access-token | tr -d '\n' | xargs -0 -I {} echo '::{"outputs":{"token":"{}"}}::'

      - id: docker_build_push
        type: io.kestra.plugin.docker.Build
        dockerfile: path/to/Dockerfile
        tags:
          - us-east1-docker.pkg.dev/my-project/xyz/xyz:latest
        push: true
        credentials:
          registry: us-east1-docker.pkg.dev
          username: oauth2accesstoken
          password: "{{ outputs.cli.vars.token }}"

loicmathieu commented 5 months ago

The io.kestra.plugin.gcp.auth.OauthAccessToken and the gcloud auth print-access-token didn't return the same kind of tokens but both should work.

What are the exception in the log and how to you authenticate to Google Cloud? We have a blueprint for that, see https://kestra.io/blueprints/133-build-a-docker-image-and-push-it-to-google-cloud-artifact-registry

relusc commented 5 months ago

Hi @loicmathieu, thanks for your answer. I've seen the blueprint and used it as an initial reference (you can see it at the top of my issues description). For both methods I am relying on the default credentials fetched from the metadata server. The underlying K8s service account of the kestra deployment is used, I have configured workload identity federation for this. This works fine in general, as gcloud auth print-access-token works, and I can also use tasks like e.g. io.kestra.plugin.gcp.gcs.Downloads without providing service account key files. only for the docker push i am facing this problem.

Oh yes, I actually forgot to add the error message sorry. However, I "only" get a standard unauthorized: authentication failed error (see screenshot):

What is the difference between the generated tokens for both approaches ? I thought they generate similar tokens

paulgrainger85 commented 5 months ago

Hi @relusc I have set up a test on GKE using Workload Identity Federation using the Kubernetes Service Account. Unfortunately I was not able to replicate your issue and this succeeded as expected. An example working flow is below.

This does seem to be a permission error however I cannot see from your example above where that could be. Would there be any further messages in the TRACE log or from the pod itself to help diagnose?

id: artifactRegDemo
namespace: demo

tasks:
  - id: fetchAuthToken
    type: io.kestra.plugin.gcp.auth.OauthAccessToken
    projectId: my-project

  - id: docker_build
    type: io.kestra.plugin.docker.Build
    dockerfile: |
      FROM docker.io/alpine:latest
      RUN apk update && apk upgrade --no-interactive
      RUN mkdir -p /app/data
      CMD ["/bin/sh"]
    tags:
      - europe-west3-docker.pkg.dev/my-project/test/test:new
    push: true
    credentials:
      registry: europe-west3-docker.pkg.dev
      username: "oauth2accesstoken"
      password: "{{ outputs.fetchAuthToken.accessToken.tokenValue }}"

  - id: print_success
    type: io.kestra.plugin.scripts.shell.Commands
    docker:
      image: europe-west3-docker.pkg.dev/my-project/test/test:new
      credentials:
        registry: europe-west3-docker.pkg.dev
        username: oauth2accesstoken
        password: "{{ outputs.fetchAuthToken.accessToken.tokenValue }}"
    commands:
      - echo success!

relusc commented 5 months ago

Hi @paulgrainger85 , thanks for your help :)

I copy and pasted your workflow into my setup and I can confirm it is working. I wanted to exclude that something is entirely wrong with my setup.

However, as you can see in my examples in the issue description, I am running all my tasks within a io.kestra.plugin.core.flow.WorkingDirectory task. Now, when I wrap your sample workflow into a io.kestra.plugin.core.flow.WorkingDirectory task, it does not work anymore.

So your example above works, this example does not (at least in my setup, maybe you can also test this on your side):

id: artifactRegDemo
namespace: prod

tasks:
  - id: main
    type: io.kestra.plugin.core.flow.WorkingDirectory # Wrapped in a WorkingDirectory task
    tasks:
    - id: fetchAuthToken
      type: io.kestra.plugin.gcp.auth.OauthAccessToken
      projectId: my-project

    - id: docker_build
      type: io.kestra.plugin.docker.Build
      dockerfile: |
        FROM docker.io/alpine:latest
        RUN apk update && apk upgrade --no-interactive
        RUN mkdir -p /app/data
        CMD ["/bin/sh"]
      tags:
        - europe-west3-docker.pkg.dev/my-project/test/test:new
      push: true
      credentials:
        registry: europe-west3-docker.pkg.dev
        username: "oauth2accesstoken"
        password: "{{ outputs.fetchAuthToken.accessToken.tokenValue }}"

    - id: print_success
      type: io.kestra.plugin.scripts.shell.Commands
      docker:
        image: europe-west3-docker.pkg.dev/my-project/test/test:new
        credentials:
          registry: europe-west3-docker.pkg.dev
          username: oauth2accesstoken
          password: "{{ outputs.fetchAuthToken.accessToken.tokenValue }}"
      commands:
        - echo success!

When using a WorkingDirectory task, the flow only succeeds when creating the token via GCloudCLI.

This also means that if I run the OauthAccessToken task outside of the WorkingDirectory task , the flow will succeed. This example works as well:

id: artifactRegDemo
namespace: prod

tasks:
  - id: fetchAuthToken
    type: io.kestra.plugin.gcp.auth.OauthAccessToken # Runs outside of the WorkingDirectory task
    projectId: my-project

  - id: main
    type: io.kestra.plugin.core.flow.WorkingDirectory # Wrapped in a WorkingDirectory task
    tasks:
    - id: docker_build
      type: io.kestra.plugin.docker.Build
      dockerfile: |
        FROM docker.io/alpine:latest
        RUN apk update && apk upgrade --no-interactive
        RUN mkdir -p /app/data
        CMD ["/bin/sh"]
      tags:
        - europe-west3-docker.pkg.dev/my-project/test/test:new
      push: true
      credentials:
        registry: europe-west3-docker.pkg.dev
        username: "oauth2accesstoken"
        password: "{{ outputs.fetchAuthToken.accessToken.tokenValue }}"

    - id: print_success
      type: io.kestra.plugin.scripts.shell.Commands
      docker:
        image: europe-west3-docker.pkg.dev/my-project/test/test:new
        credentials:
          registry: europe-west3-docker.pkg.dev
          username: oauth2accesstoken
          password: "{{ outputs.fetchAuthToken.accessToken.tokenValue }}"
      commands:
        - echo success!

relusc commented 5 months ago

Hey guys, any update on this issue ? :)

anna-geller commented 3 months ago

no update on our end. if you have an idea on what might be the root cause for the GCloud SDK to invalidate the token when it's generated in a different process (wdir task), feel free to share more info or contribute a PR. Until then, your solution here sounds very reasonable and even somewhat cleaner (best to wrap tasks in wdir task only if absolutely needed so your last version is even cleaner):

id: artifactRegDemo
namespace: prod

tasks:
  - id: fetchAuthToken
    type: io.kestra.plugin.gcp.auth.OauthAccessToken # Runs outside of the WorkingDirectory task
    projectId: my-project

  - id: main
    type: io.kestra.plugin.core.flow.WorkingDirectory # Wrapped in a WorkingDirectory task
    tasks:
    - id: docker_build
      type: io.kestra.plugin.docker.Build
      dockerfile: |
        FROM docker.io/alpine:latest
        RUN apk update && apk upgrade --no-interactive
        RUN mkdir -p /app/data
        CMD ["/bin/sh"]
      tags:
        - europe-west3-docker.pkg.dev/my-project/test/test:new
      push: true
      credentials:
        registry: europe-west3-docker.pkg.dev
        username: "oauth2accesstoken"
        password: "{{ outputs.fetchAuthToken.accessToken.tokenValue }}"

    - id: print_success
      type: io.kestra.plugin.scripts.shell.Commands
      docker:
        image: europe-west3-docker.pkg.dev/my-project/test/test:new
        credentials:
          registry: europe-west3-docker.pkg.dev
          username: oauth2accesstoken
          password: "{{ outputs.fetchAuthToken.accessToken.tokenValue }}"
      commands:
        - echo success!

anna-geller commented 1 month ago

closing for now as we couldn't reproduce -- if you still face this issue, let us know and we'll reopen it and reinvestigate

kestra-io / plugin-gcp