MountVolume.SetUp failed for volume "kube-api-access-k5lqn" : secret "default.service-account-token" not found

BartNetJS commented 1 year ago

Pre-requisites

[X] I have double-checked my configuration
[X] I can confirm the issues exists when I tested with :latest
[ ] I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

I try the use the http template, like this:

`apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: http-template- spec: entrypoint: main templates:

name: main steps:
- - name: api template: http arguments: parameters: [{name: url, value: "http://some_url"}]
name: http inputs: parameters:
- name: url http: timeoutSeconds: 5 url: "{{inputs.parameters.url}}" method: "GET" successCondition: "response.statusCode == 200" `

The workflow never stops and the pod gives: 'MountVolume.SetUp failed for volume "kube-api-access-k5lqn" : secret "default.service-account-token" not found'

I see that indeed the token doesn't exist on the default service account. I tried to create it in this way:

kubectl create secret generic default.service-account-token -n argo \ --from-file=token=$HOME/.kube/config \ --type=kubernetes.io/service-account-token \ --dry-run=client -o json | jq '(.metadata.annotations."kubernetes.io/service-account.name") |= "argo-server"' | kubectl apply -f -

I'm testing a kubernetes cluster on docker desktop with kubernetes version v1.25.2

but then I got

http-template-x2gdp-1340600742-agent: time="2023-01-10T18:23:16.968Z" level=info msg="Starting Workflow Executor" version=v3.4.4 http-template-x2gdp-1340600742-agent: time="2023-01-10T18:23:16.968Z" level=info msg="Starting Agent" requeueTime=10s taskWorkers=16 workflow=http-template-x2gdp http-template-x2gdp-1340600742-agent: Error: Get "https://10.96.0.1:443/apis/argoproj.io/v1alpha1/namespaces/argo/workflowtasksets?fieldSelector=metadata.name%3Dhttp-template-x 2gdp&watch=true": net/http: invalid header field value "Bearer apiVersion: v1\nclusters:\n- cluster:\n certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSU ......

Version

v3.4.4

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: http-template-
spec:
  entrypoint: main
  templates:
    - name: main
      steps:
        - - name: api
            template: http
            arguments:
             parameters: [{name: url, value: "http://some_url"}]
    - name: http
      inputs:
        parameters:
          - name: url
      http:
        timeoutSeconds: 5 
        url: "{{inputs.parameters.url}}"
        method: "GET"
        successCondition: "response.statusCode == 200"

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

time="2023-01-10T18:51:27.381Z" level=info msg="Processing workflow" namespace=argo workflow=http-template-bm4q7
time="2023-01-10T18:51:27.385Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=http-template-bm4q7
time="2023-01-10T18:51:27.385Z" level=info msg="Steps node http-template-bm4q7 initialized Running" namespace=argo workflow=http-template-bm4q7
time="2023-01-10T18:51:27.385Z" level=info msg="StepGroup node http-template-bm4q7-2810781370 initialized Running" namespace=argo workflow=http-template-bm4q7
time="2023-01-10T18:51:27.386Z" level=info msg="HTTP node http-template-bm4q7-742621109 initialized Pending" namespace=argo workflow=http-template-bm4q7
time="2023-01-10T18:51:27.386Z" level=info msg="Workflow step group node http-template-bm4q7-2810781370 not yet completed" namespace=argo workflow=http-template-bm4q7
time="2023-01-10T18:51:27.386Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=http-template-bm4q7
time="2023-01-10T18:51:27.386Z" level=info msg="Creating TaskSet" namespace=argo workflow=http-template-bm4q7
time="2023-01-10T18:51:27.391Z" level=info msg=reconcileAgentPod namespace=argo workflow=http-template-bm4q7
time="2023-01-10T18:51:27.439Z" level=info msg="Created Agent pod" namespace=argo podName=http-template-bm4q7-1340600742-agent workflow=http-template-bm4q7
time="2023-01-10T18:51:27.439Z" level=info msg=updateAgentPodStatus namespace=argo workflow=http-template-bm4q7
time="2023-01-10T18:51:27.450Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=85530 workflow=http-template-bm4q7
time="2023-01-10T18:51:37.442Z" level=info msg="Processing workflow" namespace=argo workflow=http-template-bm4q7
time="2023-01-10T18:51:37.443Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=http-template-bm4q7
time="2023-01-10T18:51:37.443Z" level=info msg=updateAgentPodStatus namespace=argo workflow=http-template-bm4q7
time="2023-01-10T18:51:37.444Z" level=info msg="Workflow step group node http-template-bm4q7-2810781370 not yet completed" namespace=argo workflow=http-template-bm4q7
time="2023-01-10T18:51:37.444Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=http-template-bm4q7
time="2023-01-10T18:51:37.444Z" level=info msg=reconcileAgentPod namespace=argo workflow=http-template-bm4q7

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

error: container wait is not valid for pod http-template-bm4q7-1340600742-agent

terrytangyuan commented 1 year ago

Try again after you create the secret:

apiVersion: v1
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: default
  name: default.service-account-token
type: kubernetes.io/service-account-token

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

miltalex commented 1 month ago

Try again after you create the secret:

apiVersion: v1
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: default
  name: default.service-account-token
type: kubernetes.io/service-account-token

The above solves the issue. However, should we fix that on the deployment manifests?

Edit: To add more info, I deployed version v3.5.10 with helm chart v0.42.0 I was expecting that specifying the service account on the values.yaml will create the necessary resources, or we should update the documentation to reflect any required steps.

argoproj / argo-workflows