Approvaltasks cannot be triggered by Tekton pipeline

[umtakcn]

Hi, I am using latest Tekton Pipeline, Dashboard and Automatiko versions (v1beta1 customtask) on native Kubernetes Cluster(1.25.5). I applied crd, dashboard extensions, automatiko application and example pipeline-single.yaml. But when I create pipeline, there is no approvaltask triggered. Neither on dashboard nor on the server side. Pipeline is stuck on approval step with Waiting for Pod resource until timeout runs out. No new log on automatiko application. I am using default yaml configurations. (Only changed namespaces) I can share my configurations or screenshots if you want. Can you please help me? I really want to use this extension. Thanks.

[mswiderski]

Could you provide more information on what and were was deployed from approval point of view. Like the complete set of resources to deploy approval tasks. So the more information you can provide the better so I can check where the issue might be.

[umtakcn]

First of all, I am using this version of Tekton. Tekton components are in tekton-pipelines namespace and my pipelines are in tekton namespace. With last tekton version you can not create pipelines in tekton-pipelines namespace.

I applied crd:

# Generated by Fabric8 CRDGenerator, manual edits might get overwritten!
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: approvaltasks.tekton.automatiko.io
spec:
  group: tekton.automatiko.io
  names:
    kind: ApprovalTask
    plural: approvaltasks
    singular: approvaltask
  scope: Namespaced
  versions:
  - name: v1beta1
    schema:
      openAPIV3Schema:
        properties:
          spec:
            properties:
              strategy:
                enum:
                - FOUR_EYES
                - MULTI
                - SINGLE
                type: string
              approvers:
                items:
                  type: string
                type: array
              groups:
                items:
                  type: string
                type: array
              pipeline:
                type: string
              description:
                type: string
            type: object
          status:
            properties:
              results:
                properties:
                  decision:
                    type: string
                  comment:
                    type: string
                type: object
              status:
                type: string
              reason:
                type: string
              approvalUrl:
                type: string
              message:
                type: string
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}

Then dashboard extensions:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: tekton-dashboard-approvaltasks-extension
rules:
  - apiGroups: ["tekton.automatiko.io"]
    resources: ["approvaltasks"]
    verbs: ["get", "list", "watch"]

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tekton-dashboard-approvaltasks-role-binding
roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: tekton-dashboard-approvaltasks-extension
subjects:
  - kind: ServiceAccount
    name: tekton-dashboard
    namespace: tekton-pipelines

I applied this one in tekton namespace: Because my pipelines are in tekton namespace.

apiVersion: dashboard.tekton.dev/v1alpha1
kind: Extension
metadata:
  name: approvaltasks
spec:
  apiVersion: tekton.automatiko.io/v1beta1
  name: approvaltasks
  displayname: Approval Tasks
  namespaced: true

Then I applied kubernetes-basic-pv: I changed the namespaces.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    app.quarkus.io/build-timestamp: 2022-02-05 - 11:22:03 +0000
  labels:
    app.kubernetes.io/version: 0.6.0
    app.kubernetes.io/name: automatiko-approval-task
  name: automatiko-approval-task
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    app.quarkus.io/build-timestamp: 2022-02-05 - 11:22:03 +0000
  labels:
    app.kubernetes.io/name: automatiko-approval-task
    app.kubernetes.io/version: 0.6.0
  name: automatiko-approval-task
spec:
  ports:
    - name: http
      port: 80
      targetPort: 8080
  selector:
    app.kubernetes.io/name: automatiko-approval-task
    app.kubernetes.io/version: 0.6.0
  type: ClusterIP
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: automatiko-approval-task-view
roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: view
subjects:
  - kind: ServiceAccount
    name: automatiko-approval-task
    namespace: tekton
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: approvals-cluster-role
rules:
  - apiGroups:
      - tekton.automatiko.io
    resources:
      - approvaltasks
      - approvaltasks/status
      - approvaltasks/finalizers
    verbs:
      - get
      - list
      - watch
      - create
      - delete
      - patch
      - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: josdk-crd-validating-cluster-role
rules:
  - apiGroups:
      - apiextensions.k8s.io
    resources:
      - customresourcedefinitions
    verbs:
      - get
      - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: customruns-cluster-role
rules:
  - apiGroups:
      - tekton.dev
    resources:
      - customruns
      - customruns/status
      - customruns/finalizers
    verbs:
      - get
      - list
      - watch
      - patch
      - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: approvals-cluster-role-binding
roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: approvals-cluster-role
subjects:
  - kind: ServiceAccount
    name: automatiko-approval-task
    namespace: tekton
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: approvals-crd-validating-role-binding
roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: josdk-crd-validating-cluster-role
subjects:
  - kind: ServiceAccount
    name: automatiko-approval-task
    namespace: tekton
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: runs-cluster-role-binding
roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: runs-cluster-role
subjects:
  - kind: ServiceAccount
    name: automatiko-approval-task
    namespace: tekton
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: runs-crd-validating-role-binding
roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: josdk-crd-validating-cluster-role
subjects:
  - kind: ServiceAccount
    name: automatiko-approval-task
    namespace: tekton
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: approvaltask-pv-claim
spec:
  storageClassName: local-storage
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    app.quarkus.io/build-timestamp: 2022-02-05 - 11:22:03 +0000
  labels:
    app.kubernetes.io/name: automatiko-approval-task
    app.kubernetes.io/version: 0.6.0
  name: automatiko-approval-task
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: automatiko-approval-task
      app.kubernetes.io/version: 0.6.0
  template:
    metadata:
      annotations:
        app.quarkus.io/build-timestamp: 2022-02-05 - 11:22:03 +0000
      labels:
        app.kubernetes.io/name: automatiko-approval-task
        app.kubernetes.io/version: 0.6.0
    spec:
      volumes:
       - name: approvaltask-pv-storage
         persistentVolumeClaim:
           claimName: approvaltask-pv-claim    
      containers:
        - env:
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: QUARKUS_OPERATOR_SDK_NAMESPACES
              value: tekton
            - name: QUARKUS_AUTOMATIKO_SERVICE_URL
              value: http://approve.xxx #I masked the address here. I can reach the interface via ingress just fine.
            - name: QUARKUS_AUTOMATIKO_PERSISTENCE_FILESYSTEM_PATH
              value: /data/processes
            - name: QUARKUS_AUTOMATIKO_JOBS_FILESYSTEM_PATH
              value: /data/jobs                                 
          image: automatiko/automatiko-approval-task
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /q/health/live
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 0
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 10
          name: automatiko-approval-task
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          volumeMounts:
           - mountPath: "/data"
             name: approvaltask-pv-storage              
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /q/health/ready
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 0
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 10
      serviceAccountName: automatiko-approval-task

Also I use this pv: Though I tried it without pv. No change.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: approval-pv
  labels:
    type: local
spec:
  storageClassName: local-storage
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/opt/data/tekton/approval-data"

Lastly, I applied example task and pipeline.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: print-decision
spec:
  params:
    - name: decision
      type: string
    - name: comment
      type: string
  steps:
    - name: print
      image: ubuntu
      command:
        - echo
      args:
        - "Approval done $(params.decision) with comment $(params.comment)"

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: testpipeline2
spec:
  tasks:
    - name: deploy-dev-approval-task
      taskRef:
        apiVersion: tekton.automatiko.io/v1beta1
        kind: ApprovalTask
        name: approvaltask
      params:
        - name: pipeline
          value: "$(context.pipelineRun.name)"
        - name: description
          value: "Approval from pipeline $(context.pipeline.name) for development environment"
        - name: approvers
          value:
            - "john@email.com"
    - name: deploy-dev-approved-task
      when:
        - input: $(tasks.deploy-dev-approval-task.results.decision)
          operator: in
          values: [ "true" ]
      taskRef:
        name: print-decision
      runAfter:
        - deploy-dev-approval-task
      params:
        - name: decision
          value: "APPROVED"
        - name: comment
          value: $(tasks.deploy-dev-approval-task.results.comment)
    - name: deploy-dev-rejected-task
      when:
        - input: $(tasks.deploy-dev-approval-task.results.decision)
          operator: in
          values: [ "false" ]
      taskRef:
        name: print-decision
      runAfter:
        - deploy-dev-approval-task
      params:
        - name: decision
          value: "REJECTED"
        - name: comment
          value: $(tasks.deploy-dev-approval-task.results.comment)

When I run the pipeline like this.

Nothing is created.

Application logs:

root@ist6lnxdvpm1:/opt/v1beta1/test# kubectl logs automatiko-approval-task-5949656bfd-m8zwj -n tekton
 ______     __  __     ______   ______     __    __     ______     ______   __     __  __     ______
/\  __ \   /\ \/\ \   /\__  _\ /\  __ \   /\ "-./  \   /\  __ \   /\__  _\ /\ \   /\ \/ /    /\  __ \
\ \  __ \  \ \ \_\ \  \/_/\ \/ \ \ \/\ \  \ \ \-./\ \  \ \  __ \  \/_/\ \/ \ \ \  \ \  _"-.  \ \ \/\ \
 \ \_\ \_\  \ \_____\    \ \_\  \ \_____\  \ \_\ \ \_\  \ \_\ \_\    \ \_\  \ \_\  \ \_\ \_\  \ \_____\
  \/_/\/_/   \/_____/     \/_/   \/_____/   \/_/  \/_/   \/_/\/_/     \/_/   \/_/   \/_/\/_/   \/_____/
                                                                         Powered by Quarkus 2.15.1.Final
2023-02-03 08:31:38,401 INFO  [io.qua.sma.ope.run.OpenApiRecorder] (main) CORS filtering is disabled and cross-origin resource sharing is allowed without restriction, which is not recommended in production. Please configure the CORS filter through 'quarkus.http.cors.*' properties. For more information, see Quarkus HTTP CORS documentation
2023-02-03 08:31:38,417 INFO  [io.qua.ope.run.OperatorProducer] (main) Quarkus Java Operator SDK extension 5.0.0 (commit: d3d8e72 on branch: d3d8e72e0d6a1a578831c95494ca0bce5646d61c) built on Sat Dec 17 09:10:29 GMT 2022
2023-02-03 08:31:38,418 INFO  [io.qua.ope.run.AppEventListener] (main) Starting operator.
2023-02-03 08:31:38,469 WARN  [io.fab.kub.cli.dsl.int.VersionUsageUtils] (InformerWrapper [customruns.tekton.dev/v1beta1] 1) The client is using resource type 'customruns' with unstable version 'v1beta1'
2023-02-03 08:31:38,521 WARN  [io.fab.kub.cli.dsl.int.VersionUsageUtils] (InformerWrapper [approvaltasks.tekton.automatiko.io/v1beta1] 1) The client is using resource type 'approvaltasks' with unstable version 'v1beta1'
2023-02-03 08:31:38,569 INFO  [io.quarkus] (main) automatiko-approval-task 0.6.0 native (powered by Quarkus 2.15.1.Final) started in 0.192s. Listening on: http://0.0.0.0:8080
2023-02-03 08:31:38,569 INFO  [io.quarkus] (main) Profile prod activated.
2023-02-03 08:31:38,569 INFO  [io.quarkus] (main) Installed features: [automatiko, automatiko-operator, cdi, kubernetes, kubernetes-client, mailer, micrometer, openshift-client, operator-sdk, qute, resteasy-reactive, resteasy-reactive-jackson, resteasy-reactive-qute, security, smallrye-context-propagation, smallrye-health, smallrye-openapi, swagger-ui, vertx, websockets, websockets-client]

If you need anything else. Let me know please.

[mswiderski]

thanks for the details, I think the problem is with the role binding, currently you have it like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: runs-cluster-role-binding
roleRef:
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
  name: runs-cluster-role
subjects:
  - kind: ServiceAccount
    name: automatiko-approval-task
    namespace: tekton

and the problem is with the roleRef.name which points to not existing cluster role as the role name is customruns-cluster-role

If I am not mistaken this simply prevents of any of the events to be pushed to approval task pod and that's why none of the work was triggered. Can you please adjust the name of the reference cluster role and reapply that?

[umtakcn]

Thank you. It works now. You may want to update your repo though since I copied the yaml configuration from there. It is easy to miss.

[mswiderski]

great to hear, and indeed will need to update the files as it was a leftover after migrating to v1Beta1

Thanks for reporting that and enjoy Approval Tasks - don't hesitate to share you feedback as well!

[mswiderski]

updated manifest files with this commit https://github.com/automatiko-io/automatiko-approval-task/commit/064169fdec64b47d866fc5e9d0a62b1bae8bb5ec