[bitnami/argo-workflows] Error pulling private registry images in Argo Workflows

[julien-blanchon]

Error pulling private registry images in Argo Workflows

When using Argo Workflows with a private Docker registry (Harbor in my case), I'm getting tls: failed to verify certificate: x509: certificate signed by unknown authority errors when Workflows try to pull images.

It seems the root CA for the Harbor certificate is not being trusted on the Workflow pod, even though it should be available on all nodes.

Steps to Reproduce

1. Install Harbor using Helm, with a certificate from cert-manager

   # Harbor helm values
   adminPassword: adminpassword
   
   registry:
     secret: AAAAAAAAAAAAAAAA
     credentials:
       username: harbor_registry_user
       password: harbor_registry_password
   
   externalURL: https://harborui.kind.cluster
   exposureType: ingress
   
   service:
      annotations:
         kubernetes.io/ingress.class: nginx
         cert-manager.io/cluster-issuer: ca-issuer
   
   ingress:
     core:
       ingressClassName: nginx
       tls: true
       selfSigned: false
       hostname: harborui.kind.cluster
       annotations:
         ingress.kubernetes.io/ssl-redirect: 'true'
         ingress.kubernetes.io/proxy-body-size: '0'
         nginx.ingress.kubernetes.io/ssl-redirect: 'true'
         nginx.ingress.kubernetes.io/proxy-body-size: '0'
         ingress.kubernetes.io/force-ssl-redirect: 'true'
         kubernetes.io/ingress.class: nginx
         cert-manager.io/cluster-issuer: ca-issuer
     notary:
       ingressClassName: nginx
       tls: true
       selfSigned: false
       hostname: harbornotary.kind.cluster
       annotations:
         ingress.kubernetes.io/ssl-redirect: 'true'
         ingress.kubernetes.io/proxy-body-size: '0'
         nginx.ingress.kubernetes.io/ssl-redirect: 'true'
         nginx.ingress.kubernetes.io/proxy-body-size: '0'
         ingress.kubernetes.io/force-ssl-redirect: 'true'
         kubernetes.io/ingress.class: nginx
         cert-manager.io/cluster-issuer: ca-issuer

2. Install Argo Workflows using Helm, with a certificate from cert-manager

   # Argo helm values
   controller:
     workflowNamespaces:
       - default
       - pipeline   
   
   server:
     extraArgs: [--auth-mode=server] 
   
   ingress:
     enabled: true
     tls: true
     hostname: argo.kind.cluster
     annotations: 
       kubernetes.io/ingress.class: nginx
       cert-manager.io/cluster-issuer: ca-issuer
     ingressClassName: nginx

3. Create a docker-secret with credentials for the Harbor registry

4. Create a Workflow that uses an image from the private Harbor registry, with imagePullSecrets set to the docker-secret:

   # Example workflow
   apiVersion: argoproj.io/v1alpha1
   kind: Workflow
   metadata:
     name: example
     namespace: default
   spec:
     entrypoint: entry
     templates:
       - name: entry
         container:
           image: harborui.kind.cluster/library/test:v1
     imagePullSecrets:
       - name: docker-registry

5. The Workflow fails with a TLS verification error pulling the image

Expected Behavior

The Workflow should pull the private Docker image from the Harbor registry successfully.

Environment

• Architecure: amd64
• Kubernetes version: v1.27.3
• Argo Workflows version: bitnami/argo-workflows:3.4.10
• Harbor version: bitnami/harbor:2.8.3

Certificates

• Using cert-manager for certificates
• Following this guide for root CA

Argo RBAC

Giving Argo access to docker registry secret:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: access-secrets
  namespace: default
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - list
  - watch
---  
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default-to-secrets
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: access-secrets
subjects:
- kind: ServiceAccount
  name: argo-workflows-controller
  namespace: argo-workflows

kind config

Mounting root CA on nodes:

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 31923
    hostPort: 8080
  extraMounts:
  - hostPath: $PWD/.ssl/root-ca.pem
    containerPath: /opt/ca-certificates/root-ca.pem
    readOnly: true
- role: worker
  extraMounts:
  - hostPath: $PWD/.ssl/root-ca.pem
    containerPath: /opt/ca-certificates/root-ca.pem
    readOnly: true

Possible Cause

It seems the root CA is not being trusted on the Workflow pod, even though it should be available to all nodes.

Could the Workflow pod have a separate trust store that needs the CA added? Or is there a problem mounting the node CAs into the Workflow pod?

Any ideas appreciated on how to resolve the TLS verification errors when using private registries with Argo Workflows. Thanks!

What do you see instead?

tls: failed to verify certificate: x509: certificate signed by unknown authority

Architecture Graph

graph TD
    subgraph Kubernetes Cluster
    direction TB

    CA[Root CA] --> CI(cert-manager ClusterIssuer)

    CI --> HCert(Harbor Certificate)
    CI --> ACert(Argo Certificate)

    HCert --> Harbor

    ACert --> Argo

    DockerSecret[Docker Secret] --> Argo

    Argo --> Workflow

    Workflow -.->|tls verify failed| Harbor

    end

    classDef grey fill:#ddd,stroke:#fff,stroke-width:2px;
    class CA,DockerSecret grey

EDIT:

Mounting a volume with my CA in /etc/ssl/certs in the argo controller & server did fix a first x509 error (the first one used to crash at the start of the init). But I still got a x509 error.

Here is my updated argo values for the volume:

controller:
  workflowNamespaces:
    - default
    - pipeline
  extraVolumeMounts:
    - name: certificate
      mountPath: /etc/ssl/certs/rootCA.pem # your self-signed CA part inside the secret
      subPath: ca-cert
  extraVolumes:
    - name: certificate
      configMap:
        name: custom-root-cert
server:
  extraArgs: 
    - --auth-mode=server #TODO: Fix this and use bearer token instead
  extraVolumeMounts:
    - name: certificate
      mountPath: /etc/ssl/certs/rootCA.pem # your self-signed CA part inside the secret
      subPath: ca-cert
  extraVolumes:
    - name: certificate
      configMap:
        name: custom-root-cert

Note that the issues also appear when using kubectl run test --image=$IMAGE_FROM_HARBOR. This might be a kubernetes/harbor error

[julien-blanchon]

May be related with this https://github.com/argoproj/argo-workflows/issues/5582

[javsalgar]

Hi,

In the case of argo-workflows we don't any special logic in the Bitnami packaging, so I believe it should be argo-workflows team the ones that should advise on how to fix this issue. If there's anything we need to change in the chart to allow any extra settting, we will work on that.

[julien-blanchon]

Thanks for you answert @javsalgar !

Yes, it's more a kubernetes issues.

I did fix this by mounting my ca in each node with

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 31923
    hostPort: 8080
  # mount our root certificate in a separate folder
  extraMounts:
  - hostPath: ./kubernetes/.ssl/root-ca.pem
    containerPath: /opt/ca-certificates/root-ca.pem
    readOnly: true
  - hostPath: ./kubernetes/.ssl/root-ca.pem
    containerPath: /etc/ssl/certs/ca-certificates.crt
    readOnly: true
  - containerPath: /usr/local/share/ca-certificates/mon-autorite.crt
    hostPath: ./kubernetes/.ssl/root-ca.pem
    readOnly: true
  - containerPath: /etc/docker/certs.d/artefacts.robert.local/mon-autorite.crt
    hostPath: ./kubernetes/.ssl/root-ca.pem
    readOnly: true
- role: worker
  # mount our root certificate in a separate folder
  extraMounts:
  - hostPath: ./kubernetes/.ssl/root-ca.pem
    containerPath: /opt/ca-certificates/root-ca.pem
    readOnly: true
  - hostPath: ./kubernetes/.ssl/root-ca.pem
    containerPath: /etc/ssl/certs/ca-certificates.crt
    readOnly: true
  - containerPath: /usr/local/share/ca-certificates/mon-autorite.crt
    hostPath: ./kubernetes/.ssl/root-ca.pem
    readOnly: true
  - containerPath: /etc/docker/certs.d/artefacts.robert.local/mon-autorite.crt
    hostPath: ./kubernetes/.ssl/root-ca.pem
    readOnly: true

Thanks anyway, I'm closing the isue