tls errors when pulling image, onprem harbor oci helm & image repo, local certificate authority

[lknite]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

In the past it worked just to mount the the ca.crt into the /etc/ssl/certs path, but this stopped working so I went ahead and added the cert k.home.net via the gui. If I exec into a pod and run 'openssl s_client' they all show 'Verification: OK', yet not able to pull.

To Reproduce

1. create helm chart and store in local harbor oci repo, tls certs generated using cert-manager w/ local certificate authority
2. create image and store in local harbor repo (the helm chart includes multiple helm chart dependencies, each of those references an image in the repo)
3. add local certificate authority public cert 'k.home.net' to argocd via gui
4. k apply applicationset which defines source as the helm chart in the onprem harbor oci repo
5. watch as images fail to pull

Expected behavior

normal deployment of helm chart and associated images

Version

v2.11.0+d3f33c0

Logs

$ openssl s_client -debug -showcerts -CAfile /app/config/tls/k.home.net -connect harbor.vc-prod.k.home.net:443 | grep Veri
depth=1 CN = k.home.net
verify return:1
depth=0 
verify return:1
Verification: OK
Verify return code: 0 (ok)

pod/list-controller-blocks-8bb466d6c-hrb79   0/1     ImagePullBackOff   0          16h
pod/list-api-blocks-7cf455b9d-8cv25          0/1     ImagePullBackOff   0          15m
pod/list-api-lists-6c66d94ddc-tcmqp          0/1     ImagePullBackOff   0          16h
pod/list-www-58bcd759ff-fpkqz                0/1     ImagePullBackOff   0          16h
pod/list-controller-lists-555bd74fb7-9ncb5   0/1     ImagePullBackOff   0          16h
pod/list-websocket-7fcf75bb9d-flpft          0/1     ImagePullBackOff   0          16h

Failed to pull image "harbor.vc-prod.k.home.net/list/api-blocks:0.0.9": failed to pull and unpack image "harbor.vc-prod.k.home.net/list/api-blocks:0.0.9": failed to resolve reference "harbor.vc-prod.k.home.net/list/api-blocks:0.0.9": failed to do request: Head "https://harbor.vc-prod.k.home.net/v2/list/api-blocks/manifests/0.0.9": tls: failed to verify certificate: x509: certificate signed by unknown authority

from my local linux box:

$ docker pull harbor.vc-prod.k.home.net/list/api-blocks:0.0.9
0.0.9: Pulling from list/api-blocks
4abcf2066143: Pull complete 
891ce9574a73: Pull complete 
e425669e0253: Pull complete 
d01aebfa4c88: Pull complete 
4f4fb700ef54: Pull complete 
eb21768a585e: Pull complete 
Digest: sha256:efc8eb477445c0a9d9335cabc6d907a324c1f3ea73af4282c20ac6758464f702
Status: Downloaded newer image for harbor.vc-prod.k.home.net/list/api-blocks:0.0.9
harbor.vc-prod.k.home.net/list/api-blocks:0.0.9

[lknite]

@sabre1041 Is this issue possibly a missed use case related to your fix? https://github.com/argoproj/argo-cd/pull/16656

[sabre1041]

@lknite the issue that you are facing is unrelated to Argo CD as the errors are coming from the underlying container runtime that is attempting to retrieve the container image that is specified (most likely) by the manifests associated within Argo CD. Additional configurations need to be made within the container runtime level which is outside the scope of Argo CD.

[lknite]

@sabre1041 Oh! I got it! Thank you!

(worker nodes need the public ca)