Failed to pull image "ghcr.io/dexidp/dex:v2.35.3": rpc error: code = Unknown desc = Error response from daemon: Get "https://ghcr.io/v2/": net/http: TLS handshake timeout

[ernestang98]

Following this and many tutorials, I attempt to install argocd on minikube with a MacOS by running the following commands:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

However, I encounter the following error with the dex-server as a result:

kubectl get all -n argocd

NAME                                                   READY   STATUS         RESTARTS   AGE
pod/argocd-application-controller-0                    1/1     Running        0          3m3s
pod/argocd-applicationset-controller-5fff78cd6-grq9w   1/1     Running        0          3m5s
pod/argocd-dex-server-5f9ffd8b4b-tmnzc                 0/1     ErrImagePull   0          3m5s
pod/argocd-notifications-controller-dfcc96fdd-5vqnw    1/1     Running        0          3m4s
pod/argocd-redis-7b59bffbff-ljn9r                      1/1     Running        0          3m4s
pod/argocd-repo-server-6b9585d98b-g9n7d                1/1     Running        0          3m3s
pod/argocd-server-645746d59-ts5s8                      1/1     Running        0          3m3s
...

Running kubectl describe pod/argocd-dex-server-5f9ffd8b4b-tmnzc -n argocd, we observe:

  Normal   Scheduled    3m37s                default-scheduler  Successfully assigned argocd/argocd-dex-server-5f9ffd8b4b-tmnzc to minikube
  Warning  FailedMount  3m36s                kubelet            MountVolume.SetUp failed for volume "argocd-dex-server-tls" : failed to sync secret cache: timed out waiting for the condition
  Normal   Pulling      3m31s                kubelet            Pulling image "quay.io/argoproj/argocd:v2.6.3"
  Normal   Pulled       3m22s                kubelet            Successfully pulled image "quay.io/argoproj/argocd:v2.6.3" in 3.3846098s (9.0583385s including waiting)
  Normal   Created      3m21s                kubelet            Created container copyutil
  Normal   Started      3m21s                kubelet            Started container copyutil
  Normal   BackOff      74s (x4 over 2m41s)  kubelet            Back-off pulling image "ghcr.io/dexidp/dex:v2.35.3"
  Warning  Failed       74s (x4 over 2m41s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling      61s (x4 over 3m20s)  kubelet            Pulling image "ghcr.io/dexidp/dex:v2.35.3"
  Warning  Failed       51s (x4 over 2m42s)  kubelet            Failed to pull image "ghcr.io/dexidp/dex:v2.35.3": rpc error: code = Unknown desc = Error response from daemon: Get "https://ghcr.io/v2/": net/http: TLS handshake timeout
  Warning  Failed       51s (x4 over 2m42s)  kubelet            Error: ErrImagePull

Running kubectl get pod/argocd-dex-server-5f9ffd8b4b-tmnzc -o yaml -n argocd | grep image, we observe

kubectl get pod/argocd-dex-server-5f9ffd8b4b-tmnzc -o yaml -n argocd | grep image
    image: ghcr.io/dexidp/dex:v2.35.3
    imagePullPolicy: Always
    image: quay.io/argoproj/argocd:v2.6.3
    imagePullPolicy: Always
  - image: ghcr.io/dexidp/dex:v2.35.3
    imageID: ""
        message: Back-off pulling image "ghcr.io/dexidp/dex:v2.35.3"
    image: quay.io/argoproj/argocd:v2.6.3
    imageID: docker-pullable://quay.io/argoproj/argocd@sha256:0fd690bd7b89bd6f947b4000de33abd53ebcd36b57216f1c675a1127707b5eef

Changing the image to dexidp/dex:v2.27.0 seemed to fix it, maybe it is an issue with ghcr.io?

[msandersfullstaq]

I also have the same issue, is there any update on this, other than go back to image dexidp/dex:v2.27.0 which does works

Normal Pulling 14m kubelet Pulling image "quay.io/argoproj/argocd:v2.6.5" Normal Pulled 14m kubelet Successfully pulled image "quay.io/argoproj/argocd:v2.6.5" in 1.258199581s Normal Pulling 12m (x4 over 14m) kubelet Pulling image "ghcr.io/dexidp/dex:v2.35.3" Warning Failed 12m (x4 over 14m) kubelet Failed to pull image "ghcr.io/dexidp/dex:v2.35.3": rpc error: code = Unknown desc = Error response from daemon: Head "https://ghcr.io/v2/dexidp/dex/manifests/v2.35.3": received unexpected HTTP status: 503 Service Unavailable Normal BackOff 12m (x5 over 14m) kubelet Back-off pulling image "ghcr.io/dexidp/dex:v2.35.3" Warning Failed 9m30s (x4 over 11m) kubelet Failed to pull image "ghcr.io/dexidp/dex:v2.35.2": rpc error: code = Unknown desc = Error response from daemon: Head "https://ghcr.io/v2/dexidp/dex/manifests/v2.35.2": received unexpected HTTP status: 503 Service Unavailable

[cjc7373]

This is definitely Github's issues. Check out https://www.githubstatus.com/ for any incidents currently happening.

[fredleger]

Having the same issue but GH is 100% up

Some context:

• using macbook pro silicon / M1
• i can confirm i can pull the image using docker
• argocd is installed on rancher-desktop version 1.8.1 (and the docker command uses bundled docker)

The actual error is :

kubectl get events -n argocd

4m14s       Warning   Failed              pod/argocd-dex-server-b694449c5-spzc7                    Failed to pull image "ghcr.io/dexidp/dex:v2.37.0": rpc error: code = Unknown desc = Error response from daemon: Head "https://ghcr.io/v2/dexidp/dex/manifests/v2.37.0": denied: denied

feel free to ask any details

[thirumurthis]

Observed the same behavior when installing argocd in KinD cluster. Was able to download the image using docker pull. As a workaround, edited the argocd manifest in local and updated the argocd-dex-server deployment imagePullPolicy from Always to IfNotPresent. Deployed updated manifest, and manually uploaded the image to kind cluster using kind load docker-image ghcr.io/dexidp/dex:v2.38.0 --name <kind-cluster-name>.

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  14m                   default-scheduler  Successfully assigned argocd/argocd-dex-server-7b6987df7-gt94p to argocd-control-plane
  Warning  Failed     13m                   kubelet            Failed to pull image "quay.io/argoproj/argocd:v2.12.3": failed to pull and unpack image "quay.io/argoproj/argocd:v2.12.3": failed to copy: httpReadSeeker: failed open: unexpected status code https://quay.io/v2/argoproj/argocd/blobs/sha256:86749b599e3ed5b6879a0a72170d9585ed2876e43825874faea03d6ed12437ba: 502 Bad Gateway
  Warning  Failed     13m                   kubelet            Error: ErrImagePull
  Normal   BackOff    13m                   kubelet            Back-off pulling image "quay.io/argoproj/argocd:v2.12.3"
  Warning  Failed     13m                   kubelet            Error: ImagePullBackOff
  Normal   Pulling    12m (x2 over 14m)     kubelet            Pulling image "quay.io/argoproj/argocd:v2.12.3"
  Normal   Pulled     11m                   kubelet            Successfully pulled image "quay.io/argoproj/argocd:v2.12.3" in 8.682s (56.372s including waiting). Image size: 171926860 bytes.
  Normal   Created    11m                   kubelet            Created container copyutil
  Normal   Started    11m                   kubelet            Started container copyutil
  Normal   Pulling    10m (x3 over 11m)     kubelet            Pulling image "ghcr.io/dexidp/dex:v2.38.0"
  Warning  Failed     9m56s (x3 over 10m)   kubelet            Failed to pull image "ghcr.io/dexidp/dex:v2.38.0": failed to pull and unpack image "ghcr.io/dexidp/dex:v2.38.0": failed to resolve reference "ghcr.io/dexidp/dex:v2.38.0": failed to do request: Head "https://ghcr.io/v2/dexidp/dex/manifests/v2.38.0": tls: failed to verify certificate: x509: certificate signed by unknown authority
  Warning  Failed     9m56s (x3 over 10m)   kubelet            Error: ErrImagePull
  Warning  Failed     9m18s (x4 over 10m)   kubelet            Error: ImagePullBackOff
  Normal   BackOff    4m28s (x22 over 10m)  kubelet            Back-off pulling image "ghcr.io/dexidp/dex:v2.38.0"