v2.3.4 throws context deadline exceeded (Client.Timeout exceeded while awaiting headers)

[ccfishk]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x ] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

command "argocd app create istio --repo https://github.intuit.com/services-mesh/services-mesh-release.git" throws repository not accessible

To Reproduce

On an EKS cluster after verifying the GitHub is reachable

1. kubectl create namespace argocd
2. kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
3. kubectl port-forward svc/argocd-server -n argocd 8080:443
4. argocd login localhost:8080
5. argocd app create istio --repo https://github.intuit.com/services-mesh/services-mesh-release.git --path istio/yamls --dest-server https://kubernetes.default.svc --dest-namespace default

Expected behavior

application is able to get created successfully

Screenshots

Version

argocd version argocd: v2.3.4+ac8b7df BuildDate: 2022-05-18T13:00:56Z GitCommit: ac8b7df9467ffcc0920b826c62c4b603a7bfed24 GitTreeState: clean GoVersion: go1.17.9 Compiler: gc Platform: darwin/amd64 argocd-server: v2.3.4+ac8b7df BuildDate: 2022-05-18T11:41:37Z GitCommit: ac8b7df9467ffcc0920b826c62c4b603a7bfed24 GitTreeState: clean GoVersion: go1.17.10 Compiler: gc Platform: linux/amd64 Ksonnet Version: v0.13.1 Kustomize Version: v4.4.1 2021-11-11T23:36:27Z Helm Version: v3.8.0+gd141386 Kubectl Version: v0.23.1 Jsonnet Version: v0.18.0

Logs

Paste any relevant application logs here.

[0015] rpc error: code = InvalidArgument desc = application spec for istio is invalid: InvalidSpecError: repository not accessible: rpc error: code = Unknown desc = error testing repository connectivity: Get "https://github.intuit.com/services-mesh/services-mesh-release.git/info/refs?service=git-upload-pack": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

[bluefishforsale]

+1 also having this issue using :

- name: argo-cd
    version: 4.7.0
    repository: https://argoproj.github.io/argo-helm

I have about 30 apps in Argo, and have increased the reposerver min count to 3 and enabled the HPA scaler

~feels~ like a github.com API rate limit vs argo retry timeout issue

[106062]

+1 also haveing same issue, after i update from version 2.2.x to 2.3.4. also tried to Update to 2.3.3, have same issue with git repo url using http/https.

But, having install ArgoCD 2.3.3 in a new cluster. sure 2.3.3 's http works fine.

[kraj007]

I am also facing the same issue for 'argocd repo add' command error : FATA[0162] rpc error: code = DeadlineExceeded desc = context deadline exceeded

argocd: v2.3.3+07ac038 BuildDate: 2022-03-30T01:46:59Z GitCommit: 07ac038a8f97a93b401e824550f0505400a8c84e GitTreeState: clean GoVersion: go1.17.6 Compiler: gc Platform: linux/amd64 argocd-server: v2.3.4+ac8b7df BuildDate: 2022-05-18T11:41:37Z GitCommit: ac8b7df9467ffcc0920b826c62c4b603a7bfed24 GitTreeState: clean GoVersion: go1.17.10 Compiler: gc Platform: linux/amd64 Ksonnet Version: v0.13.1 Kustomize Version: v4.4.1 2021-11-11T23:36:27Z Helm Version: v3.8.0+gd141386 Kubectl Version: v0.23.1 Jsonnet Version: v0.18.0

[dngferreira]

Version v2.4.0-rc5 has the same problem.

[ClemontX]

same problem here with different versions.

❯ argo repo add https://github.com/louislam/uptime-kuma.git
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
FATA[0019] rpc error: code = Unknown desc = error testing repository connectivity: Get "https://github.com/louislam/uptime-kuma.git/info/refs?service=git-upload-pack": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I'm running on a kubespray one-node "Cluster" with calico and tested the connection to github via a netshoot pod. A git clone worked inside this netshoot pod.

The interesting part: I spun up a kind-Cluster and deployed argocd in there and i was able to get a Repository connection and was able to browse branches and directories via the argocd-webinterface.

[yanghuijava]

I had the same problem with v2.3.4

[AnshumanTripathi]

Seeing similar issues with 2.4.2

{
    "Version": "v2.4.2+c6d0c8b",
    "BuildDate": "2022-06-21T20:42:05Z",
    "GitCommit": "c6d0c8baaa291cd68465acd7ad6bef58b2b6f942",
    "GitTreeState": "clean",
    "GoVersion": "go1.18.3",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v4.4.1 2021-11-11T23:36:27Z",
    "HelmVersion": "v3.8.1+g5cb9af4",
    "KubectlVersion": "v0.23.1",
    "JsonnetVersion": "v0.18.0"
}

[wgzhao]

I had the same promblem with V2.4.6, this problem is making me crack up.

argocd: v2.4.6+a48bca0
  BuildDate: 2022-07-12T22:56:26Z
  GitCommit: a48bca03c79b6d63be0c34d6094831bc6916b3bc
  GitTreeState: clean
  GoVersion: go1.18.3
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.4.6+a48bca0
  BuildDate: 2022-07-12T22:31:17Z
  GitCommit: a48bca03c79b6d63be0c34d6094831bc6916b3bc
  GitTreeState: clean
  GoVersion: go1.18.3
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.4.1 2021-11-11T23:36:27Z
  Helm Version: v3.8.1+g5cb9af4
  Kubectl Version: v0.23.1
  Jsonnet Version: v0.18.0

[phuongdkyt]

how to fix?

[jarrettprosser]

I'm seeing the same on v2.1.6. Interestingly, I can exec into the repo server pod and manually put the repo with git, but when I try the same in the server pod it times out.

[jmc000]

I had the same promblem with v2.4.7

[mdfaizsiddiqui]

I see the same warnings on argocd-repo-server pod's Liveness and Readiness probes -

3 Unhealthy: Liveness probe failed: Get "http://10.0.52.226:8084/healthz?full=true": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 
1 Unhealthy: Readiness probe failed: Get "http://10.0.52.226:8084/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

{
    "Version": "v2.5.10+d311fad",
    "BuildDate": "2023-02-02T15:10:21Z",
    "GitCommit": "d311fad538178ef97528edf7f09d1d5b57268f45",
    "GitTreeState": "clean",
    "GoVersion": "go1.18.10",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v4.5.7 2022-08-02T16:35:54Z",
    "HelmVersion": "v3.10.3+g835b733",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.18.0"
}

[cevatkerim]

same problem

[crenshaw-dev]

@mdfaizsiddiqui context deadline exceeded is a generic error message in golang. It can appear in many contexts. I'm guessing most folks here are getting that error when the repo-server tries to communicate with some git host.

[emirot]

@crenshaw-dev that's correct trying to connect to git. (argocd 2.4.11) Is there a way to increase that timeout, for instance by logging into the pod doing kubectl exec -it argocd-repo-server bash -n argocd I can clone a repo but it would fail in UI and I would get that in the log and popup.

time="2023-03-28T21:22:56Z" level=error msg="finished unary call with code Unknown" error="Get \"https://private-ghe.com/myorg/myrepo/info/refs?service=git-upload-pack\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2023-03-28T21:22:41Z" grpc.time_ms=15000.81 span.kind=server system=grpc

[lukaszkubisiak]

the same problem here. I'm behind the proxy. Maybe that's the issue? But connection to repository (in the UI under Settings | Repository) is successful.

[lukaszkubisiak]

This helped me: https://github.com/argoproj/argo-cd/issues/2954

[fredcsfaria]

same problem here. Some fix ?

[zodiac12k]

I also have the same problem.

[behniafb]

Still no one found a solution for this issue? Then how do you add a private repo to ArgoCD?

[myysophia]

I resolved this issue by scale up argocd-server.

[shay-ul]

Increasing memory requests and limtis for argocd-repo-server did the trick for me.

[bredamatt]

I am also having this same issue from a private EKS cluster, however isn't this just simply a networking issue? My repo has literally nothing in it atm. so not sure why it is not working.

[Venture200]

Hello everyone

I fixed the issue by increasing repo-server replicas to 2 and it started to work. Hopefully helps you.

Have a wonderful day!

EDIT: While adding second repo, got this error again