Randomly apps turn to Unknown status with "context deadline exceeded" and go back and forth between Unknown and Sync irregularly

[Masakinpo]

Describe the bug Suddenly, apps turns to Unknown status regardless of whether manifests update or not. This happens to all apps at random. After a while, the apps with Unknown status go back to Synced. Going back and forth between Unknown and Synced status happen irregularly to any apps at random.

To Reproduce

Don't know why this happens, so no idea how it reproduces.

Expected behavior

App's status keep synced.

Screenshots

Version

k8s: v1.20.7 (eks)

argocd@argo-cd-argocd-server-79f9b854df-dvbqn:~$ argocd  version
argocd: v2.0.3+8d2b13d
  BuildDate: 2021-05-27T17:38:37Z
  GitCommit: 8d2b13d733e1dff7d1ad2c110ed31be4804406e2
  GitTreeState: clean
  GoVersion: go1.16
  Compiler: gc
  Platform: linux/amd64
FATA[0000] Argo CD server address unspecified

Logs

 time="2021-10-27T02:04:41Z" level=error msg="finished unary call with code Unknown" error="context deadline exceeded (Client.Timeout or context cancellation while reading body)" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2021-10-27T02:05:26Z" grpc.service=repository.RepoServerService grpc.start_time="2021-10-27T02:04:26Z" grpc.time_ms=15000.389 span.kind=server system=grpc

remarks I've already tried the following.

• Successfully clone github repo within argo-cd-argocd-repo-server within 15 seconds (it took around 10 seconds tho)
• Augmented repo-server spec and it has enough spec for sure
• num of applications is less than 100 including workflows

It has worked well, but suddenly this unknown status flood happened last weekend 😢

[ismailyenigul]

It seems this happens when there is a performance issue at Github. I experienced this issue today for 3 different argo deployment on different clusters and accounts almost at the same time. There was issues at Github https://www.githubstatus.com/incidents/zhtplv7zd052

[Vinaum8]

same error on version 2.8.2 of argocd.

i am not cloning repositories from github, i am cloning repo from Azure DevOps