search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.8k stars 5.43k forks source link

High Memory Usage and Pod Evictions in ArgoCD - Seeking Solutions and Recommendations #16308

Open cubxxw opened 11 months ago

cubxxw commented 11 months ago

I am experiencing significant memory usage issues with ArgoCD in my Kubernetes environment, leading to multiple pod evictions. I am seeking guidance and recommendations on how to address this issue effectively.

Issue Description:

I've noticed that the argocd-dex-server and argocd-repo-server pods are being repeatedly evicted due to high memory consumption. The main error message is related to the node being low on resources, specifically ephemeral storage. Here is an excerpt from the Kubernetes logs:

root@openim-super:~# kubectl get pod -A  | grep argocd
argocd        argocd-application-controller-0                             1/1     Running                       0          68m
argocd        argocd-applicationset-controller-576988d556-xbkct           1/1     Running                       0          69m
argocd        argocd-dex-server-85f464cb94-2kpqp                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-4x8tk                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-4xpqw                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-5d8mz                          0/1     Error                         0          21m
argocd        argocd-dex-server-85f464cb94-6cvpg                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-6qj2j                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-7bbqk                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-7kjcw                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-7mfdd                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-7n4dj                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-7ttp8                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-7vf88                          0/1     ContainerStatusUnknown        1          29m
argocd        argocd-dex-server-85f464cb94-7z5g6                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-8hdxj                          0/1     ContainerStatusUnknown        1          11m
argocd        argocd-dex-server-85f464cb94-8lbfs                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-8w4ms                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-9jtzg                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-9rwlh                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-9vbhr                          0/1     Error                         0          69m
argocd        argocd-dex-server-85f464cb94-b5m7g                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-b88lb                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-bhvs4                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-chrrz                          0/1     Pending                       0          6m4s
argocd        argocd-dex-server-85f464cb94-df8wm                          0/1     Error                         0          66m
argocd        argocd-dex-server-85f464cb94-dhnpf                          0/1     PodInitializing               0          43m
argocd        argocd-dex-server-85f464cb94-f6dkq                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-fm9bz                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-fv7xh                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-fz6gg                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-gr7nq                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-gwz4n                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-hjgqf                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-hmlbn                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-hstzb                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-htfqb                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-hznsh                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-jdx4j                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-jnwh8                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-jx969                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-kgmhj                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-ksw6t                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-kv27r                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-ldbff                          0/1     Error                         0          51m
argocd        argocd-dex-server-85f464cb94-lknvd                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-mdhw5                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-mwcfs                          0/1     Error                         0          59m
argocd        argocd-dex-server-85f464cb94-mzfch                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-p86xv                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-pb9q4                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-pdzbm                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-pjcl2                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-q79nd                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-r5b4w                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-r9r84                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-rnzrz                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-rrj9m                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-rzgjh                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-sh6nj                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-stbr8                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-sxrbz                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-tzlvg                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-w5c9s                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-w829l                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-wcxcv                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-x6zlv                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-z2dh2                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-z5dxt                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-z7rfc                          0/1     Error                         0          35m
argocd        argocd-dex-server-85f464cb94-zhlzn                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-zlj9h                          0/1     Evicted                       0          21m
argocd        argocd-dex-server-85f464cb94-zqgtp                          0/1     Evicted                       0          21m
argocd        argocd-notifications-controller-6b958bc7b9-ffnkt            1/1     Running                       0          69m
argocd        argocd-redis-7d8d46cc7f-jk7qr                               1/1     Running                       0          69m
argocd        argocd-repo-server-65596cfbcc-227rn                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-249hv                         0/1     Error                         0          20m
argocd        argocd-repo-server-65596cfbcc-2g7lc                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-2trrn                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-4thfw                         0/1     Evicted                       0          13m
argocd        argocd-repo-server-65596cfbcc-5qbz4                         0/1     Error                         0          30m
argocd        argocd-repo-server-65596cfbcc-74tzl                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-765zd                         0/1     Pending                       0          6m17s
argocd        argocd-repo-server-65596cfbcc-7gd95                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-7tg2r                         0/1     Evicted                       0          13m
argocd        argocd-repo-server-65596cfbcc-84n2c                         0/1     Evicted                       0          13m
argocd        argocd-repo-server-65596cfbcc-8bkjx                         0/1     Evicted                       0          13m
argocd        argocd-repo-server-65596cfbcc-8dm7s                         0/1     Evicted                       0          13m
argocd        argocd-repo-server-65596cfbcc-8wfcr                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-8wkqq                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-9zmcw                         0/1     Error                         0          59m
argocd        argocd-repo-server-65596cfbcc-bcdz7                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-bwnjf                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-c42pr                         0/1     Evicted                       0          13m
argocd        argocd-repo-server-65596cfbcc-c7pfc                         0/1     Evicted                       0          13m
argocd        argocd-repo-server-65596cfbcc-fmk98                         0/1     ContainerStatusUnknown        1          51m
argocd        argocd-repo-server-65596cfbcc-gcmgf                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-hb9qq                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-jclrn                         0/1     Evicted                       0          13m
argocd        argocd-repo-server-65596cfbcc-k6kg6                         0/1     Error                         0          66m
argocd        argocd-repo-server-65596cfbcc-k9djp                         0/1     Evicted                       0          13m
argocd        argocd-repo-server-65596cfbcc-kv2xr                         0/1     ContainerStatusUnknown        1          43m
argocd        argocd-repo-server-65596cfbcc-l9vlt                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-mh76w                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-mjszg                         0/1     Init:ContainerStatusUnknown   1          69m
argocd        argocd-repo-server-65596cfbcc-mplrf                         0/1     Error                         0          35m
argocd        argocd-repo-server-65596cfbcc-n55zk                         0/1     Evicted                       0          13m
argocd        argocd-repo-server-65596cfbcc-p6xkj                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-rqwkw                         0/1     Evicted                       0          13m
argocd        argocd-repo-server-65596cfbcc-s9rmn                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-szn65                         0/1     Evicted                       0          13m
argocd        argocd-repo-server-65596cfbcc-t66jk                         0/1     Error                         0          12m
argocd        argocd-repo-server-65596cfbcc-v2lgv                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-vxnzg                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-x4x9t                         0/1     Evicted                       0          12m
argocd        argocd-repo-server-65596cfbcc-xlw59                         0/1     Evicted                       0          12m
argocd        argocd-server-c4558786-flpkc                                0/1     ContainerStatusUnknown        1          69m
argocd        argocd-server-c4558786-mkqjz                                1/1     Running                       0          66m
root@openim-super:~# kubectl describe pod argocd-repo-server-65596cfbcc-9zmcw
Error from server (NotFound): pods "argocd-repo-server-65596cfbcc-9zmcw" not found
root@openim-super:~# kubectl describe pod argocd-repo-server-65596cfbcc-9zmcw -n argocd
Name:             argocd-repo-server-65596cfbcc-9zmcw
Namespace:        argocd
Priority:         0
Service Account:  argocd-repo-server
Node:             192.168.0.160/192.168.0.160
Start Time:       Mon, 13 Nov 2023 11:30:12 +0800
Labels:           app.kubernetes.io/name=argocd-repo-server
                  pod-template-hash=65596cfbcc
Annotations:      <none>
Status:           Failed
Reason:           Evicted
Message:          The node was low on resource: ephemeral-storage. Threshold quantity: 6293220806, available: 5752228Ki. 
IP:               172.20.189.199
IPs:
  IP:           172.20.189.199
Controlled By:  ReplicaSet/argocd-repo-server-65596cfbcc
Init Containers:
  copyutil:
    Container ID:  containerd://8cd7d7798cc39f2a39f01f649c85dd6bc7a743143d33986f22c118431a510365
    Image:         quay.io/argoproj/argocd:v2.9.0
    Image ID:      quay.io/argoproj/argocd@sha256:1be54db1f0f5d1c0da4c9b592f8ee1921aa7b5441a780dd90ca2cad135131816
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/cp
      -n
      /usr/local/bin/argocd
      /var/run/argocd/argocd-cmp-server
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 13 Nov 2023 11:30:13 +0800
      Finished:     Mon, 13 Nov 2023 11:30:13 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/argocd from var-files (rw)
Containers:
  argocd-repo-server:
    Container ID:  containerd://90410c6c3ffdfc1f1023e4aa0f455b1d2a8c433657ffd30b9fac3dedc4d4588b
    Image:         quay.io/argoproj/argocd:v2.9.0
    Image ID:      quay.io/argoproj/argocd@sha256:1be54db1f0f5d1c0da4c9b592f8ee1921aa7b5441a780dd90ca2cad135131816
    Ports:         8081/TCP, 8084/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      /usr/local/bin/argocd-repo-server
    State:          Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Mon, 13 Nov 2023 11:32:13 +0800
      Finished:     Mon, 13 Nov 2023 11:32:13 +0800
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:8084/healthz%3Ffull=true delay=30s timeout=5s period=30s #success=1 #failure=3
    Readiness:      http-get http://:8084/healthz delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      ARGOCD_RECONCILIATION_TIMEOUT:                                <set to the key 'timeout.reconciliation' of config map 'argocd-cm'>                                          Optional: true
      ARGOCD_REPO_SERVER_LOGFORMAT:                                 <set to the key 'reposerver.log.format' of config map 'argocd-cmd-params-cm'>                                Optional: true
      ARGOCD_REPO_SERVER_LOGLEVEL:                                  <set to the key 'reposerver.log.level' of config map 'argocd-cmd-params-cm'>                                 Optional: true
      ARGOCD_REPO_SERVER_PARALLELISM_LIMIT:                         <set to the key 'reposerver.parallelism.limit' of config map 'argocd-cmd-params-cm'>                         Optional: true
      ARGOCD_REPO_SERVER_LISTEN_ADDRESS:                            <set to the key 'reposerver.listen.address' of config map 'argocd-cmd-params-cm'>                            Optional: true
      ARGOCD_REPO_SERVER_LISTEN_METRICS_ADDRESS:                    <set to the key 'reposerver.metrics.listen.address' of config map 'argocd-cmd-params-cm'>                    Optional: true
      ARGOCD_REPO_SERVER_DISABLE_TLS:                               <set to the key 'reposerver.disable.tls' of config map 'argocd-cmd-params-cm'>                               Optional: true
      ARGOCD_TLS_MIN_VERSION:                                       <set to the key 'reposerver.tls.minversion' of config map 'argocd-cmd-params-cm'>                            Optional: true
      ARGOCD_TLS_MAX_VERSION:                                       <set to the key 'reposerver.tls.maxversion' of config map 'argocd-cmd-params-cm'>                            Optional: true
      ARGOCD_TLS_CIPHERS:                                           <set to the key 'reposerver.tls.ciphers' of config map 'argocd-cmd-params-cm'>                               Optional: true
      ARGOCD_REPO_CACHE_EXPIRATION:                                 <set to the key 'reposerver.repo.cache.expiration' of config map 'argocd-cmd-params-cm'>                     Optional: true
      REDIS_SERVER:                                                 <set to the key 'redis.server' of config map 'argocd-cmd-params-cm'>                                         Optional: true
      REDIS_COMPRESSION:                                            <set to the key 'redis.compression' of config map 'argocd-cmd-params-cm'>                                    Optional: true
      REDISDB:                                                      <set to the key 'redis.db' of config map 'argocd-cmd-params-cm'>                                             Optional: true
      ARGOCD_DEFAULT_CACHE_EXPIRATION:                              <set to the key 'reposerver.default.cache.expiration' of config map 'argocd-cmd-params-cm'>                  Optional: true
      ARGOCD_REPO_SERVER_OTLP_ADDRESS:                              <set to the key 'otlp.address' of config map 'argocd-cmd-params-cm'>                                         Optional: true
      ARGOCD_REPO_SERVER_MAX_COMBINED_DIRECTORY_MANIFESTS_SIZE:     <set to the key 'reposerver.max.combined.directory.manifests.size' of config map 'argocd-cmd-params-cm'>     Optional: true
      ARGOCD_REPO_SERVER_PLUGIN_TAR_EXCLUSIONS:                     <set to the key 'reposerver.plugin.tar.exclusions' of config map 'argocd-cmd-params-cm'>                     Optional: true
      ARGOCD_REPO_SERVER_ALLOW_OUT_OF_BOUNDS_SYMLINKS:              <set to the key 'reposerver.allow.oob.symlinks' of config map 'argocd-cmd-params-cm'>                        Optional: true
      ARGOCD_REPO_SERVER_STREAMED_MANIFEST_MAX_TAR_SIZE:            <set to the key 'reposerver.streamed.manifest.max.tar.size' of config map 'argocd-cmd-params-cm'>            Optional: true
      ARGOCD_REPO_SERVER_STREAMED_MANIFEST_MAX_EXTRACTED_SIZE:      <set to the key 'reposerver.streamed.manifest.max.extracted.size' of config map 'argocd-cmd-params-cm'>      Optional: true
      ARGOCD_REPO_SERVER_HELM_MANIFEST_MAX_EXTRACTED_SIZE:          <set to the key 'reposerver.helm.manifest.max.extracted.size' of config map 'argocd-cmd-params-cm'>          Optional: true
      ARGOCD_REPO_SERVER_DISABLE_HELM_MANIFEST_MAX_EXTRACTED_SIZE:  <set to the key 'reposerver.disable.helm.manifest.max.extracted.size' of config map 'argocd-cmd-params-cm'>  Optional: true
      ARGOCD_GIT_MODULES_ENABLED:                                   <set to the key 'reposerver.enable.git.submodule' of config map 'argocd-cmd-params-cm'>                      Optional: true
      HELM_CACHE_HOME:                                              /helm-working-dir
      HELM_CONFIG_HOME:                                             /helm-working-dir
      HELM_DATA_HOME:                                               /helm-working-dir
    Mounts:
      /app/config/gpg/keys from gpg-keyring (rw)
      /app/config/gpg/source from gpg-keys (rw)
      /app/config/reposerver/tls from argocd-repo-server-tls (rw)
      /app/config/ssh from ssh-known-hosts (rw)
      /app/config/tls from tls-certs (rw)
      /helm-working-dir from helm-working-dir (rw)
      /home/argocd/cmp-server/plugins from plugins (rw)
      /tmp from tmp (rw)
Conditions:
  Type               Status
  DisruptionTarget   True 
  Initialized        True 
  Ready              False 
  ContainersReady    False 
  PodScheduled       True 
Volumes:
  ssh-known-hosts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      argocd-ssh-known-hosts-cm
    Optional:  false
  tls-certs:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      argocd-tls-certs-cm
    Optional:  false
  gpg-keys:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      argocd-gpg-keys-cm
    Optional:  false
  gpg-keyring:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  helm-working-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  argocd-repo-server-tls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  argocd-repo-server-tls
    Optional:    true
  var-files:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  plugins:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      
    SizeLimit:   <unset>
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason               Age                From               Message
  ----     ------               ----               ----               -------
  Warning  FailedScheduling     54m (x2 over 60m)  default-scheduler  0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
  Normal   Scheduled            54m                default-scheduler  Successfully assigned argocd/argocd-repo-server-65596cfbcc-9zmcw to 192.168.0.160
  Normal   Pulled               54m                kubelet            Container image "quay.io/argoproj/argocd:v2.9.0" already present on machine
  Normal   Created              54m                kubelet            Created container copyutil
  Normal   Started              54m                kubelet            Started container copyutil
  Normal   Pulling              54m                kubelet            Pulling image "quay.io/argoproj/argocd:v2.9.0"
  Warning  Evicted              52m                kubelet            The node was low on resource: ephemeral-storage. Threshold quantity: 6293220806, available: 5752228Ki.
  Warning  ExceededGracePeriod  52m                kubelet            Container runtime did not kill the pod within specified grace period.
  Normal   Pulled               52m                kubelet            Successfully pulled image "quay.io/argoproj/argocd:v2.9.0" in 243.294561ms (1m58.08659195s including waiting)
  Normal   Created              52m                kubelet            Created container argocd-repo-server
  Normal   Started              52m                kubelet            Started container argocd-repo-server
  Normal   Killing              52m                kubelet            Stopping container argocd-repo-server

Questions:

  1. What could be causing this high memory usage in ArgoCD?
  2. Are there any recommended best practices for resource allocation and management for ArgoCD in a Kubernetes environment?
  3. Are there configuration changes that could help mitigate this issue?
  4. Any other suggestions or advice would be greatly appreciated.

I am looking forward to your guidance on resolving these memory-related issues with ArgoCD.

Thank you for your assistance.

Version

Paste the output from `argocd version` here.
not install

Logs

Paste any relevant application logs here.
cubxxw commented 11 months ago 
root@openim-super:~# df -h | grep argo
tmpfs                                                                                                      63G     0   63G   0% /var/lib/kubelet/pods/bf469cbc-52fc-452b-a425-bdda893e584f/volumes/kubernetes.io~secret/argocd-repo-server-tls
tmpfs                                                                                                      63G     0   63G   0% /var/lib/kubelet/pods/ae8c92aa-9c5a-4cd8-a69b-a0b48a567130/volumes/kubernetes.io~secret/argocd-repo-server-tls
tmpfs                                                                                                      63G     0   63G   0% /var/lib/kubelet/pods/21e6d02b-7bee-4f58-9e13-80ddb7336143/volumes/kubernetes.io~secret/argocd-repo-server-tls
tmpfs                                                                                                      63G     0   63G   0% /var/lib/kubelet/pods/4e8fde05-b834-407b-8f93-89d2005c5a71/volumes/kubernetes.io~secret/argocd-dex-server-tls
tmpfs                                                                                                      63G     0   63G   0% /var/lib/kubelet/pods/4e8fde05-b834-407b-8f93-89d2005c5a71/volumes/kubernetes.io~secret/argocd-repo-server-tls
Puvendhan commented 3 months ago

i am also facing same issue, how's this been fixed?

Btijmen commented 2 months ago

I'm observing the same behaviour in an ArgoCD instance created using the community-maintained operator

k8s version v1.28.2 operator version 0.11.0

$ argocd version
argocd: v2.11.7+e4a0246
  BuildDate: 2024-07-24T14:02:54Z
  GitCommit: e4a0246c4d920bc1e5ee5f9048a99eca7e1d53cb
  GitTreeState: clean
  GoVersion: go1.22.5
  Compiler: gc
  Platform: darwin/arm64
cubxxw commented 2 months ago

good

cubxxw commented 2 months ago

@alexmt

PhoduCoder commented 1 month ago

Can you try adding persistent volume to your repo server. What is the total number of applications that your Argocd services? https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/

argocd-repo-server clones the repository into /tmp (or the path specified in the TMPDIR env variable). The Pod might run out of disk space if it has too many repositories or if the repositories have a lot of files. To avoid this problem mount a persistent volume.