search

aws / amazon-cloudwatch-agent

CloudWatch Agent enables you to collect and export host-level metrics and logs on instances running Linux or Windows server.
MIT License
441 stars 201 forks source link

Docker container does not start in EKS using Fargate #580

Closed mrvisser closed 2 years ago

mrvisser commented 2 years ago

Describe the bug When deploying the agent container to EKS (1.23) using a fargate profile, it fails with this message:

  cloudwatch-agent:
    Container ID:   containerd://93bf59c3c798cc6f78343faa9cdbe7430900ab1102306f614c14e4969dc4c808
    Image:          amazon/cloudwatch-agent:1.247355.0b252062-amd64
    Image ID:       docker.io/amazon/cloudwatch-agent@sha256:80d37f8425f783872d68fe0eeb53b93a2e77777928556454f06b9502c5ec9835
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       StartError
      Message:      failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: failed to write "100000": write /sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/pod19ff292b-27c4-4d3e-85c7-6e19ca66b0cc/93bf59c3c798cc6f78343faa9cdbe7430900ab1102306f614c14e4969dc4c808/cpu.cfs_quota_us: invalid argument: unknown
      Exit Code:    128
      Started:      Wed, 31 Dec 1969 19:00:00 -0500
      Finished:     Mon, 29 Aug 2022 11:03:44 -0400

Steps to reproduce I followed the steps relating to the Fargate launch type here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup.html, using an EKS cluster on EKS 1.23. I haven't tried 1.22 at this point.

What did you expect to see? I'd have expected this container to start up fine.

What did you see instead? The error message.

What version did you use? Version: 1.247355.0b252062, 1.247355.0b252062-amd64 and 1.247355.0b252062-arm64

What config did you use? Config: N/A

Environment OS: EKS 1.23 on fargate

Additional context

Deployment spec:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cwagent-prometheus
  namespace: amazon-cloudwatch
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cwagent-prometheus
  template:
    metadata:
      labels:
        app: cwagent-prometheus
    spec:
      containers:
        - name: cloudwatch-agent
          image: amazon/cloudwatch-agent:1.247355.0b252062
          imagePullPolicy: Always
          resources:
            limits:
              cpu:  1000m
              memory: 1000Mi
            requests:
              cpu: 200m
              memory: 200Mi
          # Please don't change below envs
          env:
            - name: CI_VERSION
              value: "k8s/1.3.11"
            - name: RUN_IN_AWS
              value: "True"
          # Please don't change the mountPath
          volumeMounts:
            - name: prometheus-cwagentconfig
              mountPath: /etc/cwagentconfig
            - name: prometheus-config
              mountPath: /etc/prometheusconfig
      volumes:
        - name: prometheus-cwagentconfig
          configMap:
            name: prometheus-cwagentconfig
        - name: prometheus-config
          configMap:
            name: prometheus-config
      terminationGracePeriodSeconds: 60
      serviceAccountName: cwagent-prometheus

Failed pod description:

Name:                 cwagent-prometheus-6475d78897-jm22x
Namespace:            amazon-cloudwatch
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 fargate-ip-10-0-6-68.us-east-2.compute.internal/10.0.6.68
Start Time:           Mon, 29 Aug 2022 11:03:25 -0400
Labels:               app=cwagent-prometheus
                      eks.amazonaws.com/fargate-profile=main
                      pod-template-hash=6475d78897
Annotations:          CapacityProvisioned: 0.25vCPU 0.5GB
                      Logging: LoggingDisabled: LOGGING_CONFIGMAP_NOT_FOUND
                      kubernetes.io/psp: eks.privileged
Status:               Running
IP:                   10.0.6.68
IPs:
  IP:           10.0.6.68
Controlled By:  ReplicaSet/cwagent-prometheus-6475d78897
Containers:
  cloudwatch-agent:
    Container ID:   containerd://93bf59c3c798cc6f78343faa9cdbe7430900ab1102306f614c14e4969dc4c808
    Image:          amazon/cloudwatch-agent:1.247355.0b252062-amd64
    Image ID:       docker.io/amazon/cloudwatch-agent@sha256:80d37f8425f783872d68fe0eeb53b93a2e77777928556454f06b9502c5ec9835
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       StartError
      Message:      failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: failed to write "100000": write /sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/pod19ff292b-27c4-4d3e-85c7-6e19ca66b0cc/93bf59c3c798cc6f78343faa9cdbe7430900ab1102306f614c14e4969dc4c808/cpu.cfs_quota_us: invalid argument: unknown
      Exit Code:    128
      Started:      Wed, 31 Dec 1969 19:00:00 -0500
      Finished:     Mon, 29 Aug 2022 11:03:44 -0400
    Ready:          False
---
    Restart Count:  2
    Limits:
      cpu:     1
      memory:  1000Mi
    Requests:
      cpu:     200m
      memory:  200Mi
    Environment:
      CI_VERSION:                   k8s/1.3.11
      RUN_IN_AWS:                   True
      AWS_STS_REGIONAL_ENDPOINTS:   regional
      AWS_DEFAULT_REGION:           us-east-2
      AWS_REGION:                   us-east-2
      AWS_ROLE_ARN:                 arn:aws:iam::<redacted>:role/eksctl-<redacted>-addon-iamservicea-Role1-1ZQQ37G9IUEF
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /etc/cwagentconfig from prometheus-cwagentconfig (rw)
      /etc/prometheusconfig from prometheus-config (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6npnt (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  prometheus-cwagentconfig:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-cwagentconfig
    Optional:  false
  prometheus-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-config
    Optional:  false
  kube-api-access-6npnt:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason           Age                From               Message
  ----     ------           ----               ----               -------
  Warning  LoggingDisabled  89s                fargate-scheduler  Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found
  Normal   Scheduled        40s                fargate-scheduler  Successfully assigned amazon-cloudwatch/cwagent-prometheus-6475d78897-jm22x to fargate-ip-10-0-6-68.us-east-2.compute.internal
  Normal   Pulled           36s                kubelet            Successfully pulled image "amazon/cloudwatch-agent:1.247355.0b252062-amd64" in 2.918018615s
  Normal   Pulled           35s                kubelet            Successfully pulled image "amazon/cloudwatch-agent:1.247355.0b252062-amd64" in 213.214317ms
  Normal   Pulling          22s (x3 over 39s)  kubelet            Pulling image "amazon/cloudwatch-agent:1.247355.0b252062-amd64"
  Normal   Created          22s (x3 over 35s)  kubelet            Created container cloudwatch-agent
  Normal   Pulled           22s                kubelet            Successfully pulled image "amazon/cloudwatch-agent:1.247355.0b252062-amd64" in 217.488977ms
  Warning  Failed           21s (x3 over 35s)  kubelet            Error: failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: failed to write "100000": write /sys/fs/cgroup/cpu,cpuacct/kubepods/burstable/pod19ff292b-27c4-4d3e-85c7-6e19ca66b0cc/cloudwatch-agent/cpu.cfs_quota_us: invalid argument: unknown
  Warning  BackOff          6s (x4 over 34s)   kubelet            Back-off restarting failed container
mrvisser commented 2 years ago

Note if I remove the limits or decrease them to 200M, the container actually schedules fine.

Here's details of the fargate node it gets scheduled on, doesn't look like it's over-allocating on the node :

Name:               fargate-ip-10-0-8-190.us-east-2.compute.internal
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    eks.amazonaws.com/compute-type=fargate
                    failure-domain.beta.kubernetes.io/region=us-east-2
                    failure-domain.beta.kubernetes.io/zone=us-east-2c
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-10-0-8-190.us-east-2.compute.internal
                    kubernetes.io/os=linux
                    topology.kubernetes.io/region=us-east-2
                    topology.kubernetes.io/zone=us-east-2c
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 31 Aug 2022 09:16:09 -0400
Taints:             eks.amazonaws.com/compute-type=fargate:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  fargate-ip-10-0-8-190.us-east-2.compute.internal
  AcquireTime:     <unset>
  RenewTime:       Wed, 31 Aug 2022 09:17:00 -0400
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Wed, 31 Aug 2022 09:16:40 -0400   Wed, 31 Aug 2022 09:16:09 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Wed, 31 Aug 2022 09:16:40 -0400   Wed, 31 Aug 2022 09:16:09 -0400   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Wed, 31 Aug 2022 09:16:40 -0400   Wed, 31 Aug 2022 09:16:09 -0400   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Wed, 31 Aug 2022 09:16:40 -0400   Wed, 31 Aug 2022 09:16:20 -0400   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:   10.0.8.190
  InternalDNS:  ip-10-0-8-190.us-east-2.compute.internal
  Hostname:     ip-10-0-8-190.us-east-2.compute.internal
Capacity:
  attachable-volumes-aws-ebs:  39
  cpu:                         2
  ephemeral-storage:           30787492Ki
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      3977000Ki
  pods:                        1
Allocatable:
  attachable-volumes-aws-ebs:  39
  cpu:                         2
  ephemeral-storage:           28373752581
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      3874600Ki
  pods:                        1
System Info:
  Machine ID:
  System UUID:                EC22EBC2-EF07-2A46-A197-F0CFC4FC477B
  Boot ID:                    2de0ed00-bfc1-4a8d-a124-dee1c4681abf
  Kernel Version:             4.14.287-215.504.amzn2.x86_64
  OS Image:                   Amazon Linux 2
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.4.13
  Kubelet Version:            v1.23.7-eks-84b4fe6
  Kube-Proxy Version:         v1.23.7-eks-84b4fe6
ProviderID:                   aws:///us-east-2c/1a1370e46c-1cbca8375a8c474e83e0c06f517e804f/fargate-ip-10-0-8-190.us-east-2.compute.internal
Non-terminated Pods:          (1 in total)
  Namespace                   Name                                   CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                   ------------  ----------  ---------------  -------------  ---
  amazon-cloudwatch           cwagent-prometheus-849cf85b94-8h8sm    200m (10%)    1 (50%)     200Mi (5%)       1000Mi (26%)   100s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests    Limits
  --------                    --------    ------
  cpu                         200m (10%)  1 (50%)
  memory                      200Mi (5%)  1000Mi (26%)
  ephemeral-storage           0 (0%)      0 (0%)
  hugepages-1Gi               0 (0%)      0 (0%)
  hugepages-2Mi               0 (0%)      0 (0%)
  attachable-volumes-aws-ebs  0           0
mrvisser commented 2 years ago

Ugh, so apparently limits and requests always have to be the same when running on fargate.

But this fargate-specific cwagent deployment has limits set higher than requests: https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks-fargate.yaml

SaxyPandaBear commented 2 years ago

I'm a little surprised this is only springing up now. I wasn't aware of a constraint on configuring limits vs requests on EKS Fargate, though I'm not super familiar with that line of work. I think this warrants further investigation

mrvisser commented 2 years ago

I'm surprised as well. But I came across this, the blue note near the top:

https://docs.aws.amazon.com/eks/latest/userguide/fargate-pod-configuration.html

Since Amazon EKS Fargate runs only one pod per node, the scenario of evicting pods in case of fewer resources doesn't occur. All Amazon EKS Fargate pods run with guaranteed priority, so the requested CPU and memory must be equal to the limit for all of the containers. For more information, see Configure Quality of Service for Pods in the Kubernetes documentation.

Then sure enough I can only get this container to start if I remove the limits or set it to the same as the requests.

mrvisser commented 2 years ago

Closing in favor of aws-sample fix: https://github.com/awslabs/amazon-eks-ami/pull/717