aws-samples / amazon-cloudwatch-container-insights

CloudWatch Agent Dockerfile and K8s YAML templates for CloudWatch Container Insights.
MIT No Attribution
162 stars 106 forks source link

cannot get pod from kubelet, err: call to /pods endpoint failed: #160

Open reddyvishnu01 opened 6 months ago

reddyvishnu01 commented 6 months ago

Seeing below Error when trying to deploy latest cloudwatch-agent (1.3xxx) in aws eks version 1.27 and 1.28 with IMDSv2 enabled. Cloudwatch agent 1.2xx was working as expected with similar AWS EKS Configuration.

2024-03-14T17:43:45Z I! {"caller":"service@v0.89.0/service.go:143","msg":"Starting CWAgent...","Version":"1.300034.0b498","NumCPU":8} 2024-03-14T17:43:45Z I! {"caller":"extensions/extensions.go:34","msg":"Starting extensions..."} 2024-03-14T17:43:45Z I! {"caller":"extensions/extensions.go:37","msg":"Extension is starting...","kind":"extension","name":"agenthealth/logs"} 2024-03-14T17:43:45Z I! {"caller":"extensions/extensions.go:45","msg":"Extension started.","kind":"extension","name":"agenthealth/logs"} 2024-03-14T17:43:45Z I! {"caller":"host/ec2metadata.go:78","msg":"Fetch instance id and type from ec2 metadata","kind":"receiver","name":"awscontainerinsightreceiver","data_type":"metrics"} 2024-03-14T17:43:45Z I! {"caller":"service@v0.89.0/service.go:178","msg":"Starting shutdown..."} 2024-03-14T17:43:45Z I! {"caller":"extensions/extensions.go:52","msg":"Stopping extensions..."} 2024-03-14T17:43:45Z I! {"caller":"service@v0.89.0/service.go:192","msg":"Shutdown complete."} Error: cannot start pipelines: cannot get pod from kubelet, err: call to /pods endpoint failed: Get "https://x.x.x.x:y/pods": Forbidden 2024-03-14T17:43:45Z E! [telegraf] Error running agent: cannot start pipelines: cannot get pod from kubelet, err: call to /pods endpoint failed: Get "https://x.x.x.x:yyy/pods": Forbidden