weaveworks / eks-quickstart-app-dev

Example flux manifests for eksctl gitops
Other
57 stars 44 forks source link

CloudWatch agent panics with panic: /rootfs/proc doesn't exists, please use latest yaml templates to launch cloudwatch-agent #12

Closed marccarre closed 5 years ago

marccarre commented 5 years ago

Problem

$ kubectl -n amazon-cloudwatch logs --follow cloudwatch-agent-58t6b
2019/08/23 12:37:20 I! I! Detected the instance is EC2
2019/08/23 12:37:20 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json ...
/opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json does not exist or cannot read. Skipping it.
2019/08/23 12:37:20 Reading json config file path: /etc/cwagentconfig/..2019_08_23_12_33_15.083201554/cwagentconfig.json ...
2019/08/23 12:37:20 Find symbolic link /etc/cwagentconfig/..data 
2019/08/23 12:37:20 Find symbolic link /etc/cwagentconfig/cwagentconfig.json 
2019/08/23 12:37:20 Reading json config file path: /etc/cwagentconfig/cwagentconfig.json ...
Valid Json input schema.
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded

2019/08/23 12:37:20 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml 
2019/08/23 12:37:20 I! AmazonCloudWatchAgent Version 1.226589.0.
2019-08-23T12:37:20Z I! Starting AmazonCloudWatchAgent (version 1.226589.0)
2019-08-23T12:37:20Z I! Loaded outputs: cloudwatchlogs
2019-08-23T12:37:20Z I! Loaded inputs: cadvisor k8sapiserver
2019-08-23T12:37:20Z I! Tags enabled: 
2019-08-23T12:37:20Z I! Agent Config: Interval:1m0s, Quiet:false, Hostname:"ip-192-168-62-114.ap-northeast-1.compute.internal", Flush Interval:1s 
2019-08-23T12:37:20Z I! k8sapiserver OnStartedLeading: ip-192-168-62-114.ap-northeast-1.compute.internal

panic: /rootfs/proc doesn't exists, please use latest yaml templates to launch cloudwatch-agent

goroutine 42 [running]:
github.com/influxdata/telegraf/plugins/processors/k8sdecorator/stores.newNodeInfo(0xc0009c2ba0)
    /local/p4clients/pkgbuild-eAgXi/workspace/src/CWAgent/src/github.com/influxdata/telegraf/plugins/processors/k8sdecorator/stores/nodeinfo.go:105 +0x22e
github.com/influxdata/telegraf/plugins/processors/k8sdecorator/stores.NewPodStore(0xc000342701, 0xe, 0xc00099aa00, 0xffffffffffffffff)
    /local/p4clients/pkgbuild-eAgXi/workspace/src/CWAgent/src/github.com/influxdata/telegraf/plugins/processors/k8sdecorator/stores/podstore.go:66 +0x80
github.com/influxdata/telegraf/plugins/processors/k8sdecorator.(*K8sDecorator).start(0xc00040c000)
    /local/p4clients/pkgbuild-eAgXi/workspace/src/CWAgent/src/github.com/influxdata/telegraf/plugins/processors/k8sdecorator/k8sdecorator.go:68 +0x7b
github.com/influxdata/telegraf/plugins/processors/k8sdecorator.(*K8sDecorator).Apply(0xc00040c000, 0xc00042e250, 0x1, 0x1, 0xc0009c2b40, 0x1, 0x1)
    /local/p4clients/pkgbuild-eAgXi/workspace/src/CWAgent/src/github.com/influxdata/telegraf/plugins/processors/k8sdecorator/k8sdecorator.go:34 +0x325
github.com/influxdata/telegraf/internal/models.(*RunningProcessor).Apply(0xc000156750, 0xc00042e230, 0x1, 0x1, 0xc00042e230, 0x1, 0x1)
    /local/p4clients/pkgbuild-eAgXi/workspace/src/CWAgent/src/github.com/influxdata/telegraf/internal/models/running_processor.go:40 +0x131
github.com/influxdata/telegraf/agent.(*Agent).flusher(0xc00000e070, 0xc0000881e0, 0xc00007ede0, 0x0, 0x0)
    /local/p4clients/pkgbuild-eAgXi/workspace/src/CWAgent/src/github.com/influxdata/telegraf/agent/agent.go:355 +0x146
github.com/influxdata/telegraf/agent.(*Agent).Run.func1(0xc000345730, 0xc00000e070, 0xc0000881e0, 0xc00007ede0)
    /local/p4clients/pkgbuild-eAgXi/workspace/src/CWAgent/src/github.com/influxdata/telegraf/agent/agent.go:403 +0x6d
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /local/p4clients/pkgbuild-eAgXi/workspace/src/CWAgent/src/github.com/influxdata/telegraf/agent/agent.go:401 +0x5be

Solution

We probably ought to update our manifests. I'll go over the documentation again and update our manifests should there be changes: