elastic / elastic-agent

Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.
Other
121 stars 131 forks source link

error fetching io.pressure in agent logs even when only using prometheus policy #371

Open matschaffer opened 2 years ago

matschaffer commented 2 years ago

I'm running agent inside docker via this docker-compose configuration https://github.com/matschaffer/elastic-otel-metrics/blob/main/docker-compose.yml#L4-L9

The agent is being enrolled into a "prometheus-otel" polic that I create than only has the prometheus metrics integration.

Here's the full policy:

prometheus-otel.policy.yml ``` id: f028f2d0-c461-11ec-befe-036853398716 revision: 4 outputs: default: type: elasticsearch hosts: - >- https://REDACTED.asia-northeast1.gcp.cloud.es.io:443 output_permissions: default: _elastic_agent_monitoring: indices: - names: - logs-elastic_agent.apm_server-default privileges: &ref_0 - auto_configure - create_doc - names: - metrics-elastic_agent.apm_server-default privileges: *ref_0 - names: - logs-elastic_agent.auditbeat-default privileges: *ref_0 - names: - metrics-elastic_agent.auditbeat-default privileges: *ref_0 - names: - logs-elastic_agent-default privileges: *ref_0 - names: - metrics-elastic_agent.elastic_agent-default privileges: *ref_0 - names: - metrics-elastic_agent.endpoint_security-default privileges: *ref_0 - names: - logs-elastic_agent.endpoint_security-default privileges: *ref_0 - names: - logs-elastic_agent.filebeat-default privileges: *ref_0 - names: - metrics-elastic_agent.filebeat-default privileges: *ref_0 - names: - logs-elastic_agent.fleet_server-default privileges: *ref_0 - names: - metrics-elastic_agent.fleet_server-default privileges: *ref_0 - names: - logs-elastic_agent.heartbeat-default privileges: *ref_0 - names: - metrics-elastic_agent.heartbeat-default privileges: *ref_0 - names: - logs-elastic_agent.metricbeat-default privileges: *ref_0 - names: - metrics-elastic_agent.metricbeat-default privileges: *ref_0 - names: - logs-elastic_agent.osquerybeat-default privileges: *ref_0 - names: - metrics-elastic_agent.osquerybeat-default privileges: *ref_0 - names: - logs-elastic_agent.packetbeat-default privileges: *ref_0 - names: - metrics-elastic_agent.packetbeat-default privileges: *ref_0 _elastic_agent_checks: cluster: - monitor prometheus-otel-docker: indices: - names: - metrics-prometheus.collector-default privileges: *ref_0 agent: monitoring: enabled: true use_output: default namespace: default logs: true metrics: true inputs: - id: prometheus/metrics-prometheus-b0d13fc8-b1c4-4ef9-b1c9-2ae8c2e448a2 name: prometheus-otel-docker revision: 2 type: prometheus/metrics use_output: default meta: package: name: prometheus version: 0.7.0 data_stream: namespace: default streams: - id: >- prometheus/metrics-prometheus.collector-b0d13fc8-b1c4-4ef9-b1c9-2ae8c2e448a2 data_stream: dataset: prometheus.collector type: metrics metrics_filters.exclude: null password: '' period: 10s hosts: - 'host.docker.internal:9464' metricsets: - collector ssl.certificate_authorities: null use_types: true bearer_token_file: '' metrics_filters.include: null rate_counters: true metrics_path: /metrics username: '' fleet: hosts: - >- https://REDACTED.fleet.asia-northeast1.gcp.cloud.es.io:443 ```

When I do this the logs are full of this error:

{"log.level":"error","@timestamp":"2022-04-26T02:34:20.691Z","log.origin":{"file.name":"metrics/metrics.go","file.line":382},"message":"error getting cgroup stats: error fetching stats for controller io: error fetching IO stats: error fetching io.pressure for path /sys/fs/cgroup:: open /sys/fs/cgroup/io.pressure: no such file or directory","ecs.version":"1.6.0"}

I see that agent.monitoring.enabled is true, so perhaps something in that code is attempting to fetch cgroup io pressure info?

I'm running on macOS with docker for mac on an intel chipset.

matschaffer commented 2 years ago

Ah, looks like I can avoid this if I disable agent metrics & logs in the policy settings

Screen Shot 2022-04-28 at 15 00 23