search

deviantony / docker-elk

The Elastic stack (ELK) powered by Docker and Compose.
MIT License
17.27k stars 6.79k forks source link

Metricbeat won't start #903

Closed VahanA9 closed 1 year ago

VahanA9 commented 1 year ago

Problem description

I have problems with Metricbeat Exiting: 1 error: error initializing cgroup reader: error finding mountpoints: open /hostfs/proc/self/mountinfo: no such file or directory

Docker setup


$ docker version

Client: Docker Engine - Community
 Cloud integration: v1.0.35
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:52:17 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Desktop
 Engine:
  Version:          24.0.2
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       659604f
  Built:            Thu May 25 21:52:17 2023
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

```console
$ docker-compose version

[OUTPUT HERE]
docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019

#### Container logs

[OUTPUT HERE]
```2023-08-29 16:25:30 {"log.level":"info","@timestamp":"2023-08-29T12:25:30.200Z","log.origin":{"file.name":"instance/beat.go","file.line":330},"message":"Setup Beat: metricbeat; Version: 8.9.0","service.name":"metricbeat","ecs.version":"1.6.0"}
2023-08-29 16:25:30 {"log.level":"info","@timestamp":"2023-08-29T12:25:30.201Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: http://elasticsearch:9200","service.name":"metricbeat","ecs.version":"1.6.0"}
2023-08-29 16:25:30 {"log.level":"info","@timestamp":"2023-08-29T12:25:30.202Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":105},"message":"Beat name: metricbeat","service.name":"metricbeat","ecs.version":"1.6.0"}
2023-08-29 16:25:30 {"log.level":"warn","@timestamp":"2023-08-29T12:25:30.212Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":91},"message":"error fetching EC2 Identity Document: operation error ec2imds: GetInstanceIdentityDocument, http response error StatusCode: 403, request to EC2 IMDS failed.","service.name":"metricbeat","ecs.version":"1.6.0"}
2023-08-29 16:25:30 {"log.level":"info","@timestamp":"2023-08-29T12:25:30.212Z","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":100},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"metricbeat","ecs.version":"1.6.0"}
2023-08-29 16:25:30 {"log.level":"info","@timestamp":"2023-08-29T12:25:30.234Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: http://elasticsearch:9200","service.name":"metricbeat","ecs.version":"1.6.0"}
2023-08-29 16:25:30 {"log.level":"info","@timestamp":"2023-08-29T12:25:30.234Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Starting metrics logging every 30s","service.name":"metricbeat","ecs.version":"1.6.0"}
2023-08-29 16:25:30 {"log.level":"info","@timestamp":"2023-08-29T12:25:30.234Z","log.origin":{"file.name":"instance/beat.go","file.line":516},"message":"metricbeat start running.","service.name":"metricbeat","ecs.version":"1.6.0"}
2023-08-29 16:25:30 {"log.level":"warn","@timestamp":"2023-08-29T12:25:30.235Z","log.logger":"cfgwarn","log.origin":{"file.name":"sysinit/init.go","file.line":79},"message":"DEPRECATED: The --system.hostfs flag will be removed in the future and replaced by a config value. Will be removed in version: 8.0.0","service.name":"metricbeat","ecs.version":"1.6.0"}
2023-08-29 16:25:30 {"log.level":"info","@timestamp":"2023-08-29T12:25:30.238Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":195},"message":"Total metrics","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000},"quota":{"us":0}},"id":"/","stats":{"periods":0,"throttled":{"ns":0,"periods":0}}},"cpuacct":{"id":"/","total":{"ns":206948700}},"memory":{"id":"/","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":52301824}}}},"cpu":{"system":{"ticks":50,"time":{"ms":50}},"total":{"ticks":180,"time":{"ms":180},"value":180},"user":{"ticks":130,"time":{"ms":130}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":11},"info":{"ephemeral_id":"78bc939c-bd70-4287-9e0b-6d285237c32c","name":"metricbeat","uptime":{"ms":102},"version":"8.9.0"},"memstats":{"gc_next":24335408,"memory_alloc":20554376,"memory_sys":48084232,"memory_total":41102568,"rss":117063680},"runtime":{"goroutines":49}},"libbeat":{"config":{"module":{"running":0,"starts":0,"stops":0},"reloads":0,"scans":0},"output":{"batches":{"split":0},"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"elasticsearch","write":{"bytes":0,"errors":0}},"pipeline":{"clients":23,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":0,"retry":0,"total":0},"queue":{"acked":0,"max_events":4096}}},"metricbeat":{"docker":{"container":{"events":0,"failures":0,"success":0},"cpu":{"events":0,"failures":0,"success":0},"diskio":{"events":0,"failures":0,"success":0},"healthcheck":{"events":0,"failures":0,"success":0},"info":{"events":0,"failures":0,"success":0},"memory":{"events":0,"failures":0,"success":0},"network":{"events":0,"failures":0,"success":0}},"elasticsearch":{"ccr":{"events":0,"failures":0,"success":0},"cluster_stats":{"events":0,"failures":0,"success":0},"enrich":{"events":0,"failures":0,"success":0},"index":{"events":0,"failures":0,"success":0},"index_recovery":{"events":0,"failures":0,"success":0},"index_summary":{"events":0,"failures":0,"success":0},"ml_job":{"events":0,"failures":0,"success":0},"node_stats":{"events":0,"failures":0,"success":0},"shard":{"events":0,"failures":0,"success":0}},"kibana":{"cluster_actions":{"events":0,"failures":0,"success":0},"cluster_rules":{"events":0,"failures":0,"success":0},"node_actions":{"events":0,"failures":0,"success":0},"node_rules":{"events":0,"failures":0,"success":0},"stats":{"events":0,"failures":0,"success":0}},"logstash":{"node":{"events":0,"failures":0,"success":0},"node_stats":{"events":0,"failures":0,"success":0}}},"system":{"cpu":{"cores":16},"load":{"1":1.3,"15":0.91,"5":1.14,"norm":{"1":0.0813,"15":0.0569,"5":0.0712}}}},"ecs.version":"1.6.0"}}
2023-08-29 16:25:30 {"log.level":"info","@timestamp":"2023-08-29T12:25:30.238Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":196},"message":"Uptime: 105.143749ms","service.name":"metricbeat","ecs.version":"1.6.0"}
2023-08-29 16:25:30 {"log.level":"info","@timestamp":"2023-08-29T12:25:30.238Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":163},"message":"Stopping metrics logging.","service.name":"metricbeat","ecs.version":"1.6.0"}
2023-08-29 16:25:30 {"log.level":"info","@timestamp":"2023-08-29T12:25:30.238Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":73},"message":"Stats endpoint ([::]:5066) finished: accept tcp [::]:5066: use of closed network connection","service.name":"metricbeat","ecs.version":"1.6.0"}
2023-08-29 16:25:30 {"log.level":"info","@timestamp":"2023-08-29T12:25:30.238Z","log.origin":{"file.name":"instance/beat.go","file.line":528},"message":"metricbeat stopped.","service.name":"metricbeat","ecs.version":"1.6.0"}
2023-08-29 16:25:30 {"log.level":"error","@timestamp":"2023-08-29T12:25:30.238Z","log.origin":{"file.name":"instance/beat.go","file.line":1274},"message":"Exiting: 1 error: error initializing cgroup reader: error finding mountpoints: open /hostfs/proc/self/mountinfo: no such file or directory","service.name":"metricbeat","ecs.version":"1.6.0"}
2023-08-29 16:25:30 Exiting: 1 error: error initializing cgroup reader: error finding mountpoints: open /hostfs/proc/self/mountinfo: no such file or directory
antoineco commented 1 year ago

I remember seeing this issue in Docker for Desktop on WSL. Is that your setup?

VahanA9 commented 1 year ago

Yes , I testing in WSL

antoineco commented 1 year ago

Unfortunately this can't be fixed in docker-elk. I reported this issue upstream a while ago but it never got any attention:

The issue is that Docker for Desktop does some magic with mounts to allow files to be mounted from your WSL instance, instead of the WSL instance that runs the Docker daemon. /proc/self is a symlink which destination varies based on the process that reads it, so the magic mount mapping breaks.

On WSL I recommend disabling the features that rely on /host/proc/self/mountinfo, such as host monitoring (which anyway don't make a lot of sense inside Docker).