search

hiveco / conntrack_exporter

Prometheus exporter for tracking network connections
Apache License 2.0
136 stars 15 forks source link

Segfault with docker #2

Closed blackrez closed 6 years ago

blackrez commented 6 years ago

Hello,

I have issues with conntrack_exporter in docker.

After a long run, conntrack_exporter crashed without logs.

In my kernel log, I found :

[586087.070984] traps: conntrack_expor[4084] general protection ip:7fc28be79196 sp:7ffea652b7e0 error:0 in libc-2.23.so[7fc28be42000+1c0000]

And the output of docker info : 

docker info
Containers: 21
 Running: 2
 Paused: 0
 Stopped: 19
Images: 66
Server Version: 18.05.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.16.13-300.fc28.x86_64
Operating System: Fedora 28 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.672GiB
Name: localhost.localdomain
ID: 7SBO:6IM6:SKBT:LFCV:PUT6:7VI7:IXB7:QIUY:LINL:QCE4:LXNH:GZZ3
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
dbendelman commented 6 years ago

@blackrez which version of conntrack_exporter are you using? The latest version v0.3 fixes some crash scenarios, might want to try it.

Also, when you write it crashes after a long time, how long (hours, days, weeks)? It might help you to run with docker run --restart=always ... in the meantime.

dbendelman commented 6 years ago

Also, can you post your full docker run command?

blackrez commented 6 years ago

This is the last version from the docker hub, it is conntrack_exporter v0.2.

dbendelman commented 6 years ago

@blackrez Docker Hub actually had the v0.3 tag for a couple of weeks now but the :latest tag wasn't updated and still pointed to v0.2. Sorry about that, I just fixed it. Please docker pull hiveco/conntrack_exporter and try again.

blackrez commented 6 years ago

Ok I will test the latest version and see if segfault happen.

blackrez commented 6 years ago

I have the same issue with v0.3 and I have a message from logs.

conntrack_exporter: src/connection_table.cc:62: void conntrackex::ConnectionTable::updateConnection(conntrackex::Connection&): Assertion `!(this->is_rebuilding && connection.hasTrackingStopped())' failed.
dbendelman commented 6 years ago

The connection updating logic was rewritten in v0.3 and that assertion had been removed. I suspect you're still somehow running v0.2.

I just verified that hiveco/conntrack_exporter:latest == hiveco/conntrack_exporter:0.3 in Docker Hub, so please make sure you pulled the latest tag again or use the 0.3 tag explicitly.

blackrez commented 6 years ago

I confirm it works great with 0.3.

Many thanks.