search

qnib / qcollect-ng

Iteration on qcollect using qframe as framework
Apache License 2.0
0 stars 0 forks source link

qcollect-ng dies when docker-service is updated #14

Open ChristianKniep opened 7 years ago

ChristianKniep commented 7 years ago

When a service is rescheduled, the qcollect-deamon dies...

2017/04/06 13:21:29 [II] Connected to '3X3U:3F7S:7TDK:HG22:ZBXM:J2K3:TLRA:H6NR:RZBD:OJCM:NOFW:MAML' w/ ServerVersion:'17.04.0-ce-rc2'
2017/04/06 13:21:29 [II] Start listener for already running 'qcollect-ng_collector.1.z0m6no19t3fqx3xu463eq24ng' [bdd9e70996a880266a7805e593388c3834802e22ed9a395663515bdfc596e802]
2017/04/06 13:21:29 [II] Start listener for already running 'qcollect-ng_frontend.1.kousg4463p5w9b59p4oiaydz6' [34d28985a4d5282bad9f8e09594e2e1bf4e03aa60558281220f3159ca4767867]
2017/04/06 13:21:29 [II] Start listener for already running 'qcollect-ng_backend.1.01ujgbpu6m4ccujxquroj21w7' [980720d8d0e834460a38368dcd70214fcc89db4a87740a33ccf5ffe68a36ff4f]
2017/04/06 13:21:48 [II] Container started ID:3a0fc8ff1cfba127102bb5984442074463afe5b660e6592f1a2fffee62ef7826
2017/04/06 13:21:48 [II] Container started ID:828d9d33edbc890748d550d1b3588e5126c7961db44c7e31f4615a57fdc4ce5e
2017/04/06 13:21:48 [II] Container started ID:64fc66590244076a25c88cba752e9f677c8d34bb96b259417b409d71653fb804
2017/04/06 13:22:52 [II] Container died ID:64fc66590244076a25c88cba752e9f677c8d34bb96b259417b409d71653fb804
panic: runtime error: index out of range

goroutine 2980 [running]:
github.com/qnib/qcollect-ng/vendor/github.com/qnib/qcollect-ng/utils.DiffCPUUsage(0x27d9614, 0xc4203b7020, 0x2, 0x4, 0x1312d00, 0x1312d00, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/local/src/github.com/qnib/qcollect-ng/vendor/github.com/qnib/qcollect-ng/utils/cpustats.go:36 +0x117
github.com/qnib/qcollect-ng/vendor/github.com/qnib/qcollect-ng/utils.DiffCPUStats(0x27d9614, 0xc4203b7020, 0x2, 0x4, 0x1312d00, 0x1312d00, 0x7731e63f2200, 0x0, 0x0, 0x0, ...)
    /usr/local/src/github.com/qnib/qcollect-ng/vendor/github.com/qnib/qcollect-ng/utils/cpustats.go:59 +0x134
lib/collectors/docker-stats.ContainerListener(0xc4203fc000, 0xc420014680, 0xc4200146c0, 0xc420012600, 0xc42021e780, 0x40, 0xc42030d1e0, 0x1f)
    /usr/local/src/github.com/qnib/qcollect-ng/collectors/docker-stats/plugin.go:59 +0x718
created by lib/collectors/docker-stats.ListenDispatcher
    /usr/local/src/github.com/qnib/qcollect-ng/collectors/docker-stats/plugin.go:105 +0xb36
ChristianKniep commented 7 years ago

Can be provoked by starting such a service:

docker service create --name sleep alpine sleep 5
ChristianKniep commented 7 years ago

docker run -d --rm --name test alpine sleep 5 works so far:

2017/04/06 14:04:54 [II] Container started ID:356464fe4766b73aec09cf0e228acc11497b2d05e8fb225250c2fdbe944cf704
2017/04/06 14:04:59 [II] Container died ID:356464fe4766b73aec09cf0e228acc11497b2d05e8fb225250c2fdbe944cf704
2017/04/06 14:05:00 Bad response getting stats for container: 356464fe4766b73aec09cf0e228acc11497b2d05e8fb225250c2fdbe944cf704
ChristianKniep commented 7 years ago

For now CPU metrics are disabled. As #3 wants to improve on them anyways...