search

raintank / worldping-api

Worldping Backend Service
Other
25 stars 18 forks source link

worldping can get stuck in busy loop and max out cpu #62

Open Dieterbe opened 7 years ago

Dieterbe commented 7 years ago

in raintank-docker i accidentially triggered max cpu usage in worldping. it was flooding the log with messages like:

2016/09/20 14:46:27 [pubsub.go:121 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:114 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:121 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:114 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:121 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:114 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:121 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:114 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:121 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:114 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:121 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:114 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:121 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:114 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:121 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:114 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:121 publish()] [E] nack message 0, body: ""
2016/09/20 14:46:27 [pubsub.go:114 publish()] [E] nack message 0, body: ""

I see ./pkg/alerting/jobqueue/pubsub.go and in ./pkg/events/pubsub.go are pretty similar. perhaps the problem is the loop that reads from pending but also writes into pending can get stuck looping. and/or something with the confirm channel.