project8 / dripline

Slow controls for medium scale physics experiments based on AMQP centralized messaging
http://www.project8.org/dripline
1 stars 0 forks source link

Alerts are not sent anymore #169

Closed guiguem closed 7 years ago

guiguem commented 7 years ago

In diopsid, which uses dripline/Go, it seems that at some point no more alert are sent:

16c 18:37:56.375 DEBU [runAmqpService] ▶ Sending a alert
16d 18:37:56.375 DEBU [send] ▶ Sending message to routing key <disk_status.WE30935>
16e 18:38:06.378 INFO [main] ▶ Alert sent: [/data/hot] All: 464.80 GB Used: 341.41 GB
16f 18:38:06.378 DEBU [runAmqpService] ▶ Sending a alert
170 18:38:06.378 INFO [main] ▶ Alert sent: [/data/warm] All: 464.80 GB Used: 341.41 GB
171 18:38:06.378 INFO [main] ▶ Sleeping now
172 18:38:06.378 DEBU [send] ▶ Sending message to routing key <disk_status.WE30935>
173 18:38:06.378 DEBU [runAmqpService] ▶ Sending a alert
174 18:38:06.378 DEBU [send] ▶ Sending message to routing key <disk_status.WE30935>
175 18:38:16.383 INFO [main] ▶ Alert sent: [/data/hot] All: 464.80 GB Used: 341.41 GB
176 18:38:16.383 INFO [main] ▶ Alert sent: [/data/warm] All: 464.80 GB Used: 341.41 GB
177 18:38:16.383 INFO [main] ▶ Sleeping now
178 18:38:26.386 INFO [main] ▶ Alert sent: [/data/hot] All: 464.80 GB Used: 341.42 GB
179 18:38:26.386 INFO [main] ▶ Alert sent: [/data/warm] All: 464.80 GB Used: 341.42 GB
17a 18:38:26.386 INFO [main] ▶ Sleeping now
17b 18:38:36.388 INFO [main] ▶ Alert sent: [/data/hot] All: 464.80 GB Used: 341.42 GB
17c 18:38:36.389 INFO [main] ▶ Alert sent: [/data/warm] All: 464.80 GB Used: 341.42 GB
17d 18:38:36.389 INFO [main] ▶ Sleeping now

This was tested on higgsino and it seems that no alert are sent after a couple of minutes. I have to restart diopsid to have it working again. Also the value returned by service.SendAlert(alert) is nil, since a isn't nil test does not trigger.. I will test soon diopsid on myrna to see what happens and if this is no server related.

guiguem commented 7 years ago

It seems that this problem on happened with higgsino, I could not reproduce with problem with the production system (myrna). Closing the issue.