search

sensu / sensu-go

Simple. Scalable. Multi-cloud monitoring.
https://sensu.io
MIT License
1.03k stars 175 forks source link

Backend not applying filters during shutdown #4112

Open emmanueljamet opened 3 years ago

emmanueljamet commented 3 years ago

Expected Behavior

When receiving a SIGTERM, sensu backend keeps applying filters on events processed until the whole backend is properly stopped.

Current Behavior

While shutting down, sensu backend executes handlers for events that should be filtered by custom filters, but are not.

Possible Solution

Maybe the components are not stopped in the right order (?)

Steps to Reproduce (for bugs)

Sensu backend is running on a Kubernetes cluster. Deleting the pod running sensu backend container is enough to see some events being sent to handler.

Context

We are running sensu on an EKS cluster based on spot instances. When AWS reclaims an instance, the pod is deleted in order to be rescheduled elsewhere. We are running about ~400 agents with an avg ~15 checks on each and each time a node is reclaimed we get a load of 10 to 30 unwanted "OK" events on our alerts dedicated slack channel. This adds some noise on our monitoring system.

The signal received cmd_signal_received

Filters errors pipelined_errors

Components shutdown order backend_shutdown_order

Your Environment

echlebek commented 3 years ago

Thanks for the bug report. We haven't verified this behaviour but it seems fairly conclusive based on your report. I'll tag it as a bug and we'll prioritize it accordingly.

Are you expecting that your javascript event filters should fail like they are in your screenshot? It seems like you have some undefined symbols.

emmanueljamet commented 3 years ago

Hi @echlebek. The 'fatigue_check' and 'period_check' are in fact the names of the filters not applied. This is not expected to fail :)

emmanueljamet commented 3 years ago

Hello, do we have an update on this issue please ?

ddes commented 3 years ago

Hi there, Do we have any news on this ?

echlebek commented 3 years ago

This is being worked on and will land in 6.5.2 or 6.6.0.

echlebek commented 3 years ago

The sensu-go filter/mutator/handler pipeline was substantially reworked in 6.5.0, so while I'm still looking into this issue, the problem may already be resolved. Please update to 6.5.1, as it might fix your problem. Thanks!

ddes commented 2 years ago

Hello @echlebek, we still have the same issue even if we are now in 6.7.2. Could this be re-opened to investigate on a fix? Thanks

echlebek commented 2 years ago

Sure, thanks for reporting.