search

metal-stack / firewall-controller

A kubernetes controller running on bare-metal firewalls, creating nftables rules, configures suricata, collects network metrics
MIT License
47 stars 4 forks source link

Emit proper error event to status when signature does not match #93

Closed Gerrit91 closed 3 years ago

Gerrit91 commented 3 years ago

In the current implementation, when a user modifies the firewall resource without using a proper signature, the controller logs stack traces and also in the respective resource there is no hint that something is failing in the background.

Would be better if the controller adds a failure event for this resource such that it's possible to see when something like this is happening.

GrigoriyMikhalkin commented 3 years ago

Context: I tried to reproduce this problem locally, but was unsuccessful. Then @Gerrit91 and I looked into problem on an actual cluster and were able to reproduce it. But it was resolved after firewall-controller was reloaded.

First explanation of this would be that with time event cache becomes overloaded with events. And because default behavior of event broadcaster is DropIfChannelFull -- broadcaster just drops events that it doesn't have time to record. Problem with this explanation is that it seems that only events with type Warning are dropped, and not a single one gets recorded.

Another explanation would be that there's bug in the method that records events. But I couldn't find anything odd there just by looking at the code.

For now, I would suggest we postpone this issue until next occurrence of this problem. And if it occurs, i will add additional log messages in event recording logic to try to see, what exactly is happening.

majst01 commented 3 years ago

Hard to reproduce, also depends on how long the firewall-controller is running, closing