Circuit breaker for individual log messages

pcdshub / lcls-twincat-general

A PLC code toolkit for LCLS TwinCAT PLC projects

https://pcdshub.github.io/lcls-twincat-general/

Other

16 stars 20 forks source link

Circuit breaker for individual log messages #42

Closed slacAWallace closed 4 years ago

slacAWallace commented 4 years ago

@n-wbrown @ghalym

klauer commented 4 years ago

Does this work? I'd imagine you'd want to stop the messages from ever going to the event system, not ignore them once they're already created

slacAWallace commented 4 years ago

You mean at where the log message is generated?

klauer commented 4 years ago

Yeah. If it gets passed to the event system, you'll see the message in TwinCAT.

Stopping the log handler (which listens for events from the event system) will cause messages to get backed up in the PLC buffers. Of course, they will also not be sent off to logstash/grafana here.

slacAWallace commented 4 years ago

@n-wbrown @klauer tell me what you think of this. I still need to pytmc it, but I think it solves both the very problematic single log-message and the trickle trip cases.

slacAWallace commented 4 years ago

@klauer @n-wbrown poke

slacAWallace commented 4 years ago

OK now it actually does what I was intending at least for the local trip. The global trickle trip still needs a test to target the condition I am trying to address, but at least the local trip will stop really spammy loggers from DoSing a PLC. I would appreciate feedback on how intuitive the circuit breaker settings are.

klauer commented 4 years ago

Given that you accepted my suggestions it looks like I was right in the interpretation of them. In that case, I think they're reasonably clear once you know the overall intent of the circuit breaker.

slacAWallace commented 4 years ago

slacAWallace commented 4 years ago

@klauer if this looks OK i am going to pull it in now. It would help us today.

klauer commented 4 years ago

👏 Let's see how it goes!