search

kostya / eye

Process monitoring tool. Inspired from Bluepill and God.
MIT License
1.19k stars 86 forks source link

Multiple flapping triggers for a single process #152

Closed smt116 closed 9 years ago

smt116 commented 9 years ago

Hi there,

is it possible to achieve something like:

trigger :flapping, times: 3, within: 2.minutes, retry_in: 5.minutes
trigger :flapping, times: 6, within: 10.minutes, retry_in: 60.minutes

?

I know that there is retry_times, but I guess that it will require manual start after that. Am I right?

kostya commented 9 years ago

try:

trigger :flapping1, times: 3, within: 2.minutes, retry_in: 5.minutes
trigger :flapping2, times: 6, within: 10.minutes, retry_in: 60.minutes

after retry_times, you can start only by hands.

smt116 commented 9 years ago

Thanks. I didn't know that I can just provide another name ;)

smt116 commented 9 years ago

It will not work. Eye will pick only the "shorter" one.

kostya commented 9 years ago

what you expect?

smt116 commented 9 years ago 
trigger :flapping1, times: 3, within: 2.minutes, retry_in: 5.minutes
trigger :flapping2, times: 6, within: 10.minutes, retry_in: 60.minutes
  1. process crashed
  2. eye tries to bring it up
  3. it tries 3 times within 2 minutes
  4. all tries were failed so it pauses for 5 minutes
  5. eye tries again after 5 minutes
  6. it tries 6 times within 10 minutes
  7. all tries were failed so it pauses for another 60 minutes
  8. back to the step 1

I have a case when process depends on a database. It will crash when user will restore a wrong database backup. Eye will keep sending an alert each 5 minutes till the database is fixed or someone from our supports takes an action.

kostya commented 9 years ago

i think about this: trigger :flapping, times: 3, within: 2.minutes, retry_in: 5.minutes, retry_times: 3, reretry_in: 60.minutes, reretry_times: 3

smt116 commented 9 years ago

For me it is a "good enough" solution, but not sure about these names. What if someone in the future will ask about one more step? rereretry? ;) Maybe better would be some hash as an option? or support for multiple definitions?