Adding a new daemon: the analyst

[aviau]

Hello :)

I am proposing the addition of a new optional daemon to Shiknen, the analyst.

This new component would allow for the following functionalities:

• Make bprules on metrics and statuses
• Generate alerts on unconfigured services and on passive configured services (ex: collectd)
• Make trending analysis on any metrics or combination of metrics.
• Possibly generate alerts and make bprules on services and statuses from different realms
• Generate new metrics based on a mix of other metrics

  It would cover the following use cases:

• Extend the capabilities of bprules.
  • For example: Two switches are connected together with two cables. I want to monitor the bandwidth usage of both cables. Currently, bprules won't give me a good idea of total usage. It would be appropriate to merge both metrics: Instead of 99% (cricial) and 0%(ok), just tell me 49%.
• In a large automated environment, it is complicated to configure hosts as they come and go. This is where passive monitoring comes in handy. However, I have to configure individual hosts if I want to receive alerts. I would like to define rules like the following: If load is above a certain threshold on any of my servers (configured or not), generate an alert.

  How the daemon would fit in the current data flow:

  1. The user creates rules and configures the analyst.
  2. From these rules, the arbiter configures the brokers so that they keep the necessary data in dedicated queues for the analysts.
  3. Receiver/Poller receives check results.
  4. The broker polls the check results broks and sends them to the database
  5. Before deleting the broks, the broker checks his config for analysts that requires them and puts the broks in dedicated queues.
  6. The analyst retrieves the broks from the appropriate brokers and analyses them. If necessary, it generates new broks/alerts.

     Details:

• A check result could be required by several analysts, the brokers keep one queue per linked analyst.
• At first, the rules would be written in Python and evaluated by the daemon. In the future, this would become modular and rules could be written with any syntax (Ex: lua).
• Analysts could analyses metrics generated by other analysts. They could also generate metrics based on analyst-generated metrics!

The analyst would greatly enhance Shinken's alerting capabilities and would also simplify alerting on passive results.

I would like to propose myself as the main developer for this new daemon, I would be working on this at Savoir-Faire Linux where we already have two other Shinken specialists and contributors.

[naparuba]

That's an interesting idea and in fact i'm already thinking about such rule system since some months.

And after look at some points, it do not fit the shinken way of thinking.

In fact, the whole shinken thing is done to minimise the inter-daemons intereaction, by a diff-bus (broks), objects that are linked togethers are put in the same schedulers (arbiter role) and do not manage the history (all in memory).

Shinken is like nagios, it's done to manage status, not metrics. There are tools for this and i don't think it's a good thing to have a deamon that do not fit the same logic that all the others.

But that's about philosophy, let's look at real use case now :)

We need such perfdata lookup for some cases. that's was triggers where done. the main problems is that they are in the scheduler that got a limited view in the objects (only a part of the It hosts). and that's why I never really loved the triggers.

One other key problems you will have to tackle, and I don't see in your post is how you will manage the distribute load between your daemons. rules assignements won't be enough for large env.

What you really need are in fact triggers (your python function that can then be LUA) in a metrology tool. It do not have to be in a monitoring tool, but only to send back alert/states to a monitoring tool.

You daemon is basically a graphite with rules that is generating perfdata into graphite as well (like the grahite agregator?) that is getting back alerts in a receiver. You cnan propose them to add such a alert levels, or maybe in collectd that already have such capabilities.

Maybe you can swith from graphite to elasticsearch if you only look at not so far data and you want scalability, but it's up to you :)

For the rule engine, you can look at riemman, it's not python nor lua, but it's still cool :+1: and you already have a ready to run event machine. If I'm not wrong it is in scala (and also the rules). It will also save you lot of coe lines :godmode:

On good thing on the shinken part can be a scheduler module that will allow the check_commands to grab data from the backend so you won't have to manage the reinsertion and the names mapping (can always be hard in the passive->active logic).

Good luk for your project, and let us know how it evoles, I think we will be please to add "function macros" linked from modules in the scehduler so we can test it in shinken. Maybe in this 2.2 version if we got time for it :+1:

[aviau]

@naparuba Thank you for your feedback :+1:

One other key problems you will have to tackle, and I don't see in your post is how you will manage the distribute load between your daemons. rules assignements won't be enough for large env.

Why? You mean that some rules will take too much load for a single daemon? This is definitely a problem we will have to think about.

What you really need are in fact triggers (your python function that can then be LUA) in a metrology tool. It do not have to be in a monitoring tool, but only to send back alert/states to a monitoring tool.

I have not thought about it like this, but you might be right. I will take some time to consider this approach.

For the rule engine, you can look at riemman

I am looking into Riemann right now. I will try to evaluate wether or not it could help me fill my use cases. I just want to make sure that it doesn't become a limitation because we cannot alter it to solve Shinken-specific problems.

Good luk for your project, and let us know how it evoles

I will try to keep this issue up-to-date.

[naparuba]

We continue on the devel mailinlist as there will be more hep from others than here.