George3d6 / Inquisitor

An easily extensible, minimal footprint monitoring tool. (Still in the testing phase)
BSD 2-Clause "Simplified" License
30 stars 4 forks source link

Discussion around endpoints #26

Open George3d6 opened 6 years ago

George3d6 commented 6 years ago

Should the web-ui be it's own endpoint rather than a default part of the receptor ?

Should endpoints be implemented as receptor plugins that listen for agent event and send data when they arrive ? Rather than separate entities that query the receptor for whatever data they need and send warnings based on that.

What kind of endpoints should be implemented (e.g. slack, IRC, email, twillio)... how focused and/or generic should they be ?

How much logic should be part of the endpoints (E.g. should a recpetor plugin check if value f(X) => warning and the endpoints only read said warning or should you be able to define said f in an endpoint ?)

Deedasmi commented 6 years ago

Could you re-explain the purpose of the endpoints? The web server, for example, required a reactor, other ports, all the fun stuff. E-mail can be a simple function call however, doesn't even need state.

Even if some state were required, if we move forward with #6, the proposed function could be handle(&mut self, Arc<Status>). This would allow a slack/irc or pretty much any other 'endpoint' I could foresee to be handled fairly easily as a plugin. Even the web server could theoretically be a plugin if we gave it a reference to the tokio core on initialization, and allowed it to hook into the event loop directly.

George3d6 commented 6 years ago

The basic idea I originally had around endpoints is that they would be separate entities, basically just a self contained binary that know how to use the receptor's API and provide warnings/metrics/other-data in whatever format a user desires. The Web_ui thing is a mistake as part of the receptor and I'll probably move it to it's own separate crate.

Some ideas for endpoints would be:

Take for example a "scaling" endpoint which does something like this:

-> Checks CPU load on machine with the name cluster_1_* during the last 10 minutes, if it exceeds an average of 80% call the script to add a new machine to cluster_1.

(This is a usecase I actually have)

Or you could have an endpoint that shuts down (or slows the requests down) certain devices during peak times. Based on information it gets from an agent plugin (e.g. a command checking the log files for the devices that spam the monitored application a lot).

You could have a more generic endpoint, say, one that checks a number of values in the receptor's database (machine name X, during time y to z, for plugin a for key b in the message json) and based on that runs an aws command (or just a command in general).

You could also have endpoints for stuff like sending warnings or displaying data in a pretty way (e.g. the slack endpoints and the web ui).

Some of these would have different permissions than the receptor machine for executing their code and might do things which are not related to the receptor at all. They could also be disabled and added independent of the receptor and written in a language other than Rust.

I don't plan on including endpoints to cover all possible usecases, indeed, the whole initial concept for inquisitor was to provide an easy to use API and a modular monitoring solution which one could use to flue whatever endpoints they wanted (e.g. you don't want to use the Web_ui in the repo, go make a graphana endpoint). But I wish to have a few endpoints that demonstrate how one can/should use the API and cover very common usecases.

George3d6 commented 6 years ago

This concept of endpoints can even be applied if the receptor doesn't store it's own state via SQLite, but rather stores it's state via plugins that create a database and possibly expose a http/tcp path to query said databse (or sends messages to the various endpoints via plugins). Though at the moment I'm still biased towards keeping SQLite (or another database) as an inherent part of the receptor, rather than expanding the definition&usecase of plugins and making a plugin out of it.