New ideas

[o]

Hi Kevin,

Here is the my some ideas about metriks:

• Storing metrics from remote servers: Looks trivial, may be this can be achieve with splitting project to two part, metriks-agent (For the collecting data from remote servers - or maybe using ssh connection or SNMP without agent), metriks-core (fetching data from remote servers, storing metrics, shiny dashboard etc).
• Configuration: as of the nature of this project, JSON looks good for writing configs.

• Main settings: Which plugins will be run, or dashboard settings.

  {
    graphs:
    [
        {
            "plugin": "nginx",
            "enabled": true,
            "configuration": {
                 "status-page-url": "foo.bar/nginx_status"
            }
        },
        {
            "plugin": "load-avg-snmp",
            "enabled": true,
            "snmp-host": "foo.baz"
        }
    ]
  }    

• Plugin definitions:

  //nginx.js
  {
    "step": 300,
    "source": "nginx.py",
    "ds": [
        {
            // datastore options, type, min, max etc. 
        }
    ],
    "rra": [
        {
            // rra options, cf, xff etc.. 
        }
    ],
    "graphs": {
        //options {}
        //items []
        //defs []
        //comment []
        //gprints []
        //other graph elements [][]
    }
  }

• Different data stores: whisper, mongodb, opentsdb etc. RRDTool having some limitations (but it's super scalable)

I'll add new ones.

Cheers!

[kvz]

Hey Osman,

Some great ideas in here, thank you! My design goals for metriks are slightly different though, I would like metriks to work with bare minimum dependencies. I want it to be up, when everything else in my network is down. That's why I want it to write static files to disk. Then when there's mongodb outage, I can still see what led up to that.

As for the plugin architecture, I'm trying to avoid having to edit config in multiple places. So enabling a plugin can be done in the plugin itself. And that's it, just your own file, no other global files to deal with. That you can break, that someone else can break, preventing all plugins from working.

I've thought about yaml and json for configuration. I like that they are fairly easy to work with and are commonly available. However when I think of quickly writing a bash plugin, such as those found in the plugins directory, it might be cumbersome to indent / find structure scope with everything you print to the stdout. So instead, I'm having metriks deal with finding scope & building nested structures from a simple delimited syntax, closely matched to what's easily possible in shell scripts. Metriks should be trivial, and a pleasure to expand with your own graphs.

However, I do have plans for exposing a nodejs endpoint so that you can programatically tap into metiks by including it's npm in your code, and grow rrds that way. This interface will obviously accept javascript objects and won't ask you to convert your native datastructures to some weird flat delimited syntax first.

I hope my design choices makes sense, let me know what you think!