auto-generate graphs from wildcard

[srgshames]

Hi,

I use graphite a lot and have a few series of equivalent metrics for different devices/servers, for example: stats.gauges.devices.*.*.status

Often new metrics of the same series are introduced, resulting in a long and dynamic list of metrics of the same series as above. Currently, whenever there are new metrics (which happens a lot) I have to add new graphs for them manually to the matching Graphite/gdash dashboards.

I want to create a dashboard that would automatically detect and display the new metrics.

I could use a simple wildcard for a single "field" in gdash:

field :status, :alias_by_node => 4,
                    :data  => "stats.gauges.devices.*.*.status"

but that would result in a single graph with tons of metrics, rendering it hard to work with and practically unusable.

What I actually want to achieve is automatically creating a graph per metric, using the wildcard. I searched a lot and didn't find a solution for this in gdash. Is it currently supported? If not, is it planned to be?

Thank you

[ripienaar]

I have considered this to treat for instance node.munin.* and spit out a per node autogenerated set of graphs but have not gotten very far down that road yet

[blysik]

This would be awesome functionality, and something I've thought about as well.

[srgshames]

I implemented it. I still need to test it a bit, but once I finished, I'll send you a pull request

[srgshames]

I sent 2 pull requests: https://github.com/ripienaar/gdash/pull/57 https://github.com/ripienaar/graphite-graph-dsl/pull/25

I attached a small description of the change in the graphite-graph-dsl pull request, thought I'd better be more documented. Also, I have an example 'graph_gen' file for you. Since I'm fairly new to pull requests in GitHub, I don't know what's the best place for such a documentation and example. Can you advise?

[ripienaar]

to help me review your feature I guess here is good, I def need to understand what this is all about better :)

[srgshames]

OK, So I added a new class (GraphiteGraphGenerator) that evaluates files with 'graph_gen' extension that look something like this:

area                    :none
background_color        "#FFFFFF"
foreground_color        "#333333"
major_grid_line_color   "#ADADAD"
minor_grid_line_color   "#E5E5E5"

generator       :context => /(stats_counts\.target\.[\w-]+\.\d+)\.[\w-]+/,
                :naming_pattern => /stats_counts\.target\.([\w-]+\.\d+)/,
                :"field_prop:scale" => 1,
                :stats => ["success", "error"],
                :target_wrapper => 'integral(%s)'

and generates from it new files with the 'graph' extension. From there on, the existing GraphiteGraph class takes over to generate the graphite URL.

The file must hold a 'generator' component that must have the following: context - a regex for graphite metrics. The generator makes a request to Graphite to retrieve the metric list (/metrics/index.json). The regex scans the list and creates a list of unique metric branches. For each branch it will create a separate graph.

stats - list of metric nodes that are appended to each branch to construct a whole metric. Each graph file will hold "field" components per stat. In the example above, a metric branch can be stats_counts.target.blah-12 and the graph file created from this branch will hold 2 field components, one with a "data" value of stats_counts.target.blah-12.success and the second with a "data" value of stats_counts.target.blah-12.error. Each stat will also be used as a name for the field that it represents.

naming_pattern - an additional regex that is executed on the already given branches. The text that is caught in the parentheses will be used both as a name of the "graph" file for this branch, and as the "title" value inside the file (with a few changes: gsub('_', ' ').gsub(' - ', ' ').gsub('.', ' - ').gsub(' ', ' ')). If no match was found, the entire branch name will be used for the file name and title.

Additional optional fields:

target_wrapper - designed for cases where you want to use a graphite function on the metric (such as integral in the example above)

filed_prop:... - additional properties for the field component in the generated graph file.

That's it. Everything else (such as the "area", "background_color" etc. components in the example above) will be passed on to all generated files. Also, if you put here regular "field" or "critical" or "warning" or any other component, they will appear in all generated files.

A few important notes:

1. You put the "graph_gen" file/s inside the dashboard directory.
2. On each page load all generated graphs in the directory (.graph files starting with "gen_") will be removed, and new files will be recreated in their place. This is to prevent any "leftover" graphs in case you removed the corresponding metrics from Graphite.
3. A name of generated graphs will follow the pattern: gen_<name generated from the "naming_pattern" field, or from the branch>.graph. This is to easily differentiate between generated graph files and "hand-made" ones, and to prevent automatic deletion of the "hand-made" ones on page load.

Another example:

area            :none

generator       :context => /(stats\.gauges\.target\.[\w-]+\.\d+)\.[\w-]+/,
                :naming_pattern => /stats\.gauges\.target\.([\w-]+\.\d+)/,
                :"field_prop:scale" => 1,
                :stats => ["status"]

[Dieterbe]

:heart_eyes:

[powdahound]

:+1::+1:

[KlavsKlavsen]

thumbs up here as well - is there any hopes for this to be a part of standard gdash?

[cmenguy]

thumbs up, pretty much the only issue I have with gdash right now.