tgross
commented
8 years ago Streaming the metrics data out on the logs output is not as useful as something like a time-series database output. For a busy metric, parsing the text/JSON of a metric out from the application logs may be expensive in terms of throughput (computation, take a look at the logstash grok implementation in https://github.com/tgross/triton-elk/tree/working for example) and storage. Additionally, the Docker logs are emitted as line-formatted, so stuffing JSON into there is going to add extra mess for log drivers like syslog or gelf.
My proposed alternative is to have a metrics backend feature to ingest metrics from the user-defined metrics checks (via attaching to stdout of the metrics check process) and then emit them into the format expected by upstream metrics collection/aggregation services. A given metric can have multiple backends -- e.g. maybe we want to permanently log the metrics in application logs but have a rolling real-time of the last week in graphite.
Each backend will need to document an API contract with the metrics check so that user knows what format to put the data onto stdout so that it can be read correctly. Because each backend needs its own set of arguments to function, the configuration format might look something like:
"metrics": [
{
"name": "my.metric.name",
"poll": 10,
"check": "/path/to/my/metrics-check",
"backends": [
{
"statsd": {
"argKey1": "argVal1",
"argKey2": "argVal2"
}
}
]
},
{
"name": "my.other.metric.name",
"poll": 5,
"check": "/path/to/my/metrics-check",
"backends": [ {"log": {}} ]
}
]Possible backends:
- statsd
- collectd
- logs: format the metric in a single line and write to stdout so the log driver can take it from there
- InfluxDB "line format" or HTTP protocol
- OpenTSDB binary or HTTP protocol
- NewRelic, possibly via their new Agent SDK
I'd propose that the initial implementation I work up will include the logs backend (easy fallback) and probably statsd because it's easy and I've implemented it before.
(Also, I'm not wild about using the term KPIs, which is semantically overburdened with business KPIs and specifically "performance" rather than arbitrary metrics. I've suggested "metrics" in its stead.)
justenwalker
misterbisson
dekobon
This is poorly thought through so far, but I wanted to open up discussion for it.
Health checking gives us a binary way to determine if the app is working or not. If the app is not healthy, we should not send any requests to it and we most likely need to spawn a replacement. But scaling depends on more than binary app health. Every app has one or more performance indicators that can be used to determine if the app is nearing overload and should be scaled up or is too lightly loaded and should be scaled down. In the spirit of what Containerbuddy does to make applications container-native, I think it might also make sense to add an awareness of those performance indicators.
Some performance indicators can be read from the system. The five minute load average reported by the kernel may work well for many apps, but many of the most interesting indicators come from the app itself.
In MySQL, the number of
Queryentries fromSHOW PROCESSLISTthat are in anyWaitingstate can be a very significant performance indicator, and it's one that is best retrieved from within the container.In Nginx, the average request processing time is useful info, but that's only output in the logs, which are hopefully not inside the container (if they were, ngxtop would be a nice tool to help understand them). But we do expect
http_stub_status_modulein our triton-nginx image, so we can look at that instead.Active connectionsvs. theworker_connectionslimit is a hugely important number there. Any delta betweenacceptsandhandledis a huge red flag. Or, perhaps, theWaitingnumber is an inverse indicator (high numbers indicate low activity, low numbers are high activity, zero could be critical). (See https://github.com/tgross/triton-nginx/issues/3 for further musing on this.)Here's what I'm imagining it would look like in the config (though it would be obviously unlikely to actually mix Nginx and MySQL in a single image):
In order, the above four KPI entries would return:
Queryentries fromSHOW PROCESSLISTthat are in anyWaitingstate.0is great.1or above can be trouble.10or more is probably critical.(${Active_connections} - ${Waiting}) / ${worker_connections}, a decimal value.1is maxed out,0.5is 50% busy.accepts - handledvars, an integer value. Anything other than0here is probably critical.The executables that get these KPIs are to be provided by the app developer/packager, though it might make sense to have a common way to get thee system load average as part of Containerbuddy.
As with the health checks, the executables would be run periodically at the time specified. The health checks could then be returned in
stdoutas newline delimited JSON as they're executed, for interpretation and use by whatever tools are reading the logs.I wanted to offer those specific use cases as as an exercise to figure out what data would be handled here. In short, I'm thinking each KPI is a numeric (decimal allowed) time series value. The executable will simply return that value and Containerbuddy can pass it on with the name given in the config.