Netflix / vector

Vector is an on-host performance monitoring framework which exposes hand picked high resolution metrics to every engineer’s browser.
http://getvector.io/
Apache License 2.0
3.58k stars 253 forks source link

display archived metrics from pcp #91

Open spiermar opened 9 years ago

fche commented 8 years ago

pmwebd can probably enable this already: have vector make a graphite query of the form

 /graphite/render?target=METRICSPEC&format=json&from=TIME&until=TIME

where METRICSPEC is (for now) a bit clumsy (hostname.pcp.met.ric.INSTANCE). You get back some fairly parsable JSON. The pcp.met.ric would have to be one that's being collected to a pmwebd-accessible archive. If vector uses any pcp metrics that are not logged by default, or perhaps not logged at high-enough rates, we could have a separate pmlogconf file to add that stuff.

natoscott commented 8 years ago

Uhm, please don't use the graphite API hacks in Vector - if this functionality is needed beyond graphite, it should become a proper part of the PMAPI + PMWEBAPI, and exposed to all tools in a more considered way. Thanks!

fche commented 8 years ago

Implementing the above idea in Vector would in no way contradict a "more considered" solution in pcp core. It would merely beat it by a year.

natoscott commented 8 years ago

| [...] It would merely beat it by a year.

That depends entirely on the competence of the person doing the PCP work.

In a not-too-distant future PCP release, we will be making the graphite PMWEBAPI(3) extensions only runtime-optionally-available from pmwebd (i.e. only available if separate package installed) to allow for reduced pcp-webapi installation footprints for the vast majority of people not needing that (unfortunately Cairo brings in alot of X11) - including Vector - so it really is not going to be a good idea to rely on the graphite API hacks, unless you are actually graphite.

APIs are important, let's do them properly for all tools. Just let me know what Vector needs @spiermar (if you could try out the graphite API and it seems appropriate, we can refactor and promote it to being a "proper" API).

And don't believe the FUD - one of the PCP developers can make it happen very quickly.

fche commented 8 years ago

A careful reader will see I noted &format=json, which does not involve running Cairo code. So even if someone bothered to gather compelling user survey/census data, and they were to complain about about pmwebd dependency footprint, and we couldn't find a non-Cairo renderer library, we still wouldn't have to move this particular bit of code.

natoscott commented 8 years ago

| A careful reader will see I noted [...]

The careful reader would have raised their eyebrows at "/graphite/[...]" and realized this is clearly not a general purpose API that other webapps should be using.

| [...] we still wouldn't have to move this particular bit of code.

Yes, we do. This kind of API needs to be (re)done properly and should be provided by libpcp for all tools to enjoy, there is nothing webapi-specific here (let alone graphite-specific).

fche commented 8 years ago

this is clearly not a general purpose API that other webapps should be using

Says who? There are numerous webapps that use that same API, far more than we bundle in webjs.

This kind of API needs to be (re)done properly and should be provided by libpcp for all tools to enjoy

When/if libpcp grows such an C level API, the graphite and non-graphite web APIs can be bound to it. The miracle of software permits higher level APIs to be preserved even when underlying implementations are changed.