Support Prometheus Remote Write as an Optional Results Visualization

[joe-elliott]

While browsing currently supported results visualizations I noticed that Prometheus is missing. Prometheus is a commonly used and powerful time series aggregation and querying solution that we would like to leverage to visualize output from k6.

We have recently just merged a few k6 benchmark tests in our repo and enabled them in CI. We (and I think others as well) could benefit from storing output metrics from these applications in Prometheus.

Feature Description

I can't say I'm familiar enough with the various visualization options and what data is available to give a good description here. I am quite familiar with the prometheus metrics format/options and can provide insight/advice from that side.

Suggested Solution (optional)

I believe the best path to supporting Prometheus would be to implement the remote write protocol. Also note by doing so you would immediately be adding support for a large number of tsdbs and backends: https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage that others may be interested in using.

The actual Prometheus RW client is exposed and usable directly in a go application: https://github.com/prometheus/prometheus/blob/master/storage/remote/client.go#L81

[na--]

Thanks for making this issue! I can't say that I am familiar with Prometheus remote write, but from skimming the docs, it seems like a much better fit for k6 than the previous attempt at integrating Prometheus (https://github.com/loadimpact/k6/pull/478). And yes, I'd say that we want to have Prometheus support in the core of k6...

As I mentioned in a PR for another potential output yesterday (https://github.com/loadimpact/k6/pull/1755#issuecomment-740708324), we plan to heavily refactor the current k6 output interface in the next few months, hopefully in time for k6 v0.31.0. The main reason for that is that the current output (or Collector, as it's known in the code) API has many issues (https://github.com/loadimpact/k6/issues/1606, https://github.com/loadimpact/k6/issues/1423 and the many linked in https://github.com/loadimpact/k6/issues/1075) and is barely fit for purpose... And we also want to to enable xk6 output extensions!

That said, even if someone contributes a quality Prometheus remote write PR with the current output interface, I think we'd be willing to merge it and port it to the new API ourselves... :sweat_smile: Or, after we refactor everything, making such a PR ourselves will probably be very high on the priority list.

Somewhat connected issues: https://github.com/loadimpact/k6/issues/858, https://github.com/loadimpact/k6/issues/343, https://github.com/loadimpact/k6/issues/1064

[dgzlopes]

Prometheus folks added experimental support for remote_write requests on the latest release.

That means, that apart from shipping results to a wide array of remote write backends, this would work with "vanilla" Prometheus too!

[simskij]

From a glance, this look as it could be as easy as just forwarding the existing samples through the prometheus remote write client That's of course not the case though. From some playing, I think the way we'll need to go about doing this is by:

1. Transform all k6 samples to the corresponding Prometheus metric, including timestamps
2. Somehow construct a time series out of these metrics (this is where I'm currently at)
3. Convert the time series into a protobuf stream
4. Send it using the WriteClient

[yorugac]

Confirming that @simskij's draft extension works :+1: Apparently, the problem with snappy compression on remote write was that Encode must be used only as a return value:

encoded := snappy.Encode(nil, buf)

And not like this:

var encoded []byte
snappy.Encode(encoded, buf)

:eyes:

[yorugac]

To track overall progress for the remote write extension see here