Upgrade Prometheus integration to use remote writing

[tsenart]

Limitations of the current exporter based approach

1. Prometheus scrapes metrics from a running vegeta attack process and assigns timestamps to samples on its server. This means result timestamps aren't accurate (i.e. they're scraping time, not result time).
2. Configuring Prometheus to scrape vegeta needs to happen out-of-band. That's a hassle!
3. Since there's no coordination between a vegeta attack process and a Prometheus server, an attack process will finish before Prometheus has the chance to scrape the latest observations.

Instead, we should do something like this: https://k6.io/docs/results-output/real-time/prometheus-remote-write/

[flaviostutz]

I think this is a good addition for distributed "attacks"!

While we don't implement this, for distributed load tests an alternative would be:

• use https://github.com/flaviostutz/etcd-registrar to start with each vegeta attacker (which will add an entry with the attacker address to a ETCD server) when the service starts and removes it from the etcd "directory" when it is killed
• configure Prometheus to observe the ETCD service list to start monitoring it as soon as the attacker starts working (see https://prometheus.io/blog/2015/08/17/service-discovery-with-etcd/)

We still will have some of the issues mentioned above, but it can help in the meanwhile.

[flaviostutz]

Would we implement this as part of the "attack" (changing the current implementation), or as a separate command?

In both situations we have to be careful about the latency introduced by the push part, so probably we have to make it run asynchronously, in a separate goroutine from the main processes.

What do you think?

[tsenart]

Sure, we'll have to be careful always when introducing changes to attack, this included! I'm leaning to change the existing implementation + introduce a separate prom-write sub-command.

[peterbourgon]

Just want to note that distributed tests shouldn't rely on an etcd server, or any other kind of external coordinator.