Open stevenmatthewt opened 6 months ago
Thanks @stevenmatthewt . I can see the request here, but what it means for data to be flushed in "pull-based sinks" like promotheus_exporter
is a bit ambiguous. In your case, you are asking for one additional scrape, but there may be other cases where:
For that reason, I don't think we'd always want to wait for just 1 additional scrape. Given the case of waiting for one more scrape doesn't seem uncommon, though, I'd be open to seeing it added as an opt-in feature.
Yeah, I was trying to think through the various scenarios this could help with earlier and couldn't come up with a clear single solution that accounted for everything. Some sort of possible configuration to try to handle the most common scenarios would be nice though, and leaving it open to improvement later. Honestly, even just a configurable shutdown delay in seconds would work – that's effectively what we've done already, but the kubernetes configuration for it can be a bit of a pain.
A note for the community
Use Cases
Currently, the graceful shutdown implementation of the Prometheus Exporter Sink isn't particularly helpful. It will wait for any in-flight HTTP requests to complete, but it doesn't actually wait for additional scrapes to ensure all metrics have been gathered. If you're running Vector in a relatively ephemeral fashion (we run a Vector container in several of our app pods in Kubernetes), this can lead to a significant number of metric samples "dropped". When a pod begins terminating, we wait for our primary container (which is emitting metrics via StatsD) to shutdown gracefully, then we send a SIGTERM to the Vector container. We'd like that Vector container's graceful shutdown to wait until the next Prometheus Scrape before exiting the process.
Attempted Solutions
For our specific example, we have Vector setup with a simple StatsD Source and a Prometheus Sink:
And I've run a Vector server with:
Then, I send some metrics via StatsD:
And on terminating the Vector server, I get the following logs:
The server begins the graceful shutdown process, but finishes it in under a second and doesn't wait for an additional scrape from our prometheus server. This isn't wrong per-se, but it's not particularly helpful since you can lose up-to one scrape interval of metrics.
Proposal
No response
References
https://github.com/vectordotdev/vector/issues/11405 is similar, but not specific to prometheus exporter. I suspect the async nature of the prometheus exporter is why it isn't fully covered by the fix there.
Version
vector 0.36.0 (aarch64-unknown-linux-gnu a5e48bb 2024-02-13 14:43:11.911392615)