search

segmentio / stats

Go package for abstracting stats collection
https://godoc.org/github.com/segmentio/stats
MIT License
208 stars 32 forks source link

Adding veneur integration for stats #86

Closed systemizer closed 6 years ago

systemizer commented 6 years ago

Note: this is a first pass. If we agree with this direction, I can go ahead and add tests.

Veneur supports the DogStatsd protocol, so I chose to inherit the statsd functionality from the existing Datadog integration.

Veneur supports Magic Tags that help with routing metrics. We need this feature to route high cardinality metrics only to the Signalfx backend.

We need to add the VeneurGlobalOnly Tag for all SignalFX metrics or else our DPM will be higher than what our contract permits. If we set VeneurGlobalOnly on the Engine's Global tags, this will affect our current Datadog metrics (which is not something we want to do).

So, by adding veneur tags only to a veneur backend as I've done here, we can:

  1. Register a Veneur Handler with VeneurGlobalOnly and enable only the SignalfxSink
  2. Register a Datadog Handler (to keep metrics as they were)
  3. Write metrics to both such that we both get high cardinality metrics in Signalfx and don't change the behavior of our current Datadog setup.

This is more complex than I would like it to be, but due to some limitations with routing metrics and feature imparity between Signalfx and Datadog, this is what I've come up with.

I'm open to other suggestions. Let me know what you think!

boggsboggs commented 6 years ago

I'm onboard with the setup here. Only one loose end to consider from the veneur Readme: Note: For global counters to report correctly, the local and global Veneur instances should be configured to have the same flush interval.

Our current datadog veneur is configured to 10s, I think. And, signalFX is configured to 60s.

We might have to send the signalFX metrics straight to the veneur-proxy hosts, like you suggested earlier.

systemizer commented 6 years ago

This is ready for review. @achille-roussel I decide not to add a tag-list to the current datadog client because global tags are currently configured at the engine level. IMO, it would only make sense to add tags at the handler level if there was specific tags that affected how metrics were handled by that particular handler (in this case, veneur uses magic tags). That being said, my feelings about the matter aren't very strong; adding a global tag list to the datadog handler would add less code but might result in confusion in the future if some people add global tags to the handler while others add it to the engine