Closed systemizer closed 6 years ago
I'm onboard with the setup here. Only one loose end to consider from the veneur Readme:
Note: For global counters to report correctly, the local and global Veneur instances should be configured to have the same flush interval.
Our current datadog veneur is configured to 10s, I think. And, signalFX is configured to 60s.
We might have to send the signalFX metrics straight to the veneur-proxy hosts, like you suggested earlier.
This is ready for review. @achille-roussel I decide not to add a tag-list to the current datadog client because global tags are currently configured at the engine level. IMO, it would only make sense to add tags at the handler level if there was specific tags that affected how metrics were handled by that particular handler (in this case, veneur uses magic tags). That being said, my feelings about the matter aren't very strong; adding a global tag list to the datadog handler would add less code but might result in confusion in the future if some people add global tags to the handler while others add it to the engine
Note: this is a first pass. If we agree with this direction, I can go ahead and add tests.
Veneur supports the DogStatsd protocol, so I chose to inherit the statsd functionality from the existing Datadog integration.
Veneur supports Magic Tags that help with routing metrics. We need this feature to route high cardinality metrics only to the Signalfx backend.
We need to add the
VeneurGlobalOnly
Tag for all SignalFX metrics or else our DPM will be higher than what our contract permits. If we setVeneurGlobalOnly
on the Engine's Global tags, this will affect our current Datadog metrics (which is not something we want to do).So, by adding veneur tags only to a veneur backend as I've done here, we can:
This is more complex than I would like it to be, but due to some limitations with routing metrics and feature imparity between Signalfx and Datadog, this is what I've come up with.
I'm open to other suggestions. Let me know what you think!