attachmentgenie / vagrant-observability

reference implementation of monitoring systems
2 stars 2 forks source link

tempo should save traces to both dcs #36

Closed attachmentgenie closed 2 years ago

attachmentgenie commented 2 years ago

turns out tempo has no build in mechanism to read or write twice, and no sdk i know of will allow for write twice behavior so we ll need some form of external proxy to implement this behavior.

Fortunately grafana-agent has remote_write tempo support since v0.14.0 which allows for an array of upstream services. so we need to expose the required receivers in the agent config. grafana agent forwards all traces then as otlp traces towards tempo, so we need to modify the tempo config to reflect that reality