search

bryanklewis / prometheus-eventhubs-adapter

Use Azure Event Hubs as a remote storage for Prometheus
Apache License 2.0
10 stars 12 forks source link

Event Hub event timestamps stuck from container creation #33

Open bailboy91 opened 8 months ago

bailboy91 commented 8 months ago

We have deployed the eventshub adapter into our prometheus stack. Our eventshub admins noticed during ingest / data translations that every event sent from prometheus has the timestamp with the exact date the container was spun up.

We are using all defaults for the container / build of the container. Nothing special in our Prometheus config for remote writes either.
We have a dev environment to test changes and pull logs from if needed.

Some troubleshooting we have done includes disabling batch events, rebuilding the container, and changing versions.

We are not experiencing any timestamp issues when we query through thanos or from prometheus directly. This is only affecting azure events hub.

bryanklewis commented 8 months ago

The timestamps are sourced from the sample at this point: https://github.com/bryanklewis/prometheus-eventhubs-adapter/blob/master/serializers/json/json.go#L64

If you debug at this point, is the sample correct before conversion? is the model data correct after creation?

bailboy91 commented 8 months ago

I am not sure how to accomplish the debugging. I've gotten a tcp dump from the container and attempted to see what data was being sent to event hubs but it is encrypted. The timestamps from the prometheus writes are current and incrementing like one would expect. Do i need to ask our event hub admins to configure something so that it's easier to troubleshoot or is that where we need to inspect the samples?

bryanklewis commented 8 months ago

A couple of things you can try then. Im not sure what your data looks like, and there maybe multiple timestamp values. Be sure to sample over several minutes.

  1. Turn on --log_level to "debug" for the adapter to see the output and verify
  2. Use prometheus "write_relabel_configs" to output some debug or force a different value
bailboy91 commented 8 months ago

Use prometheus "write_relabel_configs" to output some debug or force a different value

I had not tried this yet. I've got this going in dev and let it cruise over the weekend here. I appreciate the help!