grafana / tempo

Grafana Tempo is a high volume, minimal dependency distributed tracing backend.
https://grafana.com/oss/tempo/
GNU Affero General Public License v3.0
3.92k stars 510 forks source link

Use new conditional writes API on S3 for user-configurable overrides #4103

Open kvrhdn opened 1 week ago

kvrhdn commented 1 week ago

Is your feature request related to a problem? Please describe.

The user-configurable overrides API allows clients to write and update a subset of Tempo overrides. The overrides are stored in a bucket (separate from the traces). See also Tempo - user-configurable overrides.

User-configurable overrides are versioned and when a client wishes to update them they should supply the current version to avoid concurrent writes. GCS and Azure blob storage support this capability but AWS S3 did not so far, leading to a best-effort implementation: https://github.com/grafana/tempo/blob/64fd38eb7c97327efb2d5a2876d41bf62a75e80c/tempodb/backend/s3/s3.go#L470-L473

Describe the solution you'd like

S3 has announced they now support conditional writes. Docs on adding preconditions.

We should extend our S3 backend implementation and make use of the new preconditions.

⚠ Not all S3-compatible backends might support preconditions. We should take some time to investigate what they are planning to do and ensure those will still work.

Describe alternatives you've considered

We can keep using our best-effort implementation, since volume is low this is unlikely to cause a lot of issues.

Additional context