Closed keyolk closed 4 months ago
Based on your metrics it does seem like Tempo is performing retention, but the bucket size is still growing. If an ingester or compactor exits unexpectedly it will sometimes write a partial block that will then be "invisible" to Tempo.
We recommend setting bucket policies to remove all objects a day or so after your Tempo retention to clean up these objects. I'd recommend a similar policy for multipart uploads which s3 also likes to keep around.
The docs on this are not great. We mention the multipart upload here:
https://grafana.com/docs/tempo/latest/configuration/hosted-storage/s3/#lifecycle-policy
but no real mention of the partial blocks. If this solves your issue, I'd like to turn this into a docs issue to add these details.
This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.
Describe the bug
Its objects and bucket size keep growing and never goes down.
Also I can see about 5GB parquet datas in each block dir.
When I see its log, many lines like the belows from the compactor pods
and some metrics
To Reproduce Steps to reproduce the behavior:
Expected behavior
s3 obejcts size should be reduced
Environment:
tempo-distributed
v1.6.1Additional Context