Compactor set new retention period doesn't work

grafana / tempo

Grafana Tempo is a high volume, minimal dependency distributed tracing backend.

https://grafana.com/oss/tempo/

GNU Affero General Public License v3.0

4.04k stars 524 forks source link

Compactor set new retention period doesn't work #4336

Open petkovp opened 4 days ago

petkovp commented 4 days ago

I installed grafana tempo with helm and then I set the new retention period of the compactor 360h (15 days). The all setting were applied correctly (configMap, deployment), the tempo.conf file has new set 360h, the pod loaded the retention period.

Traces store in S3 bucket, but compactor delete traces older than 48 hours.

Could you please some help for this issue, because I am going to implement this product and I need for longer retention period that 48 hours.

compactor: config: compaction: block_retention: 360h

joe-elliott commented 4 days ago

Traces store in S3 bucket, but compactor delete traces older than 48 hours.

I would check the following:

confirm it is the compactor deleting the block by checking for a log message that shows the block deleted:

confirm the retention value is set as expected by curling the config endpoint on any compactors that are deleting blocks. /status/config
confirm that you have no overrides for the tenant whose blocks are being deleted. /status/overrides. also needs to be curled against
are you using s3? or an s3 "compliant" backend?
confirm that the system time of the compactor pods that are deleting blocks is correct.

petkovp commented 4 days ago

Hello again,

I am replying for question on previous comment.

Screenshot for compactor's log

Screenshot of config file section compactor retention

There aren't any overrides records or settings
S3 bucket - traces

Everything looks like fine.

joe-elliott commented 6 hours ago

2 Screenshot of config file section compactor retention

Not the config file. The output of the /status/config endpoint. We want to confirm the config file is being applied correctly. Please make sure you curl the endpoint of one of the compactors that's incorrectly removing blocks.

3 There aren't any overrides records or settings

Please curl the endpoint on a compactor to confirm.

confirm that the system time of the compactor pods that are deleting blocks is correct.

Did you check this? Also it would be interesting to check if all compactors or only some compactors are incorrectly removing blocks.