NationalSecurityAgency / timely

Accumulo backed time series database
https://code.nsa.gov/timely/
Apache License 2.0
377 stars 108 forks source link

Metric tablets may not fully age-off #184

Closed dtspence closed 5 years ago

dtspence commented 5 years ago

Tablets may accumulate for metrics that should be aged-off. Accumulo may create split points at certain time boundaries which can lead to orphaned metrics in older tablets. This is caused by the default compaction strategy no longer checking tablets after the split has occurred and the ratio calculation remaining unchanged.

The workaround is to invoke a full or scripted compaction.

Example:

sys.cpu-<-11days> [prev-end]
sys.cpu-<-10days> [end]

Given an age-off of 5 days and an example sys.cpu tablet above; the DefaultCompactionStrategy may have successfully compacted the tablet before the age-off is reached. Once a split has occurred, the data will no longer change and new compactions will not take place to delete data.