search

OpenTSDB / opentsdb

A scalable, distributed Time Series Database.
http://opentsdb.net
GNU Lesser General Public License v2.1
4.99k stars 1.25k forks source link

tsd server compress or delete cold metrics data #525

Open storm-dance opened 9 years ago

storm-dance commented 9 years ago

With longer and longer using time, hbase accumulate more and more metrics data. We can compress(Reduce accuracy, from one metric data in second to in day...) or delete cold metrics data to retrieve much storage and some assigned uids. Can tsd support this?

bikrant commented 9 years ago

For just the deletion of old rows, we have used TTL in hbase tables to automatically delete rows after some time. TTL can be updated at anytime and hbase will automatically apply that to all data, including the ones created before the new TTL is applied.

manolama commented 9 years ago

TTL will help with the storage size but it won't free up UIDs unfortunately. We are working on the rollup functionality that will let you store lower granularity data. Just a few bugs that we have to work out on our end.

utkarshcmu commented 8 years ago

+1 for rollup functionality

digrich commented 8 years ago

+1

utkarshcmu commented 8 years ago

@manolama I have applied TTL on "tsdb" table in HBase. Should I apply TTL on other tables as well to remove old data? How will TTL affect on all other tables except "tsdb"?

storm-dance commented 8 years ago

Now I send metrics to opentsdb and also kafka, run a stream job who reads metrics from kafka, rollup and send different granularity metrics to opentsdb. Opentsdb stores different granularity metrics to different families and applied different TTL.