search

NationalSecurityAgency / timely

Accumulo backed time series database
https://code.nsa.gov/timely/
Apache License 2.0
380 stars 108 forks source link

Implement LTTB Downsampling #25

Closed klucar closed 4 years ago

klucar commented 8 years ago

Current downsampling/aggregating options are susceptible to visually misrepresenting the data and there are better algorithms. Current downsampling scheme doesn't allow aggregators to operate on batches of samples at a time, which is required for LTTB (specifically one sample and two batches). While this could be implemented in JavaScript on the client side, server side would avoid sending large amounts of raw data to the client.

LTTB = Largest Triangle Three Buckets

From this thesis: http://skemman.is/stream/get/1946/15343/37285/3/SS_MSthesis.pdf

Good blog post here: http://blackops.io/blog/2014/05/time-series-graphs-and-downsampling/

Probably requires #21 to work but we may be able to highjack a rarely used downsample option (dev) for testing.

dlmarion commented 8 years ago

If we can determine the graph width on the users display, then we might be able to pass that to the Timely server and do further downsampling on the query results. For example, if based on the downsample and time range in the users query, they are asking for more data points than can be displayed, then we could use the LTTB algorithm to massage the value of the results.

dlmarion commented 8 years ago

The new sliding window iterator might enable this, but I think we need information from Grafana