search

burmanm / gorilla-tsc

Implementation of time series compression method from the Facebook's Gorilla paper
Apache License 2.0
210 stars 37 forks source link

Enforce positive timestamps and maximal gaps #16

Open yannbolliger opened 5 years ago

yannbolliger commented 5 years ago

The timestamp of value 0 is somewhat specially treated by the library and used as initialiser for the field storedTimestamp in both compression & decompression. This leads to a number of strange behaviours if in the middle of a series some timestamp is 0.

It might be a good idea to disallow all non-positive timestamps to protect users from those behaviours.

Also, the same could be applied for timestamps that are too far apart in time.

burmanm commented 4 years ago

Yes, the storedTimestamp could be changed to something else or a boolean instead to detect the initialization part.

The other issues are a bit more tricky and more related to the application using them, due to way the original paper was written. The application using the library should take care of the block management, so writing a timestamp too far was never an issue that the library has to manage. Same applies to the negative timestamps, if we follow the original paper (in-order inserts).