search

me-box / zestdb

ZestDB
MIT License
18 stars 3 forks source link

Supporting out-of-order writing #13

Closed jptmoore closed 6 years ago

jptmoore commented 6 years ago

The current backend log-based storage is suited to in-order time-series writes. Do we need to support out-of-order time-series writes?

cgreenhalgh commented 6 years ago

For complex values like tweets we have the requirement to be able to redact them, which is a limited kind of out of order (over-)write, but might be implemented some other way.

At the moment I could implement the drivers I am doing without writing out of order. Although I guess it would happen if someone changed the window of data being cached from an external source (e.g. twitter) to make it further in the past. Although again a work-around might be to drop all the data associated with that datasource and rebuild it (could be done by driver if there is an API method for that)

Toshbrown commented 6 years ago

Uploading google take out data would require the user to upload the files in the correct order if we do not support out-of-order writing

Also for drivers that import large amounts of data (e.g email driver) may wish to do this in reverse chronological order which requires out-of-order write support.