search

dgryski / go-tsz

Time series compression algorithm from Facebook's Gorilla paper
BSD 2-Clause "Simplified" License
538 stars 67 forks source link

Saving to and loading series from file #16

Closed glaslos closed 8 years ago

glaslos commented 8 years ago

Added helpers to dump and load a series to a file. In order to be able to gob.Encode the series, I had to export all fields. This should not breaking anything :ok_hand:

dgryski commented 8 years ago

This seems like the wrong approach. A more generic "save' function would load and store from a []byte, allowing serialization from any location and not depend on an actual local file.

It seems to me you want to temporarily pause writing and store all local state to a file, and then be able to pick it up against and resume writing values to the time series? Is this correct?

(Just trying to avoid http://xyproblem.info here..)

glaslos commented 8 years ago

You are right, my approach is too specific. I want to write the series periodically to disk or restore them when restarting a service. I'll update the PR accordingly.

Dieterbe commented 8 years ago

did you see #6 ? we were running the gob encoding/decoding for a while but it turned out to be way too slow.

Instead of functions like Load/Save, why not GobEncode and GobDecode ? that way we can later add other/better serialisation formats, just a thought.

dgryski commented 8 years ago

Adding GobEncode means we aren't tied to exporting every single identifier. MarshalBinary is even more generic.

glaslos commented 8 years ago

Thanks for the feedback. I made another attempt now going the Marshal route. I'm kind of successful but struggle restoring the datapoints in a sensible way. Here is what I got so far: https://github.com/dgryski/go-tsz/compare/master...glaslos:marshal_binary I'd be glad if someone could point out what I am missing.

glaslos commented 8 years ago

Closing due to https://github.com/dgryski/go-tsz/pull/17