Data loading

[at15]

Related #1 #9

Generated data is stored to disk and need to be inserted into TSDB

NOTE: the generated data is independent of specify tsdb and is serialized using protobuf. So it need to be transformed before inserted into TSDB

On the fly transform

• read and de serialize generated file, post into TSDB
• Pro
  • take less disk space
  • less disk IO
• Con
  • de serialization overhead
  • can not be reused

Pre transform

• read and de serialize generated file, save as the exact post format to file
• read the saved the file and post exact bytes to TSDB
• Pro
  • no de serialization overhead
  • can be reused
• Con
  • may take large disk space, due to tag and series name will be duplicated
  • larger disk IO

However when it comes to implementation these two have little different

data -> de serialization ->  pack into certain format (i.e. JSON) -> client lib -> TSDB
data -> de serialization ->  pack into certain format (i.e. JSON) -> file -> client lib -> TSDB

[at15]

There is a problem for decoding value, the value could be integer/double. The current solution is, specify the data type when load the data, one data file can only have integer or double, not both.

[at15]

as for exp backoff, besides influx comparison, there is another example https://github.com/rdegges/go-ipify

[at15]

similar to backoff, saw a Circuit Breaker Pattern and an implementation https://github.com/sony/gobreaker