search

orca-zhang / influxdb-cpp

💜 C++ client for InfluxDB.
MIT License
163 stars 83 forks source link

Sync between 2 databases. #36

Open Siegurd01 opened 3 years ago

Siegurd01 commented 3 years ago

It would be nice to add a sync feature where data can be synchronized between 2 databases on 2 hosts. The sync algorithm is kinda tricky but I made it on different PL so I can describe it's algorithm:

  1. Set up and connect to 2 databases (master A and slave B).
  2. Read only one metric from the 2 databases (reading one metric is quicker then reading whole database) on specified period to find out time arrays ("time holes" between A and B databases).
  3. Here comes the tricky part. Need to find where the time array in A is found in time array from B and if not - sync only the missing data from A to B using the resulting time array. In Matlab ismember does the trick.
  4. Transform resulting time array in to a short start-end pieces to read the data from A and write it to B using small amount of RAM.
  5. profit :)