search

Matico-Platform / matico

An open source platform for building civic tech and geospatial apps
https://matico.app
46 stars 3 forks source link

Sync API options #17

Open stuartlynn opened 2 years ago

stuartlynn commented 2 years ago

Currently the sync API simply overwrites the imported dataset. We should give the user the option of

  1. Fully replace the existing table each import
  2. Replace the rows which match on some specific set of columns if they exist in the old and new datasets and append them if they don't
  3. Append to new data to the dataset.
nofurtherinformation commented 2 years ago

Merging #16 into this:

Right now we only have the option to update a sync dataset every x seconds. It would be amazing to have the ability to more finely control the time at which a dataset is imported.

This should be done by using cron syntax under the hood with a UI to help construct the statement.

Rust has a good parser for cron syntax : https://crates.io/crates/cron we can use this to generate the next run time once a import has been successful.