lihaibh / ditto

NPM library that helps managing MongoDB snapshots easily and efficiently
https://www.npmjs.com/package/mongodb-snapshot
MIT License
48 stars 6 forks source link

Feature: conflict handling, replace documents (upsert) and exclude collections #24

Closed lihaibh closed 3 years ago

lihaibh commented 3 years ago

Improving the stability of the library as well as functionality by allowing replacement of data. Sometimes the databases that we work on, are being changed, even though that the library has the ability to clean data before performing a restore (remove_on_startup field) some other asynchronous processes can write data into the collection, which will result in a conflict error.

in order to resolve conflicts:

  1. we need an upsert mechanism to replace existing documents with certain filters.
astarget: {
   ...

      upsert: {
           "prefix_.*_suffix": (document) => pick(document, ['p1', 'p2', 'p3']),
           "collection_0": (document) => ({ p1: 'p1', p2: { $gte: 0 }}),
           "collection_1": ['p1', 'p2', 'p3'],
           "collection_2": ['p.*'],
           ".*": "_id"
      }

}
  1. exclude collections by name or regex:

    astarget: {
    ...
    exclude_collections: ["collection_1", "collection.*_suffix",...]
    }
  2. filter document to write into the target connector

    astarget: {
    ...
    writeDocument: (collectionName, document) => {
        return collectionName === 'people' && document.age > 10;
    }