Nelly-Barret / BETTER-fairificator

The fairification tools for BETTER project.
https://www.better-health-project.eu/
0 stars 0 forks source link

Add new data to the database + rollback #26

Open Nelly-Barret opened 3 weeks ago

Nelly-Barret commented 3 weeks ago

Pietro has mentionned that it would be good if we could add data incrementally to the database without having to have to reload the whole db.

I think the current code allows to know whether a resource already exists in the database by looking at its properties. This allows to not insert twice the same resources (if that happens). In any case, the database is not dropped by default, thus one can incrementally load data.

To be more tested.

We should also be able to rollback if one data file cannot be inserted properly.

Nelly-Barret commented 1 week ago

I can now add incrementally data and/or load several datasets at a time (still need tests).

For the rollback, I may have to look into transactions: https://www.mongodb.com/docs/manual/core/transactions/

Nelly-Barret commented 1 week ago

I can add new data, but I am not sure whether there are duplicates or not.

I was working on #79, and I ran the code three times (with no reset). I obtained three execution records in the Execution table (so far, so good), but their identifeir does not increment of only 1 (so far, not good):

better_multip> db.Execution.find({})
[
  {
    _id: ObjectId('667ac23daa303b4f4d942bcf'),
    identifier: { value: 'Execution/748' },
    createdAt: '2024-06-25T15:12:29.038702',
    parameters: { ... },
    analysis: {}
  },
  {
    _id: ObjectId('667ac2eeb3f18296fbc902fb'),
    identifier: { value: 'Execution/1163' },
    createdAt: '2024-06-25T15:15:26.967077',
    parameters: { ... },
    analysis: {}
  },
  {
    _id: ObjectId('667ac2fc06d0cc57c32e00a8'),
    identifier: { value: 'Execution/1578' },
    createdAt: '2024-06-25T15:15:40.891984',
    parameters: { ... },
    analysis: {}
  }
]