spine-tools / Spine-Database-API

Database interface to Spine generic data model
https://www.tools-for-energy-system-modelling.org/
GNU Lesser General Public License v3.0
6 stars 5 forks source link

Write to the same DB from concurrent processes #345

Open manuelma opened 5 months ago

manuelma commented 5 months ago

Say we have two processes commiting at roughly the same time. The following series of events might take place:

  1. The first process prepares the commit by computing the 'dirty' items from the in-memory mapping.
  2. The second process has already done that and proceeds to writing, but it writes things that conflict with the changes from the first process (or even, it writes the same things as the first process).
  3. The first process now writes but the DB complains that some constraints are violated.

To remmediate this, we need to do two things:

  1. Read the DB before computing the dirty changes to fix any conflicts already in the in-mem mapping (the conflict fixing should be done in another issue, but the reading before computing changes can be done here).
  2. On top of the above, we need to atomize the commit process so nobody else can write during the moment between we read and write.
manuelma commented 4 months ago

Done for sqlite, but not sure about other dialects. Any ideas?