RFC: concurrent updates

[jacquerie]

(Since we don't yet have a database layer, this item can wait)

There's a very interesting feature request by @dset0x: add a mechanism for concurrent updates by multiple python interpreters. The idea is that we have multiple nodes running the application, and only one node running the database. If multiple interpreters try to modify the same piece of data concurrently, the effects might depend on the order of the operations actually performed by the database. This might result in invalid data being committed to the DB, even if every modification was performed inside of a validation context. A user for this feature is the checker module, which might benefit from the concurrency speedup.

A possible solution is leaning on Redis, using it as distributed lock service: http://redis.io/topics/distlock. A problem I see with this approach is that I'm not convinced by the liveness argument outlined in http://redis.io/topics/distlock#liveness-arguments.

CC: @jalavik @MSusik @jirikuncar @kaplun

[lnielsen]

It's a very important problem to solve. That said, my first take is that locking is not the way to go. I'm would rather simply push this problem down the stack to the data store. E.g with PostgreSQL the modification should happen in a transaction (i.e. whichever process/node that commits first wins, all other fails and will have to retry - which would also be the case if a lock is obtained). For MongoDB I'm not sure how it works - updating a document is an atomic operation, but you still need to load the record I assume, hence chance is something happens in-between.

[kaplun]

:+1: (beside I am not sure yet on the point of having a storage layer in JSONAlchemy at all, since records can be instantiated on their own just for executing some transformation).

[jirikuncar]

We are using native SQLAlchemy versioning support [1] used for cross-transaction staleness detection.

[1] http://docs.sqlalchemy.org/en/latest/orm/versioning.html#configuring-a-version-counter