Persistence - robustness against Corruption of Data in Redis

[sageserpent-open]

Right now, if I were to edit the data in Redis, I could provoke a deserialization error and cause Akka to hang in the RediScala client. I could also just remove keys or add in rogue ones, etc. At the very least, I want to have error detection and I don't want the system to hang.

[sageserpent-open]

This could also happen due to code versioning issues where stored data no longer matches the codebase, but this is an issue for another story. There is a further similar issue where stored patches become obsolete (and possibly invalid wrt the bitemporal object model) due to drift in the codebase.

[sageserpent-open]

The good news is that I've managed to implement transactions in Lettuce - currently doing this on a branch where a single Redis API object is used with just the one choice of codec for all types - ie the API treats all values as being of type 'Any'.

The bad news is that I can now see that serialization is deferred until the point at which the transaction is executed - this is after the commands have been queued between the 'multi' and 'exec' bracketing points. Redis won't undo a command that has failed in the middle of transaction execution, so the upshot is that while the correct exception is thrown and the revision operation does terminate by propagating the exception, there will be partial effects committed into Redis. What to do?

[sageserpent-open]

• Plan A: do a dry-run of serialization before doing the transaction. Yuk.
• Plan B: find a way of distinguishing between the effects of failed and successful revisions, so that garbage from failed revisions does not count when Redis is queried. Might be able to do this via a two-step mapping of revision on to some GUID, then from the GUID to the revision-indexed data. That way, a successful transaction can be followed by another transaction that records the revision to GUID mapping. I don't like this, but it might work as a fallback. I'm not sure how this would play with concurrency, though - perhaps a watch could be used.
• Plan C: serialize everything up front and commit using a script. Conceptually simple, although there is the messiness of having a Lua script.

[sageserpent-open]

As it turned out, I was able to do Plan A by making the dry run into an actual Redis update, albeit into temporary data structures which then get merged into the permanent ones during the transaction proper - so serialization takes place before the transaction.

[sageserpent-open]

Got a failure in test 'should allow concurrent revisions to be attempted on distinct instances' [info] RedisCommandExecutionException was thrown during property evaluation. [info] Message: ERR MULTI calls can not be nested One to ponder....

[sageserpent-open]

This test failure was down to the test logic hitting the same world instances with multiple threads per instance - thus allowing the same Redis connection to be shared between several threads, and in turn allowing interleaving of transaction amongst other abuses of the connection.

The test is really intended to demonstrate the use of concurrent connections to Redis via distinct processes, so I've associated world instances and threads one-to-one to simulate this usage better. This removes the problem in the test logic, which leaves a 'genuine' test failure to fix.