Fully Consistent Batches

[neallester]

Ensure that all (and only) Entities added to a batch are saved to their databases as a group (by database) with exactly the information they contained when they were added to the batch. Fully consistent batches would be guaranteed to execute in the order submitted. This would work on ACID capable storage mediums.

Batch Combining/Chaining (Selected for implementation)

When an entity already in an open (uncommitted) batch is added to other batches, the batches are combined or chained such that none will commit until all are committed and all will commit within the same database transaction. The batch should retain the update action rather than the entity itself so that the entity can be released for further modifications in another (uncombined/unchanged) batch during the commit process (while the entity could be modified within a second batch during the first batches commit process, the second batch could not commit until the first batch succeeded to avoid its data stepping on the first).

This has the advantage of not requiring pessimistic locking, but it does require the system come to a consistent state naturally before saving occurs. Objects constantly being updated in different batches (e.g. objects shared between many sessions) may prevent batches from ever committing since all of the batches an object is in must be ready to commit simultaneously before the commit can occur. The system could monitor for and warn about long lived batches.

Add a protocol Batch with implementations ConsistentBatch and EventuallyConsistentBatch

Add a reference to the current batch an Entity is in. Following table indicates what should be done if an entity is already in a batch when added to a different OPEN batch (actually different batch delegates):

_	New Batch Consistent	New Batch Eventually Consistent
Existing Batch Consistent	Combine contents of both batches delegates and then assigned batch delegate to both batches	Ignore
Existing Batch Eventually Consistent	Move Entity to new batch; assigned new batch to Entity.batch	Ignore

If Entity has an existing batch which is currently SAVING:

1. New batch registers for completion notification from the existing batch and can't begin its commit process until that (now) dependent batch does. This prevents batches from stepping on each other.
2. The new batch is assigned to Entity.batch
3. Entity is added to the new batch.
4. This design enables the elimination of the existing pending updates in Entity and in Entity commit processing.

DatabaseAccessor

1. Add an associated type for the database connection.
2. The update actions returned by the accessors should be open on UUID? which will be passed to the accessor features which implement the updates. the UUID will be used to obtain the connection associated with the update transaction (for consistent batches). Add:

func connectionFor (_ UUID?) -> C  {} //generic for accessor specific connection type
func startTransaction() throws -> UUID {} // Stores connection in dictionary by that UUID
func commit(_ UUID) throws -> DatabaseUpdateResult // Commits transaction on connection associated with that UUID.

Extend parallel tests to include adding the same Entity to multiple batches. Add an outer loop to ParallelTests which takes a closure which creates the Batches Used in each ParallelTest. Test against three closures: Return EventuallyConsistentBatch, ConsistentBatch, and either/or on a 50/50 basis.

Another Possible Approach

Database operates either eventually or fully consistent. In the fully consistent implementation:

The Entity.sync feature returns (and Entity.async accepts a closure which accepts) an enum like

enum BatchSubmitResult {
    case accepted
    case rejected
}

Updates would be rejected if an Entity is already in another batch. In a more fancy implementation, firing the update closure on a dirty entity could be deferred until all previously scheduled dependent batches have committed, but updates which set up a batch deadlock would still need to be rejected.

Another approach:

protocol Batch{}
class EventuallyConsistentBatch : Batch {}
class FullyConsistentBatch : Batch {}

class Entity<T: Codable> {
    sync (batch: EventuallyConsistentBatch, closure: @escaping (inout T) -> Void) {}
    async (batch: EventuallyConsistentBatch, closure: @escaping (inout T) -> Void) {}
    sync (batch: FullyConsistentBatch, closure: @escaping (inout T) -> Void) -> BatchSubmitResult {}
    // TODO Add async fully consistent variant
}

This gives more flexibility to application developers but would need to handle the the following edge case if support for batch abort() is added:

1. Entity is added to a fully consistent batch.
2. Application attempts to add the (now dirty) Entity to an eventually consistent batch but does not because the entity is already in a fully consistent batch. In the usual case, this is ok because the entity will eventually be saved when the fully consistent batch is saved.
3. However, if the fully consistent batch is aborted, the system must ensure that the eventually consistent changes attempted in step 2 are saved.

This looks like a job for a dependency tree and topological sort.

Warning When Consistency is Broken:

When Entity is added to a fully consistent batch, set that batch into a var in the Entity. If that Entity is added to any other batch before the fully consistent batch commits, issue a warning that consistency has been broken but otherwise process as normal.

Version Access

A different but related feature is version access. This would enable read only "batches" which always work on a consistent version of the database. This enables more concurrent work because version accesses can work on a consistent version of the object model without blocking database updates.