RBMHTechnology / eventuate

Global-scale event sourcing and event collaboration with causal consistency (This project is in maintenance mode. Only critical bugs will be fixed, but there is no more feature development.).
http://rbmhtechnology.github.io/eventuate/
Apache License 2.0
708 stars 99 forks source link

Decouple replication batch size from write batch size #292

Closed magro closed 8 years ago

magro commented 8 years ago

We recently had to reduce the eventuate.log.write-batch-size (to 2 if that's relevant) to prevent "com.datastax.driver.core.exceptions.InvalidQueryException: Batch too large" - because we're running a C* version that does not yet have the fix for CASSANDRA-10876. As a consequence performance tests failed and we saw that event replication between locations increasingly diverged. This seems to be caused by the fact that for the replication batch size the eventuate.log.write-batch-size is used as well.

IMO the batching layer batch size (log/persistence) and the replication batch size should be independently configurable.

Right now it's fairly easy: the replication batch can be written as a batch - atomically. If the replication batch size could be greater than log batch size several batches would have to be written. So there are probably various options to deal with failures regarding the replication acknowledgement, what currently comes to my mind:

krasserm commented 8 years ago

@magro what's your priority for this ticket? The issues described here are addressed by #196 and #314. If it's not a blocker or high priority I'd like to close this ticket and defer support for it until work on the other two tickets start. Even if CASSANDRA-10876 has not been applied to your Cassandra version, it shouldn't harm increasing the batch size limits as explained batch write failure handling if Eventuate is the only writer to Cassandra. WDYT?

magro commented 8 years ago

This is not high priority, we can close it in favor of #196.