Implement message table 'sharding'

[cer]

In order to improve scalability of Producer and CDC, shard the message table using Message.PARTITION_ID (i.e. domain event aggregate ID):

• Multiple message tables - MESSAGE{hash(destination, partitionID) % numberOfOutboxTables} - improves scalability of producer and CDC, e.g. a polling thread per message table

• Add a partition column - MESSAGE{hash(destination, partitionID) % numberOfOutboxPartitions} - improves scalability of CDC, e.g. a polling thread per partition

[cer]

How this is configured

See this example

Eventuate MySQL 8 Docker image:

It has an environment variable - EVENTUATE_OUTBOX_TABLES - that specifies how many MESSAGE table shards to create.

See the 0.activate-additional-scripts.sh script

Service configuration

The Eventuate JDBC producer is configured using EVENTUATE_TRAM_OUTBOX_PARTITIONING_OUTBOX_TABLES env var (or the corresponding Spring Application context property eventuate.tram.outbox.partitioning.outbox.tables)

Eventuate CDC

It has an environment variable - EVENTUATE_CDC_OUTBOX_PARTITIONING_OUTBOX_TABLES - which specifies how many MESSAGE table shards to read from.

Please note:

• This has only been implemented for polling using the EventuatePolling profile and for single pipeline configuration

[cer]

To-do items:

• [x] Implement multi-pipeline configuration
• [x] Resolve outbox ID issue for DB generated IDs.
• [ ] Implement and test for reactive API
• [ ] Implement for binlog/WAL readers