linagora / tmail-backend

GNU Affero General Public License v3.0

31 stars 17 forks source link

Applicative level synchronization between two S3 #980

Open chibenwa opened 3 months ago

chibenwa commented 3 months ago

Why

Achieve real time synchronosation between 2 S3 buckets while "Asynchronous Replication" is not supported by the S3 provider

How

Write a BlobStoreDAO that write synchronously into 2 distinct S3BlobStoreDAO

Write path

Write in parallel in blobStore A and B, return any error.

Read path

Read from A, fallback to B in case of error.

In case of unavailability of A or B in order to recover writes, explicit reconfiguration to use only blob store A (or B) would be needed.

chibenwa commented 3 months ago

Note from @pvi-github

We can reuse the event bus from TMail and add the following events: BlobAdded and BlobDeleted to asynchronously manage the replication.

chibenwa commented 3 months ago

Failure mode anaysis:

S3 site 2 is unavailable: reads are unnafected on S3 site 1, writes are unaffected on S3 site1, replication queue buffered in RabbitMQ, resuming when site 2 will come back online.
S3 site 1 is unavailable: reads are unaffected as they will fallback on site 2. Writes will fails. Manual intervention will be required to swap site 1 and 2 in the configuration to resume operations (and run a manual sync when site 1 come back online)
- RabbitMQ is unavailable: No problem, James event bus handles this situation by persisting items it could not dispatch into Cassandra (EventDeadLetter) for a later catch up.

chibenwa commented 3 months ago

S3 site 1 is unavailable: reads are unaffected as they will fallback on site 2. Writes will fails. Manual intervention will be required to swap site 1 and 2 in the configuration to resume operations (and run a manual sync when site 1 come back online)