EGA-archive / LocalEGA

A federated storage for sensitive data
http://localega.readthedocs.io
Apache License 2.0
7 stars 16 forks source link

Robust shovel #106

Closed silverdaz closed 4 years ago

silverdaz commented 4 years ago

Describe the pull request:

Pull request long description:

Adding a mechanism to not lose the messages when LocalEGA is using the shovel to CentralEGA and CentralEGA is down (for a while, after a restart, for example).

The messages were backed by an ephemeral queue, but they are now backed by a durable queue and a shovel moves them to Central EGA (with the right routing key) when the connection succeeds.

This is akin to a (reverse) federated queue, but lower-level and we don't need credentials to the LocalEGA instances

One more test was added for it: Central EGA goes down when the ingestion is complete, and the message is not lost after the restart.