search

EGA-archive / LocalEGA

A federated storage for sensitive data
http://localega.readthedocs.io
Apache License 2.0
7 stars 16 forks source link

Handling inbox files events #42

Closed silverdaz closed 5 years ago

silverdaz commented 5 years ago

The inbox notifications (from the OpenSSH-based inbox) now cover the file upload, removal and renaming. Directories are only concerned when they are renamed.

Describe the pull request:

Pull request long description:

The inbox based on OpenSSH is "externalized" and it sends notifications to the local broker instead of a custom TCP server (which in turn forwarded to the local broker).

The notifications are extended to cover file upload as before, and file removal and renaming. They contain also the filesize, a checksum* and the last update timestamp.

The location of the broker is configurable and we tried both the local broker (with AMQP) and CentralEGA hellgate (with AMQPS). Peer certificate and hostname verification is also configurable. This is interesting because we'll force the hellgate server to verify the clients too (soon...ish).

The bootstrap script is adjusted to pass the relevant variable to the inbox (for the moment, still as environment variables) and uses egarchive/lega-inbox:stable. :dev or :latest are used for developement.

The inbox isolates the users in their own chrooted-environment, and they do not see anything else.

* Note that we include checksums as an array because the verify service needs to send an md5 checksum to CentralEGA for stable ids (EBI's choice). md5 is old, prone to colison, insecure and doesn't inspire confidence. Sha256 is more suitable, so we send both. Since it is simpler for CentralEGA to reuse the same Java classes to parse the JSON messages, the inbox sends an array of checksums (with only the sha256 one).

Changes made:

1) No more notification TCP server. 2) Update on the inbox and its entrypoint 3) Configuration settings for the inbox to connect to any broker. 4) Bootstrap script updates 5) Tests updates (but only to fix the filepaths. They should start with /).

Related issues:

24 and #36

Mentions:

@blankdots

nanjiangshu commented 5 years ago

Looks good to me now.

silverdaz commented 5 years ago

@blankdots: Stefan, reviewing the code from https://github.com/EGA-archive/LocalEGA-inbox is of course important, but notice that the testsuite passes, so nothing is broken. There should normally be no need to make this PR wait, for another code to be reviewed.