Description
The archival flow has been implemented in #40 but verification is missing. As a user of the archiver I want the archived data to be to be verified. If the verification failed, an error should be reported and the flow fail.
Additionally, there should be a check of the available disk size (df -h > 80%) before starting the writing operation in order to prevent failures. The task should wait to be started until there is enough free disk space.
Definition of Done
Each written datablocks are verified (via checksum) after copying to the LTS
Unit test implemented for verification and disk space check
New version is deployed on scopem-openem.ethz.ch
Additional context
The LTS provides an nfs share that can be written to, see #57
There are 2 options for verification:
Option 1: Checksum calculation started after 8h, storage team can provide service to retrieve sha-256 checksum
Option 2: Copy file back from share and verify checksum of retrieved file. Retrieval needs to happen one-by-one
Option 2 is more practical.
Note: in the future this will probably be done using the S3 protocol; the correct abstraction should be used when implement this mechanism.
phwissmann
commented
4 months ago
Description The archival flow has been implemented in #40 but verification is missing. As a user of the archiver I want the archived data to be to be verified. If the verification failed, an error should be reported and the flow fail.
Additionally, there should be a check of the available disk size (df -h > 80%) before starting the writing operation in order to prevent failures. The task should wait to be started until there is enough free disk space.
Definition of Done
Additional context The LTS provides an nfs share that can be written to, see #57
There are 2 options for verification:
Option 2 is more practical.
Note: in the future this will probably be done using the S3 protocol; the correct abstraction should be used when implement this mechanism.