Validation (fixity checking) using Archivematica Storage Service

crkn-rcdr / Digital-Preservation

Documentation and related schemas for the CRKN digital preservation system

3 stars 0 forks source link

Validation (fixity checking) using Archivematica Storage Service #2

Closed RussellMcOrmond closed 5 years ago

RussellMcOrmond commented 5 years ago

Canadiana currently does fixity checking on a regular schedule for all copies of every AIP. A system as robust will need to be created as we move to a new system for repository storage.

RussellMcOrmond commented 5 years ago

Canadiana currently does the following:

We currently have 6 different servers (5 online, 1 offline) with copies of each AIP
We have a database (CouchDB) which has a record for each AIPs on each server, and this includes a field for the most recent time fixity was run
We launch a service on each server at 17:00 which runs for approximately 12 hours. This pulls the least recently verified and runs a bagit check (On most servers we run 8 checks in parallel).
Depending on the server the fixity checks for approximately 314000 AIPs take from 20 to 40 days

RussellMcOrmond commented 5 years ago

The Storage Service has a fixitylog.

I haven't found an API call that makes this visible, but is visible in a /packages/ view for the web interface.
We may need to create a new API call that returns the "N" least recently checked AIPs (Status="Uploaded" sufficient?)
We currently do many thousands of AIP bagit checks each evening. I'm not certain a log that retains all checks will be appropriate, and we may need to create some sort of expiry of past logs.

RussellMcOrmond commented 5 years ago

Similar to #1 , we opted to author direct Swift support.