Story: Create an audit process for online Moab objects

[LynnMcRae]

This is the first level of audit, an inventory audit which just verifies consistency between Moab objects the PC Catalog inventory, that all copies that should be there are there.

Audits will need to report findings

• as current status information in the PC Inventory
• as a finding in Provenance information

Negative findings will need to trigger a recovery process. Does this initiate recovery automatically or simply report out so that recovery (manual or automated) is a subsequent process?

[julianmorley]

The audit process examines the content of the PCC, finds Moab objects, and checks to make sure that they:

• exist on disk (the online Moab is there), and if not, tell the Recovery process.
• have a recent record of a fixity check, and if not, tell the Fixity process.
• has recent records showing archive copies exist in the expected replication endpoints, and if not, tell the Archive process.

[ndushay]

possible tasks

• [ ] 10? fixture (yet real) moab objects put in a directory (read only)
• [ ] can read PCC for object name
• [ ] can read moab object directory
• [ ] can compare and figure out which objects exist in one, both or the other (maybe one by one per object; maybe comparing lists)
• [ ] can report out that it happened

[ndushay]

what is output of this process: where does it go? how is it processed? does it trigger something?

[julianmorley]

"what is output of this process: where does it go? how is it processed? does it trigger something?"

• outputs logging info for everything it does to the provenance database.
• when an expected online Moab is NOT found, it triggers/signals the recovery process
• when an expected archive Moab is NOT found, it triggers/signals the archive process
• when a recent successful fixity check result is NOT found, it triggers/signals a fixity workflow

[jmartin-sul]

spitballing a very rough possible approach to start off the development work:

• we have a directory with some fixture objects (hopefully copies of real life objects?)
• we spin up a rails app (?)
• we start by writing some methods that do the rough version of the work we want to do (read PCC for an object name, find out if the object exists in the moab directory, create a simple data structure with the status of that purported object in the moab directory)
  • start by putting those methods in the app's lib directory, or some models, maybe with the goal of turning this collection of methods into a gem that many apps can consume to get inventory/fixity status checking functionality?
• extend the app's functionality by having the checks run automatically, having the app expose the results to interested users via, e.g., a web UI, alert emails, etc.
• maybe actually break out the lib methods and/or models into a separate gem.
• ???
• profit

[julianmorley]

Diagram showing the part of the big picture that this story addresses: Preservation Core Simple Object Lifecycle - Story 2.pdf

And the Big Picture itself: Preservation.Core.Simple.Object.Lifecycle.pdf

[tingulfsen]

"The audit process examines the content of the PCC..." - it reads from the Object Inventory Store.

[tingulfsen]

Questions the audit process should be able to ask:

• Does it exist on disk?
• Can we fixity check it?
• What are the discrepancies at the object existence level?
  • By online drive
  • By single druid
  • By list of druids
  • By all known druids
  • By timestamp range
  • By arbitrary query (probably not in the MVP implementation)

[julianmorley]

I'd add:

• Does the online Moab version recorded in the OIS match the version found in the storage_root disk? ( And if it does not match, do something about that. Update the OIS record, tell Archive it has work to do )
• When was this online Moab last fixity checked? ( And if there's no evidence of a fixity check, or the check is stale, tell Audit it has work to do )
• What archive copies of this online Moab are recorded in the OIS? ( And if any of archive copies in the OIS are for older versions of this Moab, or no evidence of archived copies are found, or the evidence of the archive copies are a little old, tell Archive it has work to do )