Storage "Fix" Framework

[terrywbrady]

Note: Requires new Inventory mrt-zk implementation.

The Merritt system will implement a framework for applying fixes to cloud storage and cloud storage keys.

Fix Operations

• New operations can be defined as needed
• The fix operation should create a new system file named "mrt-fix.log" that records the details of the fix that is applied

changeToken

fix Nuxeo cloud storage keys that contain a "changeToken" url parameter

Backup Node

A new storage node will be created at SDSC for object backup. This bucket will contain a backup of an object. This is not an object replica.

Storage Endpoint: POST /fix/NODE/ARK?fix=FIX_OPERATION&run=true

• Object is backed up to the backup node
• Fix is computed
  • if run=true, fix is applied, mrt-fix.log is added to system files
  • if run=false, mrt-fix.log is returned

Merritt Admin Tool

Queuing logic is TBD

• Admin tool will invoke the storage fix operation
• If successful, Admin Tool will trigger inventory to rebuild the object
• If successful, Admin Tool will trigger re-replication of the object

[terrywbrady]

Moving to Sprint 103 since David is not working on this yet.

[dloy]

Some issues to consider:

• does "fix" require a branch of the zookeeper handling from
  • ingest request
  • storage processing
  • inventory deletion and re-add
  • replication cleanup
• guarantee that a "fix"ed object is not broken by this process
  • if the error in fileID handling is corrected (?changeToken=xxx is removed in ingest handling) - some fixed earlier versions may be replicated later as part of the cleanup process
  • ark locking is required to prevent updates happening during the actual fix

Content

A couple of content issues to note from Nuxeo testing

• Recent changes have been made to only include a single mrt-erc.txt (system/mrt-erc.txt). Earlier versions of many objects include both producer/mrt-erc.txt and system/mrt-erc.txt. The reduced file count looks like an error - which it isn't.
• Files with a common fileID (e.g. xxx.xml) can appear on each version of an object but are not identical between versions and do not collapse in delta handling. The .xml in these Nuxeo objects are physical metadata descriptions for the .tiff files and include pointers to changing storage blob content that appears to be part of the stored. A new xml file is stored each version.

[dloy]

Question of one-off vs. zookeeper control vs, db monitor

I see the procedure for handling Fix falling into 3 categories:

• one-off local management
• program controlled managing with zookeeper flow control
• one-off with db status

one-off local management

Process

A simple process that will probably not be duplicated moving forward.

For changeToken this would be a series of API calls with verification:

• get ark
• do copy - store
• do fix - store
• delete db object - inventory
• update inv db - inventory
• delete secondary - replic
• update replic - db initialization

Advantage

• only requires minimal code changes - existing API controls already exist
• can be tossed on completion because little generality

Disadvantage

• little monitoring exists outside of mostly manually started handling
• error tracking difficult

programmed control flow

Process

A flow controlled process is created to control what happens next. The flow control is at the same level as update - copy is not an API but rather a step in the flow control

For changeControl - create zookeeper flow that includes each step normally handled by each service.

This includes:

• based on an API call to do a changeToken update - ingest queues a zookeeper entry to do fix
• based on local zookeeper queue entry to do fix - ingest calls storage with specialized fix handling
• based on storage response - ingest queues a special inventory call to do fix handling in inventory (calling mechanism already exists for this but requires specialized URL)
• inventory grabs queue (as current) - identifies the special features required by the url to do a programmatic deletion of an object and then do an add
• replic would need to add zookeeper handling - which it does not currently do
• based on a new zookeeper queue - different design feature - does delete of secondary and adds
• in all cases a failure is noted and manual queued for intervention

Advantage

• once in place guarantees that steps are not bypassed in process
• works well if all future changes take a similar queueing path
• identifies errors

Disadvantage

• if specialized process flows requires enormous amounts of program flow logic and testing for what may be a one-off
• not dynamic

one-off with db status

Process

Identical to one-off local management but includes an additional step after API calls to report to a db Process Table an ok completion or of an error that occurred during the one-off handling.

Rerun logic on this would update the single db entry.

This db entry update would be either a direct update or more likely an API call (replic?) to modify the single entry for this item (ark?)

Process Table entry format

• id - automatic
• process_name - name for the overall one-off project - e.g. changeToken
• item_name - name of specific item being processed - e.g. ark or S3 key or ...
• date_process - datetime for entry
• status - ok, fail, ignored
• fail_info - optional - gives fail information to identify specific cause of a fail status

The Process Table would require periodic cleanup.

Advantage

• only requires minimal code changes - existing API controls already exist
• can be tossed on completion because little generality
• provides a common place that can be searched for local status to guarantee completion and testing against other db tables

Disadvantage

• creation and maintenance of table would be needed
• creation of API call to update table
• adds additional step to be called when error processing between API calls (e.g. in a Bash/Perl/... script)

My Recommendation

Go with the Process Table model. Requires additional setup for creation of table and setting values

• it should be possible to apply it in any one-off environment.
• it would give good accounting for missed and failed content by allowing MySQL matching of content with content in other table (e.e. objects with changeTokens)