Storage "fix" - Inventory "replace" methods

dloy commented 8 months ago

Problem

There are specific content issues that because of our data architecture require different approaches for cleanup.

Examples:

"changeToken" property in file Paths - impacts S3 data keys, S3 manifest.xml, inventory replace
outdated mimeTypes - S3 Manifest change, inventory replace
owner changes - S3 system key changes, inventory replace

A generic way of fixing all of these is not clearly defined (at this point)

Proposal

Use a storage "fix' command with inventory "replace".

storage "fix" uses specific storage code for a particular correction
inv "replace" deletes the original content for an ark and rebuild

Storage fix

A proposed format for this command:

curl -X POST http(s)://<storage service>:35121/fix/<node>/<ark>?type=<identifier of code to be applied to a specific object to correction>

/fix/ - indicates this is a fix process
node - node containing object
ark - identifies of object to be fixed
type= - identifies a specific piece of code to be applied to an exist /

When storage receives this request:

lock ark for correction
call specially implemented code to perform the identified fix

Inventory replace

A proposed format for this command:

curl -X POST -F "url=<storage service>:35121/manifest/<node>/<ark>" -F "responseForm=xml" http://<inventory service>:36121/replace

/replace - multipart request that forces a delete before the standard add is made

System fix

CRITICAL TO THIS PROCESS is that the system has corrected this problem for all subsequent versions

Example ?type=changeToken

On receiving this request, node and ark are identified
If content is not found for this node and ark then an exception
Manifest for ark is expanded to VersionMap

For all versions of object

identify any file path containing changeToken
remove changeToken content from file path: extracted file path
determine if extracted file path exists in any previous version - if so use reference to earlier version and remove the duplicated content
if no previous version of extracted file path then copy content to new extracted file path key

After:

replace call is made to inventory to delete and re-add the corrected version.

The result of this process is:

object with all previous version content represented but without duplicates caused by changeToken
corrected object

As mentioned above a changeToken fix needs to be made in the Nuxeo handling so all future versions will not include the changeToken as part of the file path name.

Comment

This approach allows a specific fix to be applied to storage content without requiring a generic approach which is not clearly understood.

The fixed content will allow all generated files to meet current restrictions: e.g. manifest.xml

Using this approach the fixes can be applied now. As these fixes get applied and identified, later generic methods or architectural changes can be made.

Note: Requires new Inventory mrt-zk implementation.

dloy commented 8 months ago

Goal: changeToken

Recreate object in Merritt as if the ?changeToken=value had not been attached as an extension on the fileID

Process

Requires that conditions causing changeToken problem no longer exist for future versions

for each version
- for each file component for each version
- if no ?changeToken=value included in fileID then output as is
- if ?changeToken=value found
  - remove changeToken=value from fileID
  - test if file component matches any previous file component in previous version testing fileID, length and digest
  - if file matches then use the the earlier version key for the current version
  - if file does not match then
    - add file component content to S3 with the current fileID key
    - retest Tika
create new manifest.xml
replace new manifest.xml
delete all S3 content from deleted file components
delete inventory for old object
create new inventory for new object

dloy commented 6 months ago

Commands

Storage (possible Ingest)

API to process in place changeToken update to an existing Merritt Object

curl -X GET "http://storage.cdlib.org:35121/fix/6501/ark%3A%2F13030%2Fm56t6p2z?type=changeToken"

API - changeToken fix

http://<storage host:port>/fix/<base node number>/<ark>?type=changeToken

Inventory

API to reset modified changeToken object in Merritt inv database

curl -X POST \
-F "url=http://storage.cdlib.org:35121/fix/6501/ark%3A%2F13030%2Fm56t6p2z?type=changeToken" \
-F "responseForm=xml" \
http://inventory.cdlib.org:36121/replace

API - inventory replace

curl -X POST \
-F "url=<storage fix url> \
-F "responseForm=xml" \
http://<inventory host:port /replace

CDLUC3 / mrt-doc