Checking uniqueness within Jobs / Published

[ghukill]

Breaking out the issue of checking uniqueness among records to new issue.

Now that combine_id has been established (or nearly), this adds different options for checking "uniqueness" among Records within Jobs, and among all Published records.

Uniqueness might have multiple meanings:

• unique in terms of combine_id
  • related to the Record's lineage in Combine, but semantically not meaningful
• unique in terms of record_id
  • which mean unique in terms of the published identifier
• unique in terms of Record document XML
  • something that has not yet been explored
  • could "fingerprint" or hash the document in some way to try and check for uniqueness in the Record's actual document contents
  • see "Considerations" below

Considerations

• adding column of MD5 hash of actual Record XML, something like doc_hash?
  • could alter how uniqueness is determined, pivot on this versus identifiers
  • would do this while saving records
    • but only for Jobs that can modify document (harvest, transform)
    • wouldn't have to even for jobs that modify record_id...
• check uniqueness window operation in save_records(), probably should use combine_id?
  • or considered MD5 column (see below)?

Uniqueness in Published Records

• probably makes sense to check uniqueness based on record_id, as this will directly effect the built-in OAI server (and any other publishing mechanisms)

[ghukill]

Some anecdotal testing has supported the idea of keeping record_id as the marker of uniqueness. When transforming record identifiers (https://github.com/WSULib/combine/pull/128), there is the potential for duplicating identifiers, and this would identify that undesirable situation quite nicely.

[ghukill]

Closing, with the potential to reopen if need be.