MI-DPLA / combine

Combine /kämˌbīn/ - Metadata Aggregator Platform
MIT License
26 stars 11 forks source link

stateio: consider recording important errors to export and import manifests #342

Open ghukill opened 5 years ago

ghukill commented 5 years ago

One consideration for exporting has been how forgiving it will be for Combine components that are not well, or partially, formed. For example, earlier versions of OAI Harvest jobs did not record the id of the OAIEndpoint used in job_details, this was added later. However, the export should still allow exporting of these Jobs, even if that tether is severed. During export, this message is sent to logs with this key bit of information:

[WARNING] 2018-10-30 19:26:15,072 core.models._collect_related_components (7914): OAI parameters not found for HarvestOAIJob @ Jul. 19, 2018, 5:51:31 PM, export may not include required OAI endpoint

This was helpful to understand why OAI endpoints not exported, but is buried in logs. Perhaps export_manifest and import_manifest should include errors section with select messages like these written. This could populate a datatable, that is then searchable and sortable. If really thinking this through, could even be structured JSON with where this occurred (e.g. sections or line numbers).

ghukill commented 5 years ago

Additional example: when supplied export_path for import_state() does not contain a, or a valid, state to import

This would further require that export and import were wrapped in try / except that ensured the StateIO that will eventually house the manifests is written.

antmoth commented 4 years ago

Relevant to the 'export, nuke, reinstall' upgrade path