Problem: Reingest can sometimes be unreliable

[ross-spencer]

Expected behaviour

Reingest should be less prone to failure. And must work for all transfer types.

Current behaviour

Failures can happen with reingest for a number of reasons, they might include:

• Issues with different compression algorithms.
• Incorrect information being written to pointer files.
• Pointer files not existing on a legacy system.
• Problems with the processing configuration affecting the desired outcome.

Additional context

Three primary components touch the reingest process:

• Archivematica
• Storage Service
• Automation-tools

This is an epic that we can use to capture all reingest issues/stories with the goal of improving this mechanism.

---

For Artefactual use: Please make sure these steps are taken before moving this issue from Review to Verified in Waffle:

• All PRs related to this issue are properly linked 👍
• All PRs related to this issue have been merged 👍
• Test plan for this issue has been implemented and passed 👍
• Documentation regarding this issue has been written and it has been added to the release notes, if needed 👍

[peterVG]

do-not-edit-start-codetree-epic-issues

Issues in this epic:

Title	Milestone	Assignees	Stage	State
Problem: API endpoint for approving reingests broke during the JSON-encoded workflow refactoring #1510	1.14.0	N/A	Open
Problem: GPG encrypted uncompressed packages are failing reingest #1431	N/A	N/A	Open
Problem: Partial- and full-reingest of an AIP containing a previously normalized file against a newly disabled rule creates an inconsistency in the AIP #1171	N/A	N/A	Open
Problem: AIPs are compressed twice during reingest #1095	N/A	N/A	Open
Problem: Partial reingest fails on previously reingested packages where files have been deleted through the re-normalization process (filegrp in METS is recorded as deleted) #1044	N/A	N/A	Open
Problem: Metadata-only reingest will not work under certain processing configurations, e.g. setting all processing options to be automated #959	1.14.0	replaceafill	Open
Problem: partial re-ingest removes logical structMap from METS file #876	N/A	N/A	Open
Problem: Snippet of code is incorrect in start_reingest in the storage service #863	1.14.0	N/A	Open
Problem: Partial reingest of AIP doesn't recognize manualNormalization directories #855	N/A	N/A	Open
Problem: There are pointer file inconsistencies post-reingest #766	N/A	N/A	Open
Problem: It is difficult to identify where reingest fails if it does so #740	N/A	N/A	Open
Problem: Package extraction can fail in the storage service if a pointer file hasn't been created #616	N/A	N/A	Open
Problem: Automation-tools reingest scripts do not handle metadata only use-case #496	N/A	N/A	Open
Problem: You cannot initiate a reingest with two different AIPs that share the same name #473	N/A	N/A	Open
Problem: Reingest will fail with normalized files when the original filename has a regex-like pattern in it (e.g. a filename with a single opening parentheses '(' ) #471	N/A	N/A	Open
Problem: Partial reingest fails during Process metadata directory, if AIP contains OCR files #336	N/A	N/A	Open
Reingest process from Uncompressed AIP to Compressed 7Z with BZip does not create zip #321	N/A	N/A	Open
Problem: Reingest via a pipeline (not via SS) does not lookup correct AIPStore #96	N/A	N/A	Open
Problem: DIP upload to ArchivesSpace fails after reingest #82	N/A	N/A	Open
Problem: Reingest assumes standard transfer type #51	N/A	N/A	Open
Problem: Assign UUIDs to objects works on a per file basis and during reingest that means it reads the METS file (n*files) times #1429	1.13.0	N/A	Closed
Problem: Reingest fails at "Store the AIP" when AIP pointer file contains an agentName with non-ASCII (Unicode) characters #1318	N/A	N/A	Closed
Problem: replicator does not replicate re-ingested AIPs or delete replicated copies of AIPS when deleted #985	1.13.0	sromkey	Closed
Problem: Partial (and Full) reingest removes mets:sourceMD from METS file #914	1.10.1	sromkey	Closed
Problem: Parse External METS shows an error when an external METS is not provided #886	1.13.1	N/A	Closed
Problem: metadata-only AIP re-ingest fails to initiate #863	N/A	N/A	Closed
Problem: 7z (copy) compression isn't represented in storage service #605	1.10.0	currmie, evelynPM	Closed
Problem: bagit created by Storage Service after reingest may have outdated tag manifest files #449	1.10.0	evelynPM	Closed
Problem: Reingest will fail given a Dataverse transfer type #107	N/A	N/A	Closed

do-not-edit-end-codetree-epic-issues