Problem: Remove all Completed SIPs not removing all completed SIPs

[evelynPM]

Expected behaviour Remove All Completed SIPs should remove all SIPs that have been stored as AIPs or that have failed and stopped processing

Current behaviour A message appears that says "There were no completed SIPs to remove" despite the fact that there are some SIPs that have been processed into AIPs and placed in storage. See attached screenshot.

Steps to reproduce Go to ingest tab and click the red button at the top right-hand corner. When I did it, there were a variety of SIPs in the tab, some completed and some not.

Your environment (version of Archivematica, OS version, etc) 1.10 bionic qa environment

Connects to https://github.com/archivematica/Issues/issues/1419.

---

For Artefactual use: Please make sure these steps are taken before moving this issue from Review to Done:

• All PRs related to this issue are properly linked 👍
• All PRs related to this issue have been merged 👍
• Test plan for this issue has been implemented and passed 👍
• Documentation regarding this issue has been written and it has been added to the release notes, if needed 👍

[sallain]

The issue that Evelyn reported above seems to happen in two scenarios:

1. The task gets stuck on incomplete SIPs. It seems like the task moves through the list of SIPs sequentially and then gets stuck on those jobs - for example, if you have a list of SIPs like this:

completed-SIP-1
completed-SIP-2
rejected-SIP
completed-SIP-3

Then SIPs 1 and 2 will be cleared, but when the task reaches rejected-SIP it stops. The task is supposed to skip over the incomplete SIPs and move on to the next completed SIP (this is how it works on the Transfer tab).

2. If a DIP was generated, the SIP cannot be cleared. It doesn't seem to matter if the DIP was uploaded, stored, or unstored - if the completed SIP includes a DIP, the feature balks on that SIP in the same way as above.

(For clarity, "completed SIP" is used to mean an AIP that has stored without error. "Incomplete SIP" means a SIP that failed, was rejected, or is still in progress.)

[replaceafill]

I think it all comes down to the sorting logic in this helper which determines what units are complete and how it considers a SIP complete only if the Remove the processing directory job (part of the Microservice: Store AIP) is the first (most recent).

If the Microservice: Upload DIP is listed first (because you answered the DIP storage questions last for example), the SIP will not be considered complete by the helper.

[evelynPM]

I just confirmed that this issue is not a regression from 1.9.x, so I'm removing the triage-release-1.10 label.

[sallain]

We also found this issue in 1.11 and 1.12