Problem: DIPs are not cleaned up from watched directories

[replaceafill]

Expected behaviour

DIPs are always removed from watched directories after a transfer finishes.

Current behaviour

DIPs are placed in two watched directories:

• uploadDIPs: contains DIPs to upload to an access system and triggers the upload
• uploadedDIPs: contains copies of the DIPs that were uploaded (or tried to be uploaded) and even though it doesn't trigger any processing, it allows users to browse the contents of the DIP

In the following scenarios, at least one of these copies is left behind:

• Uploading a DIP but not storing it leaves both copies
• Uploading a DIP and storing it in a Storage Service space that doesn't support "moving" (i.e. S3 spaces) leaves the uploadedDIPs copy
• Uploading a DIP and storing it might fail deleting the uploadDIPs copy without any error

And although it's not related to watched directories there is also a conflict between rejecting the AIP and the DIP at the same time:

• If the DIP is rejected before rejecting the AIP, the rejected DIP is overwritten
• If the AIP is rejected before a DIP is rejected, the DIP is placed as a subdirectory of the rejected AIP

Steps to reproduce

1. Set the processing configuration to create access copies.
2. Upload a DIP to an access system.
3. Do not store the DIP.
4. Check the uploadDIPs and uploadedDIPs watched directories after the transfer finishes. You'll see both copies of the DIP.

Your environment (version of Archivematica, operating system, other relevant details)

https://github.com/artefactual/archivematica/commit/2a13924d724a0f9fb4297f995441d6f02368803f

---

For Artefactual use:

Before you close this issue, you must check off the following:

• [ ] All pull requests related to this issue are properly linked
• [ ] All pull requests related to this issue have been merged
• [ ] A testing plan for this issue has been implemented and passed (testing plan information should be included in the issue body or comments)
• [ ] Documentation regarding this issue has been written and merged (if applicable)
• [ ] Details about this issue have been added to the release notes (if applicable)

[replaceafill]

We tested and compared these scenarios using the am115jammy and am116jammy servers and they work as expected now.

[fitnycdigitalinitiatives]

Hello,

I think this has inadvertantly broken the ContentDM upload process. If the transfer completes before that transfer is initiated, the DIP stored in uploadedDIPs is deleted and thus cannot be uploaded/transfered to ContentDM.

[sromkey]

@fitnycdigitalinitiatives thanks for pointing this out- I can think of three workarounds:

1. In addition to configuring Upload DIP to ContentDM, also set your processing configuration to store the DIP and retrieve it from the Storage Service.
2. Set your process configuration to not store the AIP until you choose to do so manually, to give yourself time to retrieve the DIP.
3. Set your process configuration to not store the DIP and then you can retrieve it from the Rejected directory (seems like kind of a hack!) I hope one of these works for you- apologies for forcing a change in your workflow. We're trying to be more consistent about the separation of concerns between the Archivematica dashboard and the Storage Service and not rely so much on leaving things in watched directories, which in our experience causes performance and clean up problems. This also reminds me that we should update the docs to reflect this, so thanks for that also!

[fitnycdigitalinitiatives]

Thanks Sarah, I did actually end up just tweaking our automation script to grab it from the rejected directory. Everything works the same in the end and is it a bit tidier with this workflow.