[Submit collection] Process daily the 25 oldest files and ship them to SCSB

[christinach]

What maintenance needs to be done?

Update the Submit collection job to process and send daily to SCSB the 25 oldest files.

Level of urgency

• [ ] High
• [x] Moderate because of https://github.com/pulibrary/lib_jobs/issues/763
• [ ] Low

Why is this maintenance needed?

• SCSB is able to process 25 files per day. @mzelesky has been moving 25 files per day manually to a directory that submit collection is using to process these files and send them to SCSB. Due to a large amount of Alma updates that will start coming the week of 4/22/2024 and the fact that we're already behind sending files to SCSB we need to update the code and support this change.

Acceptance criteria

• [x] Process 25 files per day. We should do this based on the timestamp value recorded in the Alma file name since Alma doesn't necessarily transfer the files via SFTP in order of creation. The format looks like this 36486988100006421_20240423_020409[022]_new_1 Per @mzelesky we should use the the file's actual timestamp not the name.
• [x] Ignore the deletion files in the count of 25 files. These files are not processed via the SubmitCollection interface. Items deleted will be withdrawn manually by holdings staff through SCSB's de-accession interface.

Notes

This was discussed and decided in the Orangelight meeting on 4/18/2024

[kevinreiss]

We likely need to edit the file glob that grabs the 25 files to exclude files with the delete naming pattern.

[kevinreiss]

We likely need to edit the file glob that grabs the 25 files to exclude files with the delete naming pattern.