Prevent empty incremental files from stopping indexing - Githubissues

pulibrary / bibdata

Local API for retrieving bibliographic and other useful data from Alma (Ruby 3.2.0, Rails 7.1.3.4)

BSD 2-Clause "Simplified" License

16 stars 7 forks source link

Prevent empty incremental files from stopping indexing #2403

Open sandbergja opened 3 months ago

sandbergja commented 3 months ago

Expected behavior

Bibdata doesn't attempt to index empty incrementals from Alma

Actual behavior

It does

Impact of this bug

Indexing was stuck, so data in the catalog data was out of date. We had to manually troubleshoot.

success criteria

[ ] Empty files from Alma don't get fetched into bibdata
[ ] We don't try to index empty files
[ ] We log an error that there was an empty file

Implementation notes, if any

We talked about adding this check to the AlmaTransferJob: https://github.com/pulibrary/bibdata/blob/main/app/jobs/alma_dump_transfer_job.rb#L42-L48
We already create an MD5 for the file -- maybe we could check the size at the same time?

Related ticket: https://github.com/pulibrary/bibdata/issues/2402

kevinreiss commented 1 month ago

The root cause that brought this issue to light was disk space on lib-sftp-prod, we've addressed this through regularly running a clean-up ansible tower process.