Automatically clean up the /alma directory

pulibrary / bibdata

Local API for retrieving bibliographic and other useful data from Alma (Ruby 3.1.0, Rails 7.0)

BSD 2-Clause "Simplified" License

16 stars 7 forks source link

Automatically clean up the /alma directory #2401

Closed rladdusaw closed 12 hours ago

rladdusaw commented 1 month ago

User story

The /alma directory has a lot of old files and is using too much disk space. We should setup an automated job to clean up old files.

Acceptance criteria

Create a periodic task to delete files that have been on the disk more than x months

[ ] Create a tasks to clean up files in the /alma directory
[ ] Schedule this task to run as a scheduled task

kevinreiss commented 3 weeks ago

acozine commented 1 week ago

The /alma/* directories live on the lib-sftp-* servers. Both lib-jobs and bibdata access the directories. If we put the cleanup task in the SFTP role, aka roles/system_ldap, it should apply to all projects that access Alma data.

kevinreiss commented 1 week ago

@mzelesky what should be our cut-off date for purging old files? Three months from today?

christinach commented 1 week ago

Will discuss this in the next Alma DACS meeting on 7/9/2024.

christinach commented 6 days ago

Discuss with the Alma tech team:

how many weeks worth of files do we want to keep?
Is it the same for all directories under /alma ?

mzelesky commented 6 days ago

In general, 3 months is a good rule, except for the following directory:

/alma/invoices: 1 year, since I just uncovered that Alma recalculates expenditures for foreign currencies when it shouldn't; the invoice export is the only way I have as of now to find out what we submitted to Prime for payment