Maintenance job should also call maintenance.sh script and others

[pcm32]

Currently the maintenance job https://github.com/galaxyproject/galaxy-helm/blob/master/galaxy/templates/cronjob-maintenance.yaml deletes the galaxy tmp. We should probably add there:

• execution of maintenance.sh script with a configurable days parameter.
• execution of the pgcleanup script to clean up older elements in the database.
• /usr/bin/tmpwatch -v --mtime --dirmtime 7d on the jobs working directory (or at least on _cleared/, which is the result of the resubmissions - the initial jobs that didn't work).

[ksuderman]

I am going to start work on this pronto as I just ran out of space on a 2TB volume. I will add as many of the jobs that main uses as I can.

https://github.com/galaxyproject/usegalaxy-playbook/blob/main/env/main/host_vars/galaxy-web-05.tacc.utexas.edu.yml

[nuwang]

That's great! It's also worth checking whether we can adapt something like this: https://github.com/usegalaxy-au/infrastructure/pull/1733/files#diff-cf003cea0be89303fa5922fc7be3947c14bb07c1d32add1e233381589e7be2aeR82

Instead of using the max wall time, it checks the oldest running job to determine what to clean.

[ksuderman]

I just saw your comment and we could definitely do something like that. We may also want to check the job status, but baby steps.