Periodically enable GitLab cleanup policy

[dsotirho-ucsc]

https://github.com/DataBiosphere/azul/issues/5822#issuecomment-1868114168

Every four weeks, on every instance, the system administrator should set the number of clean-up workers to 4, wait an hour. Set number back to 0.

[dsotirho-ucsc]

Assignee to complete description and consider next steps.

[achave11-ucsc]

The data volume usage alarm recently went off in anvilprod as well, in addition to anvildev where it's been in alarm state for approximately two weeks.

[dsotirho-ucsc]

GitLab dev data use jumped from 72 to 77% today.

[hannes-ucsc]

Assignee to create issue template with four week schedule for the Monday between operator duty changes. Include link to each instance, e.g.,

https://gitlab.dev.singlecell.gi.ucsc.edu/admin/application_settings/ci_cd#application_setting_container_registry_expiration_policies_worker_capacity

[dsotirho-ucsc]

@hannes-ucsc: "GitLab recently added an alternative registry implementation which is currently in beta. The beta might have improved garbage collection which may make this ticket obsolete. Assignee to provide links and information."

[hannes-ucsc]

This is the respective documentation: https://docs.gitlab.com/ee/administration/packages/container_registry_metadata_database.html

This is the feedback issue: https://gitlab.com/gitlab-org/gitlab/-/issues/423459#supported-feature-status

From that issue it is evident that if we were to adopt this feature now, we would have to provide a dedicated PSQL database. See https://gitlab.com/gitlab-org/gitlab/-/issues/423459#supported-feature-status

This would be to much of an effort for us at this time. I'll create a new issue about migrating to the new registry database once it can make use of the omnibus-provided PSQL database.

[hannes-ucsc]

Assignee to proceed with the original plan: https://github.com/DataBiosphere/azul/issues/5908#issuecomment-1987534423