Many stale images in gitlab-dind and GitLab registry #5728

Closed dsotirho-ucsc closed 7 months ago

GitLab container registry

Screen Shot 2023-11-27 at 2 56 15 PM

gitlab-dind

[ec2-user@ip-172-71-0-215 ~]$ sudo docker exec -it gitlab-dind docker image ls --format=json | jq -r '[.CreatedAt, .ID, .Size, .Repository, .Tag] | join("\t")' | sort -u -k 1,1
2020-05-08 16:48:14 +0000 UTC   3c0d8004a1fb    1.43GB  rycus86/pycharm 2019.3.5
2020-09-10 18:32:25 +0000 UTC   3687eb5ea744    871MB   python  3.5
2020-10-09 18:00:33 +0000 UTC   d1226e1554f8    60.4MB  gitlab/gitlab-runner-helper x86_64-264446b2
2020-12-05 01:07:53 +0000 UTC   558380375f1a    774MB   122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.elastic.co/elasticsearch/elasticsearch  7.10.1
2020-12-29 02:47:00 +0000 UTC   97396fa3d959    1.42GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  999
2021-03-25 22:21:58 +0000 UTC   a9275369af8c    70.2MB  gitlab/gitlab-runner-helper x86_64-54944146
2021-03-31 05:03:23 +0000 UTC   61560e1cedb8    911MB   node    14.16.0-buster
2021-07-16 19:12:58 +0000 UTC   ae0f95f06fc6    1.38GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  6638
2022-01-24 19:33:44 +0000 UTC   40d5e650e7cc    66.9MB  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-98daeee0
2022-01-31 18:37:35 +0000 UTC   fb1b155d1740    1.34GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  12319
2022-02-26 08:37:55 +0000 UTC   061d986990f8    1.74GB  clevercanary/ng-cli-karma-python3   13.2.5
2022-03-16 20:15:09 +0000 UTC   b083053d75df    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  12425
2022-05-11 08:10:47 +0000 UTC   dec08d7606c5    889MB   python  3.9.12-buster
2022-05-26 22:58:09 +0000 UTC   5d91a9c26bbf    224MB   docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/runner   <none>
2022-07-24 23:03:12 +0000 UTC   c526698c011d    67MB    registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-76984217
2022-07-25 21:19:00 +0000 UTC   f1148a65c439    1.37GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  16191
2022-08-08 23:13:45 +0000 UTC   1b7f75c03529    1.37GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  16169
2022-09-03 12:03:37 +0000 UTC   41de1ac06681    67MB    registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-32fc1585
2022-09-06 19:43:18 +0000 UTC   8b3e68cd646f    67.1MB  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-bbcb5aba
2022-09-20 03:28:10 +0000 UTC   5b55ab01aee9    1.37GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  17244
2022-09-21 01:42:18 +0000 UTC   51ba51a1ce23    1.38GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  17275
2022-09-26 21:06:14 +0000 UTC   fcf176c12733    1.37GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  17427
2022-10-04 00:03:35 +0000 UTC   3941ee56c787    67.1MB  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-43b2dc3d
2022-10-05 01:32:15 +0000 UTC   59b6edd21672    1.38GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  17634
2022-10-06 19:55:40 +0000 UTC   5c9831000917    1.38GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  17681
2022-10-07 23:11:03 +0000 UTC   725f4074d0f3    1.38GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  17711
2022-10-10 16:07:25 +0000 UTC   3226a570e615    1.38GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  17778
2022-10-11 15:58:56 +0000 UTC   32326086d7cf    1.47GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  17803
2022-10-12 18:26:12 +0000 UTC   527feb574ba7    1.47GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  17834
2022-10-20 13:48:46 +0000 UTC   7e0c76d16fae    1.47GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  18034
2022-10-21 15:38:46 +0000 UTC   e56f70af145b    1.46GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  18062
2022-10-25 18:42:30 +0000 UTC   f603b96af596    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  18172
2022-10-26 23:58:42 +0000 UTC   3f13747542eb    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  18205
2022-10-28 00:03:18 +0000 UTC   d453be8dd58f    67.1MB  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-526d939d
2022-11-02 21:03:20 +0000 UTC   738d1c064670    67.1MB  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-0d4137b8
2022-11-07 21:12:38 +0000 UTC   1cf3c70778ef    1.4GB   docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  18503
2022-11-18 06:38:37 +0000 UTC   74c96e4e1362    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  18771
2022-11-24 00:11:07 +0000 UTC   96f188680df6    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  18915
2022-11-28 19:03:07 +0000 UTC   83c51133dca7    68.2MB  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-133d7e76
2022-12-01 21:04:58 +0000 UTC   4cc8ac9ddd5f    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  19115
2022-12-02 01:34:50 +0000 UTC   9d983dff995d    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  19121
2022-12-05 01:13:55 +0000 UTC   319a87053249    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  19197
2022-12-09 23:36:39 +0000 UTC   3525caf31556    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  19331
2022-12-10 08:15:06 +0000 UTC   06debe7667d6    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  19342
2022-12-20 18:15:30 +0000 UTC   836ea59e6c18    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  19616
2023-01-04 00:13:10 +0000 UTC   24bb22eac4bc    68.2MB  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-6d480948
2023-01-05 18:40:49 +0000 UTC   a6d359c125bd    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  20008
2023-01-07 01:25:53 +0000 UTC   4e15cf015aea    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  20043
2023-01-31 22:03:11 +0000 UTC   7623e8322503    68.5MB  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-12335144
2023-02-09 00:39:01 +0000 UTC   27802f026775    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  20886
2023-02-22 20:28:02 +0000 UTC   981976c04568    1.38GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  21228
2023-02-27 22:33:14 +0000 UTC   5a78ae04d0ee    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  21352
2023-03-01 23:03:12 +0000 UTC   80629a0b91d7    68.9MB  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-d540b510
2023-03-12 05:55:29 +0000 UTC   176fbc4c51ac    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  21669
2023-04-05 15:03:09 +0000 UTC   ca9feba42316    64.6MB  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-dcfb4b66
2023-04-24 21:30:38 +0000 UTC   a8cdefaad713    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  22811
2023-05-03 17:49:53 +0000 UTC   2533e7e29b79    65.8MB  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-436955cb
2023-05-16 01:40:26 +0000 UTC   fbcef6a49716    2.3GB   122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/ucscgi/azul-pycharm  2022.3.3
2023-05-23 23:09:54 +0000 UTC   6fa62d04dbd7    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  23582
2023-06-01 03:07:03 +0000 UTC   34e4a6c9b6c4    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  23797
2023-06-04 18:18:52 +0000 UTC   06f060df1367    1.43GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  23892
2023-06-05 02:22:08 +0000 UTC   9649369652f5    1.43GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  23902
2023-06-06 16:18:53 +0000 UTC   62b6da9285df    906MB   122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/library/python   3.9.17-bullseye
2023-06-08 02:23:08 +0000 UTC   6f1b00af1450    65.9MB  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-79704081
2023-07-04 01:03:11 +0000 UTC   726619a14dc7    65.9MB  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-85586bd1
2023-07-07 16:03:10 +0000 UTC   e0c2aa911388    1.44GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  24793
2023-07-12 02:42:06 +0000 UTC   73e4a004cf47    1.44GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  24913
2023-07-13 17:58:35 +0000 UTC   66bab67d3472    1.44GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  24960
2023-07-15 02:20:39 +0000 UTC   2c41bbfcfa6d    690MB   122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/ucscgi/azul-elasticsearch    7.17.10-2
2023-07-17 22:04:46 +0000 UTC   39364702171f    1.44GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  25073
2023-07-18 01:35:17 +0000 UTC   579831f6f60b    161MB   docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/runner   <none>
2023-07-20 22:14:15 +0000 UTC   72fc9ca5c2f3    1.44GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  25154
2023-07-21 22:58:41 +0000 UTC   b66be4c31050    1.47GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  25185
2023-07-26 21:04:53 +0000 UTC   ee523fcbaf6b    1.44GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  25316
2023-07-28 17:40:05 +0000 UTC   8a3d4194b8e0    1.49GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  25376
2023-07-29 02:25:33 +0000 UTC   c5a24a31b529    1.49GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  25392
2023-08-01 17:04:54 +0000 UTC   b39e718928fb    1.49GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 25487
2023-08-07 23:59:12 +0000 UTC   2f3ef643ffbe    1.25GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/deps 25657
2023-08-08 21:50:00 +0000 UTC   085db4d51eb0    1.49GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 25682
2023-08-24 17:49:56 +0000 UTC   0bfb45a2011a    1.49GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 26090
2023-09-12 21:42:11 +0000 UTC   3d583f40dc1f    1.5GB   docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 26586
2023-09-13 00:03:10 +0000 UTC   575d9879996f    64.1MB  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-8ec04662
2023-09-15 23:26:22 +0000 UTC   03771c524870    1.5GB   docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 26670
2023-09-18 17:32:24 +0000 UTC   02ef86b31632    1.5GB   docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 26740
2023-09-22 16:37:15 +0000 UTC   9b862e77d99f    1.5GB   docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 26856
2023-09-28 17:45:15 +0000 UTC   1d5e97c22793    1.51GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 27010
2023-09-29 20:37:32 +0000 UTC   4ab09e196ae9    2.3GB   122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/ucscgi/azul-pycharm  2022.3.3-3
2023-10-02 22:00:16 +0000 UTC   d2495ece8638    1.51GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 27126
2023-10-06 22:10:18 +0000 UTC   9094466fab28    1.51GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 27239
2023-10-09 23:02:51 +0000 UTC   6250e19f611e    2.33GB  122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/ucscgi/azul-pycharm  2022.3.3-4
2023-10-11 23:35:30 +0000 UTC   a9754e2c304b    712MB   122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/ucscgi/azul-elasticsearch    7.17.10-4
2023-10-18 02:17:45 +0000 UTC   8c805160ba57    1.51GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 27534
2023-10-20 16:54:24 +0000 UTC   9573457f4ce2    1.51GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 27609
2023-10-26 23:05:32 +0000 UTC   5b5b68c92574    1.66GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 27776
2023-10-27 17:13:15 +0000 UTC   81a969943bc1    1.66GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 27799
2023-10-28 02:46:58 +0000 UTC   541b5fa604d5    1.66GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 27811
2023-10-31 16:46:55 +0000 UTC   484fe096818c    1.66GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 27906
2023-11-07 04:05:05 +0000 UTC   21d53c42b0b3    1.66GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 28072
2023-11-08 02:25:56 +0000 UTC   506bba297b95    2.2GB   122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/ucscgi/azul-pycharm  2023.2.3-5
2023-11-18 07:55:38 +0000 UTC   3d233bd976d5    685MB   122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/ucscgi/azul-elasticsearch    7.17.15-5
2023-11-20 04:19:27 +0000 UTC   b9a38e6e7b3d    350MB   docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/runner   <none>
2023-11-26 20:41:13 +0000 UTC   4ea739ce84ee    2.05GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 28593

[ ] Security design review completed; the Resolution of this issue does not …
- [ ] … affect authentication; for example:
- OAuth 2.0 with the application (API or Swagger UI)
- Authentication of developers with Google Cloud APIs
- Authentication of developers with AWS APIs
- Authentication with a GitLab instance in the system
- Password and 2FA authentication with GitHub
- API access token authentication with GitHub
- Authentication with
- [ ] … affect the permissions of internal users like access to
- Cloud resources on AWS and GCP
- GitLab repositories, projects and groups, administration
- an EC2 instance via SSH
- GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
- [ ] … affect the permissions of external users like access to
- TDR snapshots
- [ ] … affect permissions of service or bot accounts
- Cloud resources on AWS and GCP
- [ ] … affect audit logging in the system, like
- adding, removing or changing a log message that represents an auditable event
- changing the routing of log messages through the system
- [ ] … affect monitoring of the system
- [ ] … introduce a new software dependency like
- Python packages on PYPI
- Command-line utilities
- Docker images
- Terraform providers
- [ ] … add an interface that exposes sensitive or confidential data at the security boundary
- [ ] … affect the encryption of data at rest
- [ ] … require persistence of sensitive or confidential data that might require encryption at rest
- [ ] … require unencrypted transmission of data within the security boundary
- [ ] … affect the network security layer; for example by
- modifying, adding or removing firewall rules
- modifying, adding or removing security groups
- changing or adding a port a service, proxy or load balancer listens on
[ ] Documentation on any unchecked boxes is provided in comments below

As for the GitLab registry, there are a few administrative actions to be taken:

1) On older GitLab instances, the cleanup needs to be enabled globally 2) On all instances, a clean-up policy needs to be set up for each project (90 days is the longest possible retention) 3) The clean-up policy only deletes tags, the resulting dangling images will need to be removed using garbage collection which we'll need to add as a timer unit to systemd

Only the system administrator can perform 1 and 2, 3 can be done by anyone, and requires a PR.

Our weekly purge job should actually have removed many of the images listed above. Spike to determine why that's not working.

Our weekly purge job should actually have removed many of the images listed above. Spike to determine why that's not working.

Typo in --filter option of prune images command

https://docs.docker.com/engine/reference/commandline/image_prune/

The until filter can be Unix timestamps, date formatted timestamps, or Go duration strings (e.g. 10m, 1h30m) computed relative to the daemon machine’s time. Supported formats for date formatted time stamps include RFC3339Nano, RFC3339, 2006-01-02T15:04:05, 2006-01-02T15:04:05.999999999, 2006-01-02Z07:00, and 2006-01-02.

Index: terraform/gitlab/gitlab.tf.json.template.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/terraform/gitlab/gitlab.tf.json.template.py b/terraform/gitlab/gitlab.tf.json.template.py
--- a/terraform/gitlab/gitlab.tf.json.template.py   (revision 4fedfaa30ef3105122875bd3af97f01e18c7bea8)
+++ b/terraform/gitlab/gitlab.tf.json.template.py   (date 1701304390761)
@@ -1836,7 +1836,7 @@
                                     'prune',  # … to delete, …
                                     '--force',  # … without prompting for confirmation, …
                                     '--all',  # … all images …
-                                    f'--filter "until={30 * 24}"',  # … except those from more recent builds.
+                                    f'--filter "until={30 * 24}h"',  # … except those from more recent builds.
                                     #
                                     # If we deleted more recent images, we
                                     # would risk failing the requirements

Tested manually on gitlab dev however my shell timed out before command completed.

[ec2-user@ip-172-71-0-215 log]$ sudo docker exec -it gitlab-dind /bin/sh
/ #
/ # docker image prune --all --filter "until=720"
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
/ #
/ # docker image prune --all --filter "until=720h"
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Connection to ssh.gitlab.dev.singlecell.gi.ucsc.edu closed by remote host.
Connection to ssh.gitlab.dev.singlecell.gi.ucsc.edu closed.

Upon reconnection I was able to confirm a bunch of images had been pruned

[ec2-user@ip-172-71-0-215 ~]$ sudo docker exec -it gitlab-dind docker image ls --format=json | jq -r '[.CreatedAt, .ID, .Size, .Repository, .Tag] | join("\t")' | sort -u -k 1,1
2020-10-09 18:00:33 +0000 UTC   d1226e1554f8    60.4MB  gitlab/gitlab-runner-helper x86_64-264446b2
2020-12-29 02:47:00 +0000 UTC   97396fa3d959    1.42GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  999
2021-03-25 22:21:58 +0000 UTC   a9275369af8c    70.2MB  gitlab/gitlab-runner-helper x86_64-54944146
2021-07-16 19:12:58 +0000 UTC   ae0f95f06fc6    1.38GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  6638
2022-01-24 19:33:44 +0000 UTC   40d5e650e7cc    66.9MB  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-98daeee0
2022-01-31 18:37:35 +0000 UTC   fb1b155d1740    1.34GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  12319
2022-03-16 20:15:09 +0000 UTC   b083053d75df    1.39GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  12425
2022-07-24 23:03:12 +0000 UTC   c526698c011d    67MB    registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper   x86_64-76984217
2022-07-25 21:19:00 +0000 UTC   f1148a65c439    1.37GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev  16191
2023-10-31 16:46:55 +0000 UTC   484fe096818c    1.66GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 27906
2023-11-07 04:05:05 +0000 UTC   21d53c42b0b3    1.66GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 28072
2023-11-08 02:25:56 +0000 UTC   506bba297b95    2.2GB   122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/ucscgi/azul-pycharm  2023.2.3-5
2023-11-18 07:55:38 +0000 UTC   3d233bd976d5    685MB   122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/ucscgi/azul-elasticsearch    7.17.15-5
2023-11-20 04:19:27 +0000 UTC   b9a38e6e7b3d    350MB   docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/runner   <none>
2023-11-26 20:41:13 +0000 UTC   4ea739ce84ee    2.05GB  docker.gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/dev-deps 28593
[ec2-user@ip-172-71-0-215 ~]$

data disk usage for GitLab dev now at ~58.2%

Screen Shot 2023-11-29 at 4 55 35 PM

[ec2-user@ip-172-71-0-215 ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           7.8G  652K  7.8G   1% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/nvme0n1p1   20G  7.5G   13G  38% /
tmpfs           7.8G     0  7.8G   0% /tmp
/dev/nvme1n1    197G  109G   79G  59% /mnt/gitlab
tmpfs           1.6G     0  1.6G   0% /run/user/1000

@hannes-ucsc: "Assignee to implement the fix found by @dsotirho-ucsc in a PR but also increase the image retention from 30 days to 90. The PR checklist should contain items for the administrative tasks necessary to set up the registry cleanup policy for GitLab."

Assignee to also run the manual purge on anvildev ASAP.

Assignee to also run the manual purge on anvildev ASAP.

[ec2-user@ip-172-73-0-46 ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           7.8G  648K  7.8G   1% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/nvme0n1p1   20G  7.7G   13G  39% /
tmpfs           7.8G     0  7.8G   0% /tmp
/dev/nvme1n1    148G  106G   35G  76% /mnt/gitlab
tmpfs           1.6G     0  1.6G   0% /run/user/1000
[ec2-user@ip-172-73-0-46 ~]$
[ec2-user@ip-172-73-0-46 ~]$
[ec2-user@ip-172-73-0-46 ~]$ sudo docker exec -it gitlab-dind /bin/sh
/ #
/ # docker image prune --all --filter "until=2160h"
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
…
Total reclaimed space: 24.49GB
/ #
/ # exit
[ec2-user@ip-172-73-0-46 ~]$
[ec2-user@ip-172-73-0-46 ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           7.8G  648K  7.8G   1% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/nvme0n1p1   20G  7.7G   13G  39% /
tmpfs           7.8G     0  7.8G   0% /tmp
/dev/nvme1n1    148G   81G   60G  58% /mnt/gitlab
tmpfs           1.6G     0  1.6G   0% /run/user/1000

Assignee to also run the manual purge on anvilprod ASAP.

Assignee to also run the manual purge on anvilprod ASAP.

[ec2-user@ip-172-74-0-28 ~]$
[ec2-user@ip-172-74-0-28 ~]$ df -h | grep gitlab
/dev/nvme1n1    148G  108G   33G  77% /mnt/gitlab
[ec2-user@ip-172-74-0-28 ~]$
[ec2-user@ip-172-74-0-28 ~]$ sudo docker exec -it gitlab-dind /bin/sh
/ #
/ # docker image ls | wc -l
6463
/ #
/ # docker image prune --all --filter "until=2160h"
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
…
Total reclaimed space: 31.98GB
/ #
/ # docker image ls | wc -l
2179
/ #
/ # exit
[ec2-user@ip-172-74-0-28 ~]$
[ec2-user@ip-172-74-0-28 ~]$
[ec2-user@ip-172-74-0-28 ~]$ df -h | grep gitlab
/dev/nvme1n1    148G   75G   66G  54% /mnt/gitlab
[ec2-user@ip-172-74-0-28 ~]$

For demo, attempt to reproduce on every GitLab instance. Show evidence that timer units ran successfully.

Created #5860 to address left-over stale images
We noticed that we don't see any output from the unit that runs gitlab-ctl. Apparently output from docker exec against a running container does not get captured by the /etc/docker/daemon.json so we should remove the StandardOutput and StandardError options from any unit that uses docker exec.
We weren't able to demo this completely, hence ticket is being moved back to In progress.
We also had to disable the registry clean-up https://github.com/DataBiosphere/azul/issues/5822#issuecomment-1876139942

DataBiosphere / azul

Many stale images in gitlab-dind and GitLab registry #5728

GitLab container registry

gitlab-dind