search

goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
23.13k stars 4.66k forks source link

Tag retention: Unexpected cleanup for a tag with incorrect `PushedTime` and `PulledTime` values #11837

Closed carletes closed 3 years ago

carletes commented 4 years ago

Using Harbor version v1.10.0-6b84b62f, I have several retention rules defined in a project, one of which is the following:

I got a tag incorrectly deleted shortly after it was pushed --- probably because its PushedTime and PulledTime were recorded incorrectly. The tag in question is develop-4e67f38. This is the log from the retention process which deleted it:

2020-05-05T19:00:06Z [INFO] [/pkg/retention/job.go:77]: Run retention process.
 Repository: webdev/webdev-django 
 Rule Algorithm: or 
 Dry Run: false
2020-05-05T19:00:08Z [INFO] [/pkg/retention/job.go:92]: Load 26 candidates from repository webdev/webdev-django
2020-05-05T19:00:08Z [INFO] [/pkg/retention/job.go:191]: 
|                                 Digest                                  |               Tag               | Kind  | Labels |     PushedTime      |     PulledTime      |     CreatedTime     | Retention |
|-------------------------------------------------------------------------|---------------------------------|-------|--------|---------------------|---------------------|---------------------|-----------|
| sha256:2d985010379e6e5793ac362230e4c9ffac2c0c05c8543f85109be557038a7891 | 2020-03-04-001                  | image |        | 2020/03/04 13:44:28 | 0001/01/01 00:00:00 | 2020/03/02 15:14:45 | RETAIN    |
| sha256:1d258a2ddc56dfcd210c59b8570da29c56f5f168b37720de71c803c3cb6348f3 | 2020-03-25-001                  | image |        | 0001/01/01 00:00:00 | 0001/01/01 00:00:00 | 2020/03/25 16:04:54 | RETAIN    |
| sha256:64b225df02587760ac80be61dca0e5f45514c098e9777aeb058bd3dac1b76864 | 2020-04-08-001                  | image |        | 2020/04/08 13:40:58 | 0001/01/01 00:00:00 | 2020/04/07 10:49:30 | RETAIN    |
| sha256:3b2f2d8b232d4db0a5e59c8f48158bf07951d914bbf7ee429a95300cec1bdacf | develop                         | image |        | 2020/05/05 16:19:21 | 2020/05/05 18:35:42 | 2020/05/05 16:14:10 | RETAIN    |
| sha256:273912c250fb867a3b73d0c1ad1fa2f37461afe5e1877baaa479a483081c8420 | develop-2db26e4                 | image |        | 2020/05/05 13:33:30 | 2020/05/05 13:35:10 | 2020/05/05 12:46:38 | RETAIN    |
| sha256:e126360a004a7a8bdc9b641f494508397e6edf5c98d10d935c3e100794bd84eb | develop-4e67f38                 | image |        | 0001/01/01 00:00:00 | 0001/01/01 00:00:00 | 2020/05/05 16:09:47 | DEL       |
| sha256:8ab58fdf1d7f441f4f6daa8257b1349ff521f76053a8d208c848dc2086f9290f | develop-5c12d32                 | image |        | 2020/05/05 11:56:00 | 2020/05/05 11:57:50 | 2020/05/05 10:21:51 | RETAIN    |
| sha256:d035c70c83d98528d443f4046045a821643de5a3c1d4e36deed7fef5a699fa81 | develop-9d12cf3                 | image |        | 2020/05/05 14:13:21 | 2020/05/05 14:14:53 | 2020/05/05 10:54:42 | RETAIN    |
| sha256:5ef62dc66a92c02fb6006eebe3dc0815995298194e8df6f84253b5dfc937e4d1 | develop-b193195                 | image |        | 2020/05/05 14:22:35 | 2020/05/05 14:23:54 | 2020/05/05 14:13:51 | RETAIN    |
| sha256:b1e61456441acbbfa7d8948da0aa8f770ab7924084708ab1a2357fa2b2d27f33 | develop-e5df104                 | image |        | 2020/05/05 16:03:50 | 2020/05/05 16:10:01 | 2020/05/05 16:02:15 | RETAIN    |
| sha256:ae53dd253e5c3c66b692a59c8485f035b4b2dcbc2584b4c890c7507ee1b6d753 | develop-e779651                 | image |        | 2020/05/01 08:56:23 | 0001/01/01 00:00:00 | 2020/04/28 18:47:07 | RETAIN    |
| sha256:64b225df02587760ac80be61dca0e5f45514c098e9777aeb058bd3dac1b76864 | feature-efas_ericha_status_v2   | image |        | 2020/04/07 10:51:39 | 2020/04/07 11:04:16 | 2020/04/07 10:49:30 | RETAIN    |
| sha256:8b5cc4f0cfea2c88088709bc8075e74236b5a25f658d531f119ac66dc0b65f6e | master                          | image |        | 2020/05/05 14:26:36 | 0001/01/01 00:00:00 | 2020/04/28 18:47:00 | RETAIN    |
| sha256:73d1891c8d0effda23848b5083a35809ae758e5f3f4992fdd510ccc917d18858 | master-672efe5                  | image |        | 2020/04/27 16:43:16 | 0001/01/01 00:00:00 | 2020/04/27 15:08:06 | RETAIN    |
| sha256:ace47863481608d8815717fd580252ecb3cbf580eab6206655c13e7bdad09ab7 | master-8c01f99                  | image |        | 2020/04/27 15:18:33 | 0001/01/01 00:00:00 | 2020/04/27 15:12:43 | RETAIN    |
| sha256:ed0f1f96525fa4da6826d5cd8ff148c35fbbf34fd96b114c07d592e05bb6c9d4 | master-f85a7d7                  | image |        | 0001/01/01 00:00:00 | 0001/01/01 00:00:00 | 2020/04/24 08:57:11 | RETAIN    |
| sha256:6df8a182ff7fa4f0a4187ae2db6be9ba1991f310c2af5ca41b75b35c5e6f2d60 | wdqms-2020-03-16-001            | image |        | 0001/01/01 00:00:00 | 0001/01/01 00:00:00 | 2020/03/16 10:01:12 | RETAIN    |
| sha256:ae53dd253e5c3c66b692a59c8485f035b4b2dcbc2584b4c890c7507ee1b6d753 | wdqms-2020-04-30-001            | image |        | 2020/04/30 12:52:45 | 2020/05/04 16:10:37 | 2020/04/28 18:47:07 | RETAIN    |
| sha256:e90b9ddffd7bd9ec0bd68da19950409bdbc6d5523a823ac9f470080ee7759682 | webplots-2020-04-22-001         | image |        | 2020/04/27 10:58:50 | 0001/01/01 00:00:00 | 2020/04/27 10:47:50 | RETAIN    |
| sha256:ed0f1f96525fa4da6826d5cd8ff148c35fbbf34fd96b114c07d592e05bb6c9d4 | webplots-2020-04-22-001-f85a7d7 | image |        | 2020/04/24 08:58:46 | 0001/01/01 00:00:00 | 2020/04/24 08:57:11 | RETAIN    |
| sha256:ec1491aa91c9c746d69125f6ada38989a1b34fb505bb7e4e0c1deff314bd851a | webplots-2020-04-27-001         | image |        | 2020/04/27 15:37:20 | 0001/01/01 00:00:00 | 2020/04/27 15:26:17 | RETAIN    |
| sha256:ace47863481608d8815717fd580252ecb3cbf580eab6206655c13e7bdad09ab7 | webplots-2020-04-27-001-8c01f99 | image |        | 2020/04/27 15:18:36 | 0001/01/01 00:00:00 | 2020/04/27 15:12:43 | RETAIN    |
| sha256:26b569d94972a74ea6146833fe93318c7253a2486f6bdd567648278c54d9b8db | webplots-2020-04-27-002         | image |        | 2020/04/29 16:36:49 | 0001/01/01 00:00:00 | 2020/04/27 16:29:03 | RETAIN    |
| sha256:73d1891c8d0effda23848b5083a35809ae758e5f3f4992fdd510ccc917d18858 | webplots-2020-04-27-002-672efe5 | image |        | 2020/04/27 16:43:18 | 0001/01/01 00:00:00 | 2020/04/27 15:08:06 | RETAIN    |
| sha256:f0b560275240644b3b3cb26d8999ba65416d1368dc6af2666c98605bd80e85cb | webplots-2020-04-29-002         | image |        | 2020/04/27 17:02:14 | 0001/01/01 00:00:00 | 2020/04/27 15:41:44 | RETAIN    |
| sha256:8b5cc4f0cfea2c88088709bc8075e74236b5a25f658d531f119ac66dc0b65f6e | webplots-2020-04-30-002         | image |        | 2020/05/05 14:26:36 | 0001/01/01 00:00:00 | 2020/04/28 18:47:00 | RETAIN    |

Note that there are other tags with incorrect values for PushedTime and PulledTime, too. Luckily they are covered by other retention rules which do not depend on the pulled/pushed time.

Is this a known issue?

bitsf commented 4 years ago

We have fixed some similar issue introduced in harbor 2.0, but not sure why pushedtime is empty in 1.10.x yet. Do you have any special operation that how tag "2020-03-25-001" create ?

xaleeks commented 4 years ago

seems this is not an issue with tag retention engine, but how pushtimes are calculated. @carletes Can you elaborate on how these are pushed to harbor?

carletes commented 4 years ago

Can you elaborate on how these are pushed to harbor?

Sure! We push images from our CI build plans after tests run successfully, by doing something like this. We first push each image, using the image tag matching the Git repository branch it was built from:

$ docker push xxx/webdev/webdev-django:develop

Then we create a new tag that uniquely identifies the image we're pushing:

$ docker tag xxx/webdev/webdev-django:develop xxx/webdev/webdev-django:develop-XXYYZZ

And then we push that tag:

$ docker push xxx/webdev/webdev-django:develop-XXYYZZ

The aim of this is to have xxx/webdev/webdev-django:<branch-name> always pointing to the image built from the Git HEAD of the given branch, and xxx/webdev/webdev-django:<branch-name>-<some-id> to the image we tested in the CI plan (so that we may then deploy to our Kubernetes cluster exactly what we tested).

The <some-id> bit is the first seven characters of docker inspect xxx/webdev/webdev-django:<branch-name> --format '{{ .ID }}'.

We run several build plans whenever a new Git commit is detected. It might be the case that several build plans running on different CI hosts end up doing something like this:

# Plan A on CI host `foo`
$ docker push xxx/webdev/webdev-django:develop

      # Plan B on CI host `bar`
      $ docker push xxx/webdev/webdev-django:develop

# Plan A on CI host `foo`
$ docker tag xxx/webdev/webdev-django:develop xxx/webdev/webdev-django:develop-1234567

      # Plan B on CI host `bar`
      $ docker tag xxx/webdev/webdev-django:develop xxx/webdev/webdev-django:develop-7654321

# Plan A on CI host `foo`
$ docker push xxx/webdev/webdev-django:develop-1234567

      # Plan B on CI host `bar`
      $ docker push xxx/webdev/webdev-django:develop-7654321

We're using Docker version 19.03.4 on some CI hosts, and version 19.03.7 on some others.

bitsf commented 3 years ago

Not reproduced that PushedTime be '0001/01/01' with harbor.1.10.0 as the case. And 1.10.x is not actively maintained, please try newer harbor release.