search

goharbor / harbor

An open source trusted cloud native registry project that stores, signs, and scans content.
https://goharbor.io
Apache License 2.0
23.91k stars 4.74k forks source link

Regex support for retention rules #12877

Open jabbors opened 4 years ago

jabbors commented 4 years ago

Is your feature request related to a problem? Please describe. I find it impossible to define rules that would allow me to keep all "tagged" images while only retaining the e.g. last 10 images originating from automated builds.

Background to our problem.

The code for our applications lives in a monorepo. Every merge to master triggers a job that detects which applications has changed, builds a new image for that application (with a tag generated using git describe --tags --always --dirty='-dev' --match="$app-v[0-9]*" | sed "s/$app-//") and uploads the image. For release, we create a git tag on a commit, which triggers a build only building a specific image ~with the~ (tagged with the provided tag) and uploads it.

Thus for an arbitrary application we might end up with tags like these:

I've been experimenting with rules and combining them but haven't come across a working set of rules that would only delete images with generated tags.

Describe the solution you'd like I'd like to be able to defined a regex specifying to keep all images with tags matching the pattern eg. ^v\d+\.\d+\.\d+$ combined with another rule specifying to retain the x most recently pushed images matching another pattern eg. -\d+-g\w+

bitsf commented 4 years ago

Hi, I'm not quite understand your requirement, can you give a more detailed example, like which to retain and which to delete ?

BTW, if two images have same digest, they will both be retained now, if anyone match the retain rule.

We don't support regex as harbor is using doublestar library to match the pattern, you can try use v*.0 to match v1.1.0, or may try use tags excluding v*-*

jabbors commented 4 years ago

Below are the tags pushed for an application order by date in ascending order.

Note 1: Other application may have more or less tags that this application and there may be all kinds of tags with a semantic version number, eg. v2.0.0 and v3.0.1-41-g32a3cb4

Note 2: Tag latest has the same digest as the most recently used image from with a generated tag, in this case v1.19.0-65-g8e18722c

Note 3: Tag latest will never have the same digest as a tag trigger by a manual release of an application. E.g. if the next version to be pushed has tag v1.20.0, tag latest will still have the same digest as v1.19.0-65-g8e18722c

v1.17.0
v1.17.0-41-g409f00e2
v1.17.0-48-g56b2d40f
v1.17.0-49-g3c7433d8
v1.17.0-51-g353b0670
v1.17.0-52-g35a20c48
v1.17.0-54-g7a7ec24a
v1.17.0-55-gc19617d0
v1.17.0-57-gddd19f8b
v1.17.0-58-g2e4ca8ad
v1.17.0-70-g60110a83
v1.17.0-76-ga0f40f1d
v1.17.0-77-g93d624b2
v1.17.0-81-g9f0c66c7
v1.17.0-82-g1409cc47
v1.17.0-83-g5f7437ad
v1.17.0-84-g618602a9
v1.18.0
v1.18.0-1-g2d2a2759
v1.18.0-3-gc6f89b43
v1.18.0-5-g77b585d6
v1.18.0-6-g3a4bc2ac
v1.18.0-7-gfe2a9a2d
v1.18.0-8-g66001fa2
v1.18.0-20-g98ae314f
v1.18.0-22-geac7c8e2
v1.18.0-23-gce0ca64d
v1.18.0-24-g0fc0bf5b
v1.18.0-25-g85f7eb27
v1.18.0-26-geaaa0d26
v1.18.0-27-gcf83511c
v1.18.0-28-gcb564761
v1.18.0-29-ge5781756
v1.18.0-31-g46baa66b
v1.18.0-33-gfdb4705e
v1.18.0-36-gb1e22a74
v1.19.0
v1.19.0-2-g38a0a27d
v1.19.0-3-g9127f372
v1.19.0-4-ge35c5854
v1.19.0-6-gb917c385
v1.19.0-7-g73ccc9de
v1.19.0-9-gf3bda44c
v1.19.0-11-gcd008a0d
v1.19.0-12-ge10e835f
v1.19.0-14-gb9bfcec5
v1.19.0-15-gb5ef674d
v1.19.0-17-g2fa00ae1
v1.19.0-18-gd62f0efe
v1.19.0-20-g218d5e78
v1.19.0-21-g4479cf90
v1.19.0-22-g38da7f46
v1.19.0-23-g6ad86e76
v1.19.0-24-ge6371a91
v1.19.0-25-gaea2bac6
v1.19.0-26-g5fdac10e
v1.19.0-27-ga031ba53
v1.19.0-29-g3207df93
v1.19.0-30-gfda28737
v1.19.0-31-g69a18740
v1.19.0-32-gb450bf6a
v1.19.0-33-g66d00d5a
v1.19.0-34-gec959976
v1.19.0-35-g862dd518
v1.19.0-44-gb207ce78
v1.19.0-45-gf5da7387
v1.19.0-46-gc4138a02
v1.19.0-48-g28f091cc
v1.19.0-50-g26f995e9
v1.19.0-53-g6c2d5ba4
v1.19.0-55-ge7e744a7
v1.19.0-57-gb4f605b6
v1.19.0-59-g47a8d9aa
v1.19.0-61-g954e1edd
v1.19.0-63-g8b03e2e9
v1.19.0-64-g9b57c65b
v1.19.0-65-g8e18722c
latest

I'd like to define rules that when applied, results in the below list of tags retained.

v1.17.0
v1.18.0
v1.19.0
v1.19.0-59-g47a8d9aa
v1.19.0-61-g954e1edd
v1.19.0-63-g8b03e2e9
v1.19.0-64-g9b57c65b
v1.19.0-65-g8e18722c
latest
jabbors commented 4 years ago

may try use tags excluding v-

Thanks for the clarification of which library is used. I'll do some experiments with your suggestion and read up more on the doublestar library.

jabbors commented 4 years ago

Finally managed to produce rules that deletes according to our needs. The two rules are:

Initially, I had in mind that all images tagged without a semantic version number would be deleted, which these rules don't do, but I changed my mind as we happen to have a special cases when a developer manually builds and uploads images that are used for debugging or developing in the cloud, and I don't want these images to get deleted automatically.

jabbors commented 4 years ago

I'll let owners decide if this issue shall be closed or not as others may tag images differently and are in need to regular expressions.

thoro commented 4 years ago

I also believe the usage of doublestar (a directory matching library) seems inadequate for matching on labels.

Could someone explain the reasoning behind it?

A simple solution would be to allow regex or doublestar, if the pattern starts and end with / regex is used, otherwise doublestar.

viceice commented 3 years ago

I also like to have a regex matcher, as i'm using the following tags:

v2.0.0
v2.0.0-beta.2.12
v2.0.0-beta.2
v2.0.0-beta.1.1234
v2.0.0-beta.1

I'd like to remove all /^v\d+\.\d+\.\d+-.+/ but retain latest 25

Wykiki commented 3 years ago

Hello,

I also would like to have a regex matcher :

For me, if Harbor team do not want to provide regex, it would be really nice to have something on Harbor that let us try the rules on a predefined set of repo/tags, with some common examples.

bitsf commented 3 years ago

@steven-zou do you consider add regex

steven-zou commented 2 years ago

@steven-zou do you consider add regex

To keep the consistent pattern in the whole harbor, if support regx in retention, other places also need to to be enhanced. That would be a big work.

@xaleeks if you have further comments here.

HammerNL89 commented 2 years ago

We would also like to see regex support for retention policies. The current doublestar implementation is just to limited to do proper valid semver matching on tags, you match a lot of non valid semver tags as well..

Would be nice if we could just use the 'official semver regex' e.g: https://regex101.com/r/Ly7O1x/3/

philraj commented 2 years ago

@steven-zou do you consider add regex

To keep the consistent pattern in the whole harbor, if support regx in retention, other places also need to to be enhanced. That would be a big work.

@xaleeks if you have further comments here.

I'm not sure it really needs to be implemented everywhere all at once. It would be the best for Harbor to present a unified feature set for pattern matching, but it would already be a big enhancement if tag retention alone could use regular expressions. Right now it's quite difficult to set up a reasonable set of rules when you have complex needs. It would make a lot of people happy even if it was only available for tag retention, that's for sure. Maybe a gradual enhancement makes more sense?

github-actions[bot] commented 2 years ago

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

pkalemba commented 1 year ago

any plans to implement this?

mhoyer commented 11 months ago

In addition, having regex support for the immutable tags policies would have the same benefits of enabling more sophisticated rules. E.g. here: we want to restrict moving (mutuable) tags only for semver related major(.minor) variants:

sha256:0000000 
 → 1.0.0  < immutable
 → 1.0    < mutable
 → 1      < mutable

With release of the hotfix of 1.0.1 image it should look like:

sha256:0000000 
 → 1.0.0  < immutable
sha256:1111111
 → 1.0.1  < immutable
 → 1.0    < mutable
 → 1      < mutable
Weissger commented 5 months ago

If you just need to match SemVer releases you can try to use something like:

v{[0-9],[0-9][0-9],[0-9][0-9][0-9],[0-9][0-9][0-9][0-9]}.{[0-9],[0-9][0-9],[0-9][0-9][0-9],[0-9][0-9][0-9][0-9]}.{[0-9],[0-9][0-9],[0-9][0-9][0-9],[0-9][0-9][0-9][0-9]}

It's ugly and verbose but it should support version numbers to the max number of v9999.9999.9999. At least the underlying go library matches versions like v1.3.32 successfully with this pattern.

mtt0 commented 1 month ago

Any updates? QAQ