Tags filtering / skip tags that already exist in the target registry

[yannh]

Hello :wave:

I have implemented some minimal logic for tag filtering: https://github.com/yannh/dregsy#tags-filtering , support for wildcards, comparison operators, and exclude tag ranges.

I have also implemented an option called "skipExistingTags" - dregsy will skip tags that already exist in the target registry. Since it might cause issues with tags that are not immutable such as 'latest' it is deactivated by default.

To easily implement both these features, I have removed the docker relay - see https://github.com/xelalexv/dregsy/issues/21

Do you see a path to merge this back here? I am happy to work on clean PRs. But I am not sure how to implement this efficiently using the docker relay :/

Thanks! :+1:

[yannh]

:wave: I actually realised that skopeo already supports a sync subcommand that does most of what I m trying to implement here... the yaml format it supports is not fantastic (does not allow to configure the target repository name, for example) but it might be good enough.

[xelalexv]

Yes, I was actually thinking to leverage Skopeo's sync command for solving issue #16. I also think that the YAML format for sync isn't quite stellar, but that would probably not be an issue since we'd be generating it on the fly based on the config of the dregsy sync task. But I haven't looked into it in detail yet.

[mkhodave]

@xelalexv : can you please give some examples for tag filtering for skopeo sync command or any plan for add enhancement in dregsy tags field to allow tag filtering ?

[xelalexv]

Tag filtering is not yet supported. It's next on my TODO list, but I can't really give you an estimate when that is going to happen.

[sathieu]

@mkhodave If you need tag filtering and your target is a gitlab instance, take a look at Gitlabracadabra.

I plan to implement other destinations in the future.

[mkhodave]

@sathieu : my target is google artifact registry , does tag filtering mentioned in Gitlabracadabra also can be used in my use case ?

[sathieu]

@mkhodave No (at least not yet).

[mkhodave]

@xelalexv & @sathieu : Thanks for quick reply, really helpful for planning task around Google artifact registry

[xelalexv]

I looked into this last night, and created an initial implementation for semver-based tags filtering. So you can do something like this now:

    tags:
    - "semver: >=1.31.0 <1.31.9"
    - "1.29.4"
    - "latest"

This would sync all tags describing versions equal to or larger than 1.31.0, but lower than 1.31.9. In addition, the verbatim tags 1.29.4 and latest would also be synced. Note that tags of an image need to conform to the semver specification 2.0.0 in order to be considered during filtering. I'm using the blang/semver lib for this, so see their page or the GoDoc for more info on how to write semver filter expressions. You can add multiple semver:... expressions under tags (they get ORed). Next up is regex, so that you can also express a filter via regular expressions.

I pushed a Docker image with tag issue22, so if you get a chance to try this out, feedback would be welcome :wink:

[mkhodave]

@xelalexv : I am trying as following but getting error & tag 3.10.0 getting synced

 mappings:
      - from: library/alpine
        to: <project-id>/upstream/alpine
        tags:
          - "semver: >=3.11.0 <3.11.11"
          - "3.10.0"

time="2021-05-11T13:13:04Z" level=info msg="dregsy 0.3.6"
time="2021-05-11T13:13:05Z" level=info msg="skopeo version 1.2.1 commit: 1b6af3b15975f3142462b6affa85af708feb657f\n"
time="2021-05-11T13:13:05Z" level=info msg="relay ready" relay=skopeo
time="2021-05-11T13:13:05Z" level=info msg="syncing task" source=registry.hub.docker.com target=europe-west1-docker.pkg.dev task=index
time="2021-05-11T13:13:05Z" level=info msg=mapping from=/library/alpine to=/<project-id>/upstream/alpine
time="2021-05-11T13:13:05Z" level=info msg="syncing tag" tag="semver: >=3.11.0 <3.11.11"
time="2021-05-11T13:13:05Z" level=debug msg="time=\"2021-05-11T13:13:05Z\" level=fatal msg=\"Invalid source name docker://registry.hub.docker.com/library/alpine:semver: >=3.11.0 <3.11.11: invalid reference format\""
time="2021-05-11T13:13:05Z" level=error msg="exit status 1"
time="2021-05-11T13:13:05Z" level=info msg="syncing tag" tag=3.10.0
time="2021-05-11T13:13:10Z" level=debug msg="Getting image source signatures"
time="2021-05-11T13:13:19Z" level=debug msg="Copying blob sha256:921b31ab772b38172fd9f942a40fae6db24decbd6706f67836260d47a72baab5"
time="2021-05-11T13:13:44Z" level=debug msg="Copying config sha256:4d90542f0623c71f1f9c11be3da23167174ac9d93731cf91912922e916bab02c"
time="2021-05-11T13:13:55Z" level=debug msg="Writing manifest to image destination"
time="2021-05-11T13:13:57Z" level=debug msg="Storing signatures"
time="2021-05-11T13:13:57Z" level=error msg="errors during sync"
time="2021-05-11T13:13:57Z" level=debug msg="stopping tasks"
time="2021-05-11T13:13:57Z" level=debug msg="task exited" task=index
time="2021-05-11T13:13:57Z" level=debug msg="exit main"
time="2021-05-11T13:13:57Z" level=error msg="one or more tasks had errors, please see log for details"

[mkhodave]

I pulled fresh image from dregsy & trying

[xelalexv]

You need to use the dregsy image tagged issue22.

[xelalexv]

And here goes regex support for tag filtering. It uses standard Golang regular expressions, same as image mapping. Expanding on above example, you can now do:

tags:
    - "semver: >=1.31.0 <1.31.9"
    - "regex: 1\.26\.[0-9]-(glibc|uclibc|musl)"
    - "1.29.4"
    - "latest"

This would additionally sync any 1.26.x image with suffix -glibc, -uclibc, or -musl. I updated the issue22 image over at DockerHub.

[mkhodave]

I have used issue22 image tag & it worked , I will give try for regex option as well & let you know , Thanks for adding this , it really helps

[xelalexv]

done with PR #54