xelalexv / dregsy

Keep container registries in sync
https://buymeacoffee.com/xelalex
Apache License 2.0
262 stars 52 forks source link

Tags filtering / skip tags that already exist in the target registry #22

Closed yannh closed 3 years ago

yannh commented 4 years ago

Hello :wave:

I have implemented some minimal logic for tag filtering: https://github.com/yannh/dregsy#tags-filtering , support for wildcards, comparison operators, and exclude tag ranges.

I have also implemented an option called "skipExistingTags" - dregsy will skip tags that already exist in the target registry. Since it might cause issues with tags that are not immutable such as 'latest' it is deactivated by default.

To easily implement both these features, I have removed the docker relay - see https://github.com/xelalexv/dregsy/issues/21

Do you see a path to merge this back here? I am happy to work on clean PRs. But I am not sure how to implement this efficiently using the docker relay :/

Thanks! :+1:

yannh commented 4 years ago

:wave: I actually realised that skopeo already supports a sync subcommand that does most of what I m trying to implement here... the yaml format it supports is not fantastic (does not allow to configure the target repository name, for example) but it might be good enough.

xelalexv commented 4 years ago

Yes, I was actually thinking to leverage Skopeo's sync command for solving issue #16. I also think that the YAML format for sync isn't quite stellar, but that would probably not be an issue since we'd be generating it on the fly based on the config of the dregsy sync task. But I haven't looked into it in detail yet.

mkhodave commented 3 years ago

@xelalexv : can you please give some examples for tag filtering for skopeo sync command or any plan for add enhancement in dregsy tags field to allow tag filtering ?

xelalexv commented 3 years ago

Tag filtering is not yet supported. It's next on my TODO list, but I can't really give you an estimate when that is going to happen.

sathieu commented 3 years ago

@mkhodave If you need tag filtering and your target is a gitlab instance, take a look at Gitlabracadabra.

I plan to implement other destinations in the future.

mkhodave commented 3 years ago

@sathieu : my target is google artifact registry , does tag filtering mentioned in Gitlabracadabra also can be used in my use case ?

sathieu commented 3 years ago

@mkhodave No (at least not yet).

mkhodave commented 3 years ago

@xelalexv & @sathieu : Thanks for quick reply, really helpful for planning task around Google artifact registry

xelalexv commented 3 years ago

I looked into this last night, and created an initial implementation for semver-based tags filtering. So you can do something like this now:

    tags:
    - "semver: >=1.31.0 <1.31.9"
    - "1.29.4"
    - "latest"

This would sync all tags describing versions equal to or larger than 1.31.0, but lower than 1.31.9. In addition, the verbatim tags 1.29.4 and latest would also be synced. Note that tags of an image need to conform to the semver specification 2.0.0 in order to be considered during filtering. I'm using the blang/semver lib for this, so see their page or the GoDoc for more info on how to write semver filter expressions. You can add multiple semver:... expressions under tags (they get ORed). Next up is regex, so that you can also express a filter via regular expressions.

I pushed a Docker image with tag issue22, so if you get a chance to try this out, feedback would be welcome :wink:

mkhodave commented 3 years ago

@xelalexv : I am trying as following but getting error & tag 3.10.0 getting synced

 mappings:
      - from: library/alpine
        to: <project-id>/upstream/alpine
        tags:
          - "semver: >=3.11.0 <3.11.11"
          - "3.10.0"
time="2021-05-11T13:13:04Z" level=info msg="dregsy 0.3.6"
time="2021-05-11T13:13:05Z" level=info msg="skopeo version 1.2.1 commit: 1b6af3b15975f3142462b6affa85af708feb657f\n"
time="2021-05-11T13:13:05Z" level=info msg="relay ready" relay=skopeo
time="2021-05-11T13:13:05Z" level=info msg="syncing task" source=registry.hub.docker.com target=europe-west1-docker.pkg.dev task=index
time="2021-05-11T13:13:05Z" level=info msg=mapping from=/library/alpine to=/<project-id>/upstream/alpine
time="2021-05-11T13:13:05Z" level=info msg="syncing tag" tag="semver: >=3.11.0 <3.11.11"
time="2021-05-11T13:13:05Z" level=debug msg="time=\"2021-05-11T13:13:05Z\" level=fatal msg=\"Invalid source name docker://registry.hub.docker.com/library/alpine:semver: >=3.11.0 <3.11.11: invalid reference format\""
time="2021-05-11T13:13:05Z" level=error msg="exit status 1"
time="2021-05-11T13:13:05Z" level=info msg="syncing tag" tag=3.10.0
time="2021-05-11T13:13:10Z" level=debug msg="Getting image source signatures"
time="2021-05-11T13:13:19Z" level=debug msg="Copying blob sha256:921b31ab772b38172fd9f942a40fae6db24decbd6706f67836260d47a72baab5"
time="2021-05-11T13:13:44Z" level=debug msg="Copying config sha256:4d90542f0623c71f1f9c11be3da23167174ac9d93731cf91912922e916bab02c"
time="2021-05-11T13:13:55Z" level=debug msg="Writing manifest to image destination"
time="2021-05-11T13:13:57Z" level=debug msg="Storing signatures"
time="2021-05-11T13:13:57Z" level=error msg="errors during sync"
time="2021-05-11T13:13:57Z" level=debug msg="stopping tasks"
time="2021-05-11T13:13:57Z" level=debug msg="task exited" task=index
time="2021-05-11T13:13:57Z" level=debug msg="exit main"
time="2021-05-11T13:13:57Z" level=error msg="one or more tasks had errors, please see log for details"
mkhodave commented 3 years ago

I pulled fresh image from dregsy & trying

xelalexv commented 3 years ago

You need to use the dregsy image tagged issue22.

xelalexv commented 3 years ago

And here goes regex support for tag filtering. It uses standard Golang regular expressions, same as image mapping. Expanding on above example, you can now do:

tags:
    - "semver: >=1.31.0 <1.31.9"
    - "regex: 1\.26\.[0-9]-(glibc|uclibc|musl)"
    - "1.29.4"
    - "latest"

This would additionally sync any 1.26.x image with suffix -glibc, -uclibc, or -musl. I updated the issue22 image over at DockerHub.

mkhodave commented 3 years ago

I have used issue22 image tag & it worked , I will give try for regex option as well & let you know , Thanks for adding this , it really helps

xelalexv commented 3 years ago

done with PR #54