search

docker-archive / migrator

Tool to migrate Docker images from Docker Hub or v1 registry to a v2 registry
Apache License 2.0
160 stars 82 forks source link

Do not migrate tags that exist at the target repository already #91

Closed madmuffin1 closed 8 years ago

madmuffin1 commented 8 years ago

We are migrating a very large repository, and while we migrate, new tags will be pushed. Also, we are not really switching over to the new repository at instant, but there will be a couple of days after beginning of migration up to when we change the DNS to target the v2 repository.

The current implementation would try to push all images again as soon as we start the second run, which doesn't really bother when it comes to pushing them again, but the images will be pulled down to the migration machine anyways, which consumes a lot of time.

To make this faster, I added a routine that will compare the list of tags at the target with the tags found at the source. If a tag is present at the target, it will not try to push it again.

Note: I was unable to test the part when migrating from the official docker.io hub which apparently works a bit different. A quick test if that works would be very much appreciated.

mbentley commented 8 years ago

Also, the Jenkins job did run and I can see that some of the tests did successfully skip migrating images because they already exist in the destination registries. https://ci.mbentley.dckrindy.io/job/migrator-pr/77/consoleFull

You can see that the tests with insecure URLs failed to filter out existing images: run a test migration from v1 to v2 (http; w/auth) and run a test migration from v1 to v2 (http; w/o auth)

But you can see that run a test migration from v1 to v2 (w/auth) and run a test migration from v1 to v2 (w/o auth) skipped those images that are already in the registry.

madmuffin1 commented 8 years ago

Thanks for the quick feedback. I agree to all your change requests and I will see into them tonight or tomorrow.

madmuffin1 commented 8 years ago

Would you see this to be a new default behavior or something disabled by default? I would also update the docs accordingly.

If you don't mind, I would just amend the existing commit to make sure this one feature is contained in a single commit

mbentley commented 8 years ago

I think I would like it disabled by default to retain the previous behavior. And squashing commits would be great. Thanks!

madmuffin1 commented 8 years ago

If I didn't miss anything, I should have fixed all the requests you made. I decided for less code-duplication and moved the check if the parameter is set into the method that queries the existing tags and adapted its name.

Ready for re-review.

TanmayaAPYL commented 8 years ago

@mbentley it will be great if you can review this soon, as this may help us as well. @madmuffin1 Thanks for your efforts!

mbentley commented 8 years ago

retest this please

mbentley commented 8 years ago

Also, my apologies, from merging another PR, there is now a small conflict.

madmuffin1 commented 8 years ago

Fixed and rebased.

mbentley commented 8 years ago

Thanks! I appreciate the contribution!