search

docker-archive / migrator

Tool to migrate Docker images from Docker Hub or v1 registry to a v2 registry
Apache License 2.0
160 stars 82 forks source link

Issue in Registry Migration #80

Closed Kaleesastha closed 7 years ago

Kaleesastha commented 8 years ago

I tried to migrate the registry V1 to V2 our own registry. I'm facing an issue like

[INFO] Getting a list of images from cmtools

[ERROR] curl => API failure [ERROR] Migration from v1 to v2 failed!

My command is :-

sudo docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -e V1_REGISTRY=admin -e V2_REGISTRY=test -e NO_LOGIN=true -e V1_NO_LOGIN=true -e V2_NO_LOGIN=true -e USE_INSECURE_URL=true -e V1_USE_HTTP=true -e V2_USE_HTTP=true docker/migrator

Kindly help me to do the migration.

Thanks,

Kaleesastha

mbentley commented 8 years ago

I'm guessing the FQDN to your v1 isn't http://admin and your v2 isn't http://test, is it? V1_REGISTRY and V2_REGISTRY should be the FQDN to the registries. I also noticed that you had the env var USE_INSECURE_URL and it needs to be USE_INSECURE_CURL. You might give this command a try (replacing the v1 and v2 URLs in V1_REGISTRY and V2_REGISTRY):

sudo docker run -it --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -e V1_REGISTRY=registryv1.example.com \
  -e V2_REGISTRY=registryv2.example.com \
  -e NO_LOGIN=true \
  -e USE_INSECURE_CURL=true \
  -e USE_HTTP=true \
  docker/migrator
Kaleesastha commented 8 years ago

mbentley,

Thanks for your quick response . I had given registry url with corresponding port, the migration starts and going on.

If any issues i'll open this ticket again.

Thanks,

Kaleesastha

Kaleesastha commented 8 years ago

Mbentley,

While registry migrating i'm facing below issue

failed to register layer: devmapper: Thin Pool has 159137 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior

I need to add any option in daemon .? If so kindly guide me .

Thanks,

Kaleesastha

mbentley commented 8 years ago

It sounds like you have a system that is using devicemapper. The default Docker install using the devicemapper graph driver creates a thin poll that is 100 GB in size so even if you have more disk space, you won't be able to utilize more than 100 GB unless you increase the loopback filesystem to use more space (https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/#/increase-capacity-on-a-running-device) or if you configure devicemapper for direct-lvm (https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/#/for-a-direct-lvm-mode-configuration) which will use a block device for storage instead of the less performant loop-lvm method.

For production systems, configuring direct-lvm is the best option for a longer term solution but for the time being, it would likely be easiest to increase the size of the loop-lvm pool (first option) on this server. Right now because of how the migrator tool works, it has to be able to pull all of your images down to your local system so make sure you have enough disk space which will be approximately the same amount that your v1 registry requires. This means your loop-lvm device needs to be large enough AND your filesystem where /var/lib/docker exists, assuming defaults, needs to be large enough. I believe there is still an option issue to do the migration in stages but that has not been implemented yet.

mbentley commented 8 years ago

@Kaleesastha - I just went ahead and added migration where you can specify how many images you want to migrate at a time in https://github.com/docker/migrator/pull/81. This would certainly help your situation.

Kaleesastha commented 8 years ago

@mbentley - Docker Private Registry migration done successfully and re-tagged into my new host ( http://mynewdomain:5000. Now i'm facing another issue , i.e.,, it prompts username for my new registry url

[!!] Please login to mynewdomain.com:5000: Username:

If i leaved it as empty it encounters an issue like

FATA[0083] Error response from daemon: Get https://mynewdomain.com:5000/v1/users/: tls: oversized record received with length 20527

Even while migrating i passed an environment variable as -e V1_USE_HTTP=true & -e V2_USE_HTTP=true . BTW, why the api url using https mode instead of HTTP . Does environment variable didn't pass / load properly .?

Appreciated the suggestions to fix the issue and clarify my queries..

Thanks, Kaleesastha

mbentley commented 8 years ago

Hmm let me take a look at the code to see if something was missed.

*edit: I did a test of v1 http to v2 http without auth on both and it seemed to work so it may be parameter specific.

mbentley commented 8 years ago

So V2_USE_HTTP seems to be ok. Do you have --insecure-registry mynewdomain.com:5000 set for your daemon arguments? The Docker engine will default to https connections and then fall back to insecure http if you have --insecure-registry set. The error that you are getting is related to the engine itself and not something migrator actually manages.

Do you have authentication enabled for your v2 registry? I see that in your original command that you have the NO_LOGIN variables set which should prevent you from being prompted for credentials. Can you verify that you still have NO_LOGIN=true? Could you provide your updated docker run command so I can try to reproduce the issue as closely as possible?

Kaleesastha commented 8 years ago

@mbentley Thanks for your suggestions to add --insecure-registry option in daemon arguments. This is working fine and proceeding with pushing images from v1 registry to v2 registry, While pushing i faced an another issue and the log is given below for your reference

FATA[0193] open /zoho/docker/devicemapper/mnt/e2fb1a467f882c1e3d7dac7e29ce75a9c92da0d701138d43698e4c054114c37b/rootfs/.wh..wh.plnk/587.1584301: no such file or directory [ERROR] Failed to push mynewdomain:5000/mynamespace/myimagename:latest

Appreciate suggestions about why this might be happening or how to troubleshoot.

Thanks, Kaleesastha

mbentley commented 8 years ago

Hmm, I'm not exactly sure what might be the root cause; seems to be a devicemapper graph driver issue though.

Kaleesastha commented 7 years ago

Hi mbentley,

        I gonna migrate our private registry from V1 to V2 in another machine. While migrating, i'm getting following error 

        [ERROR] Docker daemon not accessible. Is the Docker socket shared into the container as a volume?

       But file available in the machine . refer the below line 

      [root@kaleesastha build]# ls -ltrh /var/run/docker.sock
      srw-rw----. 1 root docker 0 Nov 26 18:04 /var/run/docker.sock

      I'm using the docker version is 1.8.2 , is there any version required to migrate the private registry? If so , kindly let me know .

      Appreciate suggestions about why this might be happening or how to troubleshoot.

Thanks, Kaleesastha

mbentley commented 7 years ago

What is your docker run command that you used, minus any credentials?

On Sat, Nov 26, 2016 at 7:39 AM Kaleesastha notifications@github.com wrote:

Hi mbentley,

    I gonna migrate our private registry from V1 to V2 in another machine. While migrating, i'm getting following error

    [ERROR] Docker daemon not accessible. Is the Docker socket shared into the container as a volume?

   But file available in the machine . refer the below line

  [root@kaleesastha build]# ls -ltrh /var/run/docker.sock
  srw-rw----. 1 root docker 0 Nov 26 18:04 /var/run/docker.sock

  I'm using the docker version is 1.8.2 , is there any version required to migrate the private registry? If so , kindly let me know .

  Appreciate suggestions about why this might be happening or how to troubleshoot.

Thanks, Kaleesastha

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/docker/migrator/issues/80#issuecomment-263061329, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZS7QSvWmClqTWHyrB0YmneVnoM0kS4ks5rCCiJgaJpZM4JNiLO .

Kaleesastha commented 7 years ago

docker run -it -v /var/run/docker.sock:/var/run/docker.sock -e V1_REGISTRY=machine1:5000 -e V2_REGISTRY=machine2:5002 -e NO_LOGIN=true -e USE_HTTP=true docker/migrator 2>&1 | tee migrator.log

mbentley commented 7 years ago

Well, your command certainly looks fine. Docker 1.8.2 should work just fine; as a matter of fact, inside the image, we have a Docker 1.6.2 client for some additional backwards compatibility. Try a basic test: docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock docker/migrator docker info

This would mimic the actual test that it is performing: https://github.com/docker/migrator/blob/7770eb154882c0c67dd7ea1f4bc60201d80a110f/migrator.sh#L150. Are you running SELinux in enforcing mode by chance?

Kaleesastha commented 7 years ago

@mbentley I have executed the above command it encounters an error. Seems inside an image, docker command is not visible ,

FATA[0000] Get http:///var/run/docker.sock/v1.18/info: dial unix /var/run/docker.sock: permission denied. Are you trying to connect to a TLS-enabled daemon without TLS?

But, it was working in docker version 1.10.3 and above . Shall i proceed v that version?

mbentley commented 7 years ago

Yeah, so it sounds like a SELinux issue that may have been fixed in a newer version of the Engine. If you are able, upgrading to a new version of the engine sounds like should solve it.

cardinall commented 7 years ago

hi @mbentley Im facing with same issue, I used the command as you told sudo docker run -it --rm \ -v /var/run/docker.sock:/var/run/docker.sock \ -e V1_REGISTRY=registry.example.xyz:5000 \ -e V2_REGISTRY=registry.example.xyz:5555 \ -e NO_LOGIN=true \ -e USE_INSECURE_CURL=true \ -e USE_HTTP=true \ -e MIGRATION_INCREMENT=10 \ docker/migrator The difference is, that my both v1 and v2 containers runs in same host serving the same DNS uses different buckets and different ports. Error is the same: [ERROR] curl => API failure [ERROR] Migration from v1 to v2 failed! I digged into /usr/local/bin/migrator.sh and found out the command loading repositories: REPO_LIST=$(curl ${V1_OPTIONS} ${V1_PROTO}://${AUTH_CREDS}@${V1_REGISTRY}/v1/search?q= | jq -r '.results | .[] | .name') || catch_error "curl => API failure" but using it in cli I got error 22 bash-4.3# curl --insecure -sf http://registry.example.xyz:5000/v1/search?q= bash-4.3# echo $? 22 could you help to understand what is wrong in my story?

p.s. docker version is 1.6.2 on the host

mbentley commented 7 years ago

Try the same command but just with -kv for flags so we can get the headers and response from your registry. My first guess might be that searching isn't enabled on your registry but that curl command should shed more light on the issue.

cardinall commented 7 years ago

Thanks for operative answer, here is the result [root@ls-dc1-vm-bam-03 ~]# curl --insecure -kv http://registry.example.xyz:5000/v1/search?q=

!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> 404 Not Found Not Found The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.

mbentley commented 7 years ago

Im currently on mobile but that looks like search isn't enabled on your v1 registry. I'd have to look up how you'd enable that, should be an env bar that is added if it is a container or an entry in the config file.

cardinall commented 7 years ago

Thanks a lot for search idea, I have already found that env variable (SEARCH_BACKEND), I have set it true in cloned container, and it works Thanks again.

mbentley commented 7 years ago

No problem; I'm going to go head and close this issue but I'm happy to help if you run into any other issues.

cardinall commented 7 years ago

Thanks you @mbentley :), yes pls close the issue

cardinall commented 7 years ago

hi @mbentley , sorry I could not find better please to contact with you. (if there is, pls share the info) my question: Is there a way to use registry proxy cache instead of registry migration? for more details pls look at https://forums.docker.com/t/registry-proxy-cache-instead-of-registry-migration/28116 topic

mbentley commented 7 years ago

I'm on freenode as mbentley if you want to contact me there.

I can't say that I have tried that. The last time I looked into the proxy caching that was documented in the blog post; it was only caching of public images on Docker Hub. I haven't kept up on the capabilities built in to the registry so I am not certain if that would be possible or not.

Having about 7k images would certainly be problematic and migrating that many images from v1 to v2 would be an extremely long process, especially without the ability to retry on failure. What would using a registry proxy cache buy you specifically that you're looking for? Hiding some of the issues that you're seeing with v1?

cardinall commented 7 years ago

hi @mbentley Yes today I found also in https://docs.docker.com/registry/recipes/mirror/ that it is impossible caching private registry. My current issue with v1 is pull requests time out in rush hours period. About a month ago, when we had about 1 hundred images less, we did not have any issue with v1. Now my plan is migrate all actual images (I can filter , only about 1K images), then use new registry, . Monitor one month to check if one of useful images is absent, manually migrate it too. And at least create mechanism to delete old images which branch is already deleted/merged. Thanks for rely @mbentley , I will use freenode in case of new blockers. thanks again