search

estesp / manifest-tool

Command line tool to create and query container image manifest list/indexes
Apache License 2.0
751 stars 93 forks source link

manifest-tool seems to be missing support for private registries running on http #14

Closed KurtStam closed 5 years ago

KurtStam commented 8 years ago

[root@centos7 ~]# manifest pushml centos-manifest.yaml INFO[0000] Retrieving digests of images...
INFO[0000] Image "172.30.172.11:5000/default/aarch64-centos:7" is digest sha256:5e96e595fef57a24e42924787ecb272dca6dae83d9e965471601b251e67eeafe; size: 1180 INFO[0000] Image "172.30.172.11:5000/default/centos:7" is digest sha256:8f3eea49e622b9cfb525616f74a33dd4b259e8ecaa107bb422bc9da91fad7ecb; size: 3393 FATA[0000] Failed to setup HTTP client to repository: Ping of V2 registry failed: Get https://172.30.172.11:5000/v2/: http: server gave HTTP response to HTTPS client

estesp commented 8 years ago

Been a busy week; sorry to just now get back to you @KurtStam

Looks like the query of digest codepath was OK with HTTP, but the push/create path is not. I can look into solving this, but also note we are starting work in Docker to make manifest manipulation (including manifest list creation) part of the official docker client; so existence of this tool long-term is tenuous and dependent on my ability to maintain :)

aitorhh commented 7 years ago

I have the same problem as @KurtStam with my private repository. I was trying to do the same as the option in the /etc/docker/daemon.json:

cat /etc/docker/daemon.json
{
        "insecure-registries": ["x.x.x.x:5000"]
}

I try some things like:

createml.go@func setupRepo
options.InsecureRegistries = append(options.InsecureRegistries, "x.x.x.x:5000")

or 

createml.go@getHTTPClient
endpoint.TLSConfig.InsecureSkipVerify = true

But none of these worked.

@estesp is there a solution?

In the meantime I will create a self-signed certificate for my server, and try again.

Thanks!

maitredede commented 6 years ago

Hi, Same problem : I have to deal with an insecure registry that works only in http (and I can't activate https), docker pull/push works fine. But neither docker manifest commands nor your tool works.

clnperez commented 6 years ago

there was just a fix put into the manifest command from the docker cli regarding https. you wouldn't happen to be using rhel/centos would you @MaitreDede ? https://github.com/docker/cli/pull/1378

Edit: http/https, not just https.

ekristen commented 5 years ago

Bump. This is still an issue, probably just need an --insecure flag added for push.

estesp commented 5 years ago

Better late than never? I've opened #70 to correct this issue and will validate/merge it prior to releasing a 1.0.0-final

ekristen commented 5 years ago

Awesome! Definitely will help make testing mulit-arch stuff locally way easier.

ekristen commented 5 years ago

Thanks.

huxiaoliang commented 5 years ago

@estesp it seems --insecure doesn't work.

  1. I get the tools from wget https://github.com/estesp/manifest-tool/releases/download/v1.0.0-rc2/manifest-tool-linux-amd64

  2. for the image from docker hub, it works well

    
[root@icp-registry1 opt]# ./manifest-tool-linux-amd64 inspect busybox
Name:   busybox (Type: application/vnd.docker.distribution.manifest.list.v2+json)
Digest: sha256:954e1f01e80ce09d0887ff6ea10b13a812cb01932a0781d6b0cc23f743a874fd
* Contains 8 manifest references:
1    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
1       Digest: sha256:f79f7a10302c402c052973e3fa42be0344ae6453245669783a9e16da3d56d5b4
1  Mfst Length: 527
1     Platform:
1           -      OS: linux
1           - OS Vers: 
1           - OS Feat: []
1           -    Arch: amd64
1           - Variant: 
1           - Feature: 
1     # Layers: 1
     layer 1: digest = sha256:fc1a6b909f82ce4b72204198d49de3aaf757b3ab2bb823cb6e47c416b97c5985

2 Mfst Type: application/vnd.docker.distribution.manifest.v2+json 2 Digest: sha256:6c84217406361b23552450c776fa929955ac4c5dbe337dd858094e7a79109003 2 Mfst Length: 527 2 Platform: 2 - OS: linux 2 - OS Vers: 2 - OS Feat: [] 2 - Arch: arm 2 - Variant: v5 2 - Feature: 2 # Layers: 1 layer 1: digest = sha256:c83038a50f6e0d7181947b4991cf3993435db7e3462c0bd13c3a4ae97d6b432c

3 Mfst Type: application/vnd.docker.distribution.manifest.v2+json 3 Digest: sha256:ae924ca6c548e21348940acf610e8a6b61227a6f3d9ebdb1bcbc6e1ea0a51b26 3 Mfst Length: 527 3 Platform: 3 - OS: linux 3 - OS Vers: 3 - OS Feat: [] 3 - Arch: arm 3 - Variant: v6 3 - Feature: 3 # Layers: 1 layer 1: digest = sha256:ff0ca67c9bda32fa3a301324fb4c7bd54430e981a0adcf219559a2a3c73fe713

4 Mfst Type: application/vnd.docker.distribution.manifest.v2+json 4 Digest: sha256:6b1ef683f696c503a0afb93d06684a9a70f8f793a90220eb0da569be116ce1d1 4 Mfst Length: 527 4 Platform: 4 - OS: linux 4 - OS Vers: 4 - OS Feat: [] 4 - Arch: arm 4 - Variant: v7 4 - Feature: 4 # Layers: 1 layer 1: digest = sha256:2b35d97f9c8117d50d5d2c9164acf00aa03e41d0a80f9bc2b2044e3e92fa9688

5 Mfst Type: application/vnd.docker.distribution.manifest.v2+json 5 Digest: sha256:93a453ec951f06e2b60269add04b7b37b03c9f5ee4362d714e5ba15d3c4c0f77 5 Mfst Length: 527 5 Platform: 5 - OS: linux 5 - OS Vers: 5 - OS Feat: [] 5 - Arch: arm64 5 - Variant: v8 5 - Feature: 5 # Layers: 1 layer 1: digest = sha256:b04ab0589b9a6d0d597a66bae318d4b08520957d4acfc7bf75496e38d3d7c8d3

6 Mfst Type: application/vnd.docker.distribution.manifest.v2+json 6 Digest: sha256:8e2b1f48d056ac63cde2d1b7c14e35f2cc582aa7223b50ddaf24b769e6d705ec 6 Mfst Length: 527 6 Platform: 6 - OS: linux 6 - OS Vers: 6 - OS Feat: [] 6 - Arch: 386 6 - Variant: 6 - Feature: 6 # Layers: 1 layer 1: digest = sha256:79e848d156eaf50a600bb6129f0ee47b2fa6280d25a52d99d7ee48445f186103

7 Mfst Type: application/vnd.docker.distribution.manifest.v2+json 7 Digest: sha256:a05a8023142d1496e6c2b85101da7491b6347fd9605cdfca8b3a063dfa948748 7 Mfst Length: 528 7 Platform: 7 - OS: linux 7 - OS Vers: 7 - OS Feat: [] 7 - Arch: ppc64le 7 - Variant: 7 - Feature: 7 # Layers: 1 layer 1: digest = sha256:628fa7149e26dcaa64b2ae1ece67309565d6f7e0a04b97f5813998b196226d80

8 Mfst Type: application/vnd.docker.distribution.manifest.v2+json 8 Digest: sha256:8b3c63feea43a24fe7f080c3a59aa9ba036416ac62d4f86a802815b7ecd7e1bd 8 Mfst Length: 528 8 Platform: 8 - OS: linux 8 - OS Vers: 8 - OS Feat: [] 8 - Arch: s390x 8 - Variant: 8 - Feature: 8 # Layers: 1 layer 1: digest = sha256:681c8c6f047294a46fd0f6a2da7

3. the image from my private registry without auth failed

[root@xhu-proxy1 opt]# docker pull 9.30.160.61:5000/ibmcom-amd64/pause-amd64:3.1 3.1: Pulling from ibmcom-amd64/pause-amd64 Digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d Status: Image is up to date for 9.30.160.61:5000/ibmcom-amd64/pause-amd64:3.1 [root@xhu-proxy1 opt]# /opt/manifest-tool-linux-amd64 --insecure inspect 9.30.160.61:5000/ibmcom-amd64/pause-amd64:3.1 FATA[0000] Get https://9.30.160.61:5000/v2/: http: server gave HTTP response to HTTPS client [root@xhu-proxy1 opt]# 


4. my registry was setup by follow command:

docker run -d -p 5000:5000 -v /usr/local/registry:/var/lib/registry --restart=always --name registry registry:2

MrZXR commented 5 years ago

I had the same problem with huxiaoliang

MrZXR commented 5 years ago

@estesp it seems --insecure doesn't work.

  1. I get the tools from wget https://github.com/estesp/manifest-tool/releases/download/v1.0.0-rc2/manifest-tool-linux-amd64
  2. for the image from docker hub, it works well
[root@icp-registry1 opt]# ./manifest-tool-linux-amd64 inspect busybox
Name:   busybox (Type: application/vnd.docker.distribution.manifest.list.v2+json)
Digest: sha256:954e1f01e80ce09d0887ff6ea10b13a812cb01932a0781d6b0cc23f743a874fd
 * Contains 8 manifest references:
1    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
1       Digest: sha256:f79f7a10302c402c052973e3fa42be0344ae6453245669783a9e16da3d56d5b4
1  Mfst Length: 527
1     Platform:
1           -      OS: linux
1           - OS Vers: 
1           - OS Feat: []
1           -    Arch: amd64
1           - Variant: 
1           - Feature: 
1     # Layers: 1
         layer 1: digest = sha256:fc1a6b909f82ce4b72204198d49de3aaf757b3ab2bb823cb6e47c416b97c5985

2    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
2       Digest: sha256:6c84217406361b23552450c776fa929955ac4c5dbe337dd858094e7a79109003
2  Mfst Length: 527
2     Platform:
2           -      OS: linux
2           - OS Vers: 
2           - OS Feat: []
2           -    Arch: arm
2           - Variant: v5
2           - Feature: 
2     # Layers: 1
         layer 1: digest = sha256:c83038a50f6e0d7181947b4991cf3993435db7e3462c0bd13c3a4ae97d6b432c

3    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
3       Digest: sha256:ae924ca6c548e21348940acf610e8a6b61227a6f3d9ebdb1bcbc6e1ea0a51b26
3  Mfst Length: 527
3     Platform:
3           -      OS: linux
3           - OS Vers: 
3           - OS Feat: []
3           -    Arch: arm
3           - Variant: v6
3           - Feature: 
3     # Layers: 1
         layer 1: digest = sha256:ff0ca67c9bda32fa3a301324fb4c7bd54430e981a0adcf219559a2a3c73fe713

4    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
4       Digest: sha256:6b1ef683f696c503a0afb93d06684a9a70f8f793a90220eb0da569be116ce1d1
4  Mfst Length: 527
4     Platform:
4           -      OS: linux
4           - OS Vers: 
4           - OS Feat: []
4           -    Arch: arm
4           - Variant: v7
4           - Feature: 
4     # Layers: 1
         layer 1: digest = sha256:2b35d97f9c8117d50d5d2c9164acf00aa03e41d0a80f9bc2b2044e3e92fa9688

5    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
5       Digest: sha256:93a453ec951f06e2b60269add04b7b37b03c9f5ee4362d714e5ba15d3c4c0f77
5  Mfst Length: 527
5     Platform:
5           -      OS: linux
5           - OS Vers: 
5           - OS Feat: []
5           -    Arch: arm64
5           - Variant: v8
5           - Feature: 
5     # Layers: 1
         layer 1: digest = sha256:b04ab0589b9a6d0d597a66bae318d4b08520957d4acfc7bf75496e38d3d7c8d3

6    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
6       Digest: sha256:8e2b1f48d056ac63cde2d1b7c14e35f2cc582aa7223b50ddaf24b769e6d705ec
6  Mfst Length: 527
6     Platform:
6           -      OS: linux
6           - OS Vers: 
6           - OS Feat: []
6           -    Arch: 386
6           - Variant: 
6           - Feature: 
6     # Layers: 1
         layer 1: digest = sha256:79e848d156eaf50a600bb6129f0ee47b2fa6280d25a52d99d7ee48445f186103

7    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
7       Digest: sha256:a05a8023142d1496e6c2b85101da7491b6347fd9605cdfca8b3a063dfa948748
7  Mfst Length: 528
7     Platform:
7           -      OS: linux
7           - OS Vers: 
7           - OS Feat: []
7           -    Arch: ppc64le
7           - Variant: 
7           - Feature: 
7     # Layers: 1
         layer 1: digest = sha256:628fa7149e26dcaa64b2ae1ece67309565d6f7e0a04b97f5813998b196226d80

8    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
8       Digest: sha256:8b3c63feea43a24fe7f080c3a59aa9ba036416ac62d4f86a802815b7ecd7e1bd
8  Mfst Length: 528
8     Platform:
8           -      OS: linux
8           - OS Vers: 
8           - OS Feat: []
8           -    Arch: s390x
8           - Variant: 
8           - Feature: 
8     # Layers: 1
         layer 1: digest = sha256:681c8c6f047294a46fd0f6a2da7
  1. the image from my private registry without auth failed
[root@xhu-proxy1 opt]# docker pull  9.30.160.61:5000/ibmcom-amd64/pause-amd64:3.1
3.1: Pulling from ibmcom-amd64/pause-amd64
Digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
Status: Image is up to date for 9.30.160.61:5000/ibmcom-amd64/pause-amd64:3.1
[root@xhu-proxy1 opt]# /opt/manifest-tool-linux-amd64 --insecure  inspect 9.30.160.61:5000/ibmcom-amd64/pause-amd64:3.1
FATA[0000] Get https://9.30.160.61:5000/v2/: http: server gave HTTP response to HTTPS client 
[root@xhu-proxy1 opt]#
  1. my registry was setup by follow command:
 docker run -d  -p 5000:5000  -v /usr/local/registry:/var/lib/registry  --restart=always  --name registry  registry:2

I solved this problem temporarily by adding temporary code. in docker/createml.go, line 336 endpoints, err := registryService.LookupPushEndpoints(reference.Domain(repoInfo.Name)) endpoints = endpoints[1:]

for docker/inspect.go, also modify endpoints.

this worked for me. I'm sorry for the ugly code because I haven't used golang before T_T

huxiaoliang commented 5 years ago

@MrZhaoAtBJ could you help create pr to fix this issue?

MrZXR commented 5 years ago

@huxiaoliang sorry I cannot. I am fresh new to this field and I do not have enough ability to write golang code for production environment T_T

estesp commented 5 years ago

Let me take a look; I had tried several commands with a private registry on port 5000 so I thought it was working properly. The "hack" to just ignore the first entry (which is https I assume) and go to the http-only entry needs to be a more complete "fallback" mechanism that tries the endpoint entries in order. Should be fixable.

MrZXR commented 5 years ago

Yeah, I added some debug log, first entry is https. Maybe can skip trying https entry.

estesp commented 5 years ago

See #77 for a more complete fix