Open desolat opened 5 years ago
@desolat from the log, I think you delete the registry address, right? And from the response including www-authenticate, VIC assumes it needs to include 'bearer' not BASIC.
@yuyangbj I don't get what you want to say or ask. As being already said, VIC has a ongoing problem here which was also described in #6633. I just want to be able to use docker-compose pull
against Nexus 3 Docker Repositories also on VIC VCH's. And apparently VIC does not correctly parse the www-authenticate header and does not correctly try basic auth when requested by the registry.
@desolat VIC currently only support Docker registry v2 protocols as Yang posted above: https://docs.docker.com/registry/spec/auth/token/ We will test locally and see if there is a way to talk to this registry through v2 protocols.
@desolat I set up a nexus docker registry locally, and after I tick "Allow anonymous docker pull ( Docker Bearer Token Realm required )", I can login through VCH. This is expected as registry v2 protocol requires bearer token. Then I noticed Nexus images' manifests do not include "Signature" key which is required by VCH to verify the digest during image pull. This is insecure if "Signature" key is ignored in manifest.
Pulling webproxy ... error
ERROR: for webproxy FetchToken (https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken) failed: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
FetchToken (https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken) failed: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
The repository is available on a non-standard port and that port is missing in the token URL! That's probably a separate bug. Should I create another issue for that?
Here's the VCH log:
Apr 3 2019 10:26:25.935Z INFO PullImage: reference: nexus.kiebackpeter.kup:18443/quasar/webproxy:latest, 127.0.0.1:2377, portlayer: "127.0.0.1:2377"
Apr 3 2019 10:26:25.935Z DEBUG [BEGIN] op=202.164 [vic/lib/imagec.(*ImageC).PullImage:479] quasar/webproxy
Apr 3 2019 10:26:25.935Z DEBUG [BEGIN] op=202.164 [vic/lib/imagec.(*ImageC).prepareTransfer:554] quasar/webproxy
Apr 3 2019 10:26:25.935Z DEBUG [BEGIN] op=202.164 [vic/lib/imagec.(*ImageC).ParseReference:164] quasar/webproxy
Apr 3 2019 10:26:25.935Z DEBUG [ END ] op=202.164 [vic/lib/imagec.(*ImageC).ParseReference:164] [21.468µs] quasar/webproxy
Apr 3 2019 10:26:25.936Z INFO Using UUID (420204a9-efc8-93b4-346f-bc911ca66876) for imagestore name
Apr 3 2019 10:26:25.936Z DEBUG Running with portlayer
Apr 3 2019 10:26:25.936Z DEBUG [BEGIN] op=202.164 [vic/lib/imagec.PingPortLayer:35] 127.0.0.1:2377
Apr 3 2019 10:26:25.937Z DEBUG [ END ] op=202.164 [vic/lib/imagec.PingPortLayer:35] [1.004ms] 127.0.0.1:2377
Apr 3 2019 10:26:25.937Z DEBUG [BEGIN] op=202.164 [vic/lib/imagec.LearnRegistryURL:74] nexus.kiebackpeter.kup:18443
Apr 3 2019 10:26:25.937Z DEBUG op=202.164: Trying https scheme for &imagec.Options{Reference:(*reference.taggedRef)(0xc4205ddad0), Registry:"nexus.kiebackpeter.kup:18443", Image:"quasar/webproxy", Tag:"latest", Destination:"/tmp", Host:"127.0.0.1:2377", Storename:"420204a9-efc8-93b4-346f-bc911ca66876", Username:"", Password:"", Token:(*fetcher.Token)(nil), Timeout:3600000000000, Outstream:(*ioutils.WriteFlusher)(0xc4214d3180), InsecureSkipVerify:false, InsecureAllowHTTP:true, ImageManifestSchema1:(*imagec.Manifest)(nil), ImageManifestSchema2:(*schema2.DeserializedManifest)(nil), ManifestDigest:"", RegistryCAs:(*x509.CertPool)(0xc4201cc480), Standalone:false, ImageStore:""}
Apr 3 2019 10:26:25.937Z DEBUG [BEGIN] op=202.164 [vic/pkg/registry.Reachable:33] nexus.kiebackpeter.kup:18443
Apr 3 2019 10:26:25.937Z DEBUG URL: https://nexus.kiebackpeter.kup:18443/v2/
Apr 3 2019 10:26:25.937Z DEBUG [BEGIN] op=202.164 [vic/pkg/fetcher.(*URLFetcher).Head:416] https://nexus.kiebackpeter.kup:18443/v2/
Apr 3 2019 10:26:26.049Z DEBUG [ END ] op=202.164 [vic/pkg/fetcher.(*URLFetcher).Head:416] [112.09599ms] https://nexus.kiebackpeter.kup:18443/v2/
Apr 3 2019 10:26:26.049Z DEBUG [ END ] op=202.164 [vic/pkg/registry.Reachable:33] [112.239287ms] nexus.kiebackpeter.kup:18443
Apr 3 2019 10:26:26.049Z DEBUG [ END ] op=202.164 [vic/lib/imagec.LearnRegistryURL:74] [112.521137ms] nexus.kiebackpeter.kup:18443
Apr 3 2019 10:26:26.049Z DEBUG [BEGIN] op=202.164 [vic/lib/imagec.LearnAuthURL:99] nexus.kiebackpeter.kup:18443/quasar/webproxy:latest
Apr 3 2019 10:26:26.049Z DEBUG op=202.164: Pinging https://nexus.kiebackpeter.kup:18443/v2/quasar/webproxy/manifests/latest
Apr 3 2019 10:26:26.049Z DEBUG [BEGIN] op=202.164 [vic/pkg/fetcher.(*URLFetcher).Ping:393] https://nexus.kiebackpeter.kup:18443/v2/quasar/webproxy/manifests/latest
Apr 3 2019 10:26:26.091Z DEBUG header = http.Header{"Server":[]string{"nginx/1.10.2"}, "Connection":[]string{"keep-alive"}, "Www-Authenticate":[]string{"Bearer realm=\"https://nexus.kiebackpeter.kup/v2/token\",service=\"https://nexus.kiebackpeter.kup/v2/token\""}, "Content-Security-Policy":[]string{"sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation"}, "Docker-Distribution-Api-Version":[]string{"registry/2.0"}, "Date":[]string{"Wed, 03 Apr 2019 10:27:53 GMT"}, "Content-Type":[]string{"application/json"}, "Content-Length":[]string{"113"}, "X-Content-Type-Options":[]string{"nosniff"}}
Apr 3 2019 10:26:26.099Z DEBUG [ END ] op=202.164 [vic/pkg/fetcher.(*URLFetcher).Ping:393] [49.315766ms] https://nexus.kiebackpeter.kup:18443/v2/quasar/webproxy/manifests/latest
Apr 3 2019 10:26:26.099Z DEBUG [ END ] op=202.164 [vic/lib/imagec.LearnAuthURL:99] [49.509769ms] nexus.kiebackpeter.kup:18443/quasar/webproxy:latest
Apr 3 2019 10:26:26.099Z DEBUG [BEGIN] op=202.164 [vic/lib/imagec.FetchToken:156] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.099Z DEBUG op=202.164: URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.099Z DEBUG [BEGIN] op=202.164 [vic/pkg/fetcher.(*URLFetcher).FetchAuthToken:211] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.099Z DEBUG 202.165: [OperationFromContext] [vic/pkg/fetcher.(*URLFetcher).Fetch:135]
Apr 3 2019 10:26:26.099Z DEBUG [BEGIN] op=202.165 [vic/pkg/fetcher.(*URLFetcher).Fetch:136] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.099Z DEBUG 202.166: [OperationFromContext] [vic/pkg/fetcher.(*URLFetcher).fetchToString:367]
Apr 3 2019 10:26:26.099Z DEBUG [BEGIN] op=202.166 [vic/pkg/fetcher.(*URLFetcher).fetchToString:368] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.099Z DEBUG 202.167: [OperationFromContext] [vic/pkg/fetcher.(*URLFetcher).fetch:237]
Apr 3 2019 10:26:26.099Z DEBUG [BEGIN] op=202.167 [vic/pkg/fetcher.(*URLFetcher).fetch:238] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.100Z DEBUG Setting user-agent to vic/v1.5.0
Apr 3 2019 10:26:26.179Z DEBUG [ END ] op=202.167 [vic/pkg/fetcher.(*URLFetcher).fetch:238] [79.422828ms] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.179Z ERROR Fetch (https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken) to string error: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.179Z DEBUG [ END ] op=202.166 [vic/pkg/fetcher.(*URLFetcher).fetchToString:368] [79.690368ms] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.179Z DEBUG Error: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.179Z DEBUG [ END ] op=202.165 [vic/pkg/fetcher.(*URLFetcher).Fetch:136] [79.862435ms] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.179Z ERROR Download failed: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.179Z DEBUG [ END ] op=202.164 [vic/pkg/fetcher.(*URLFetcher).FetchAuthToken:211] [80.040588ms] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.179Z ERROR op=202.164: FetchToken (https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken) failed: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.179Z DEBUG [ END ] op=202.164 [vic/lib/imagec.FetchToken:156] [80.257806ms] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.179Z ERROR Failed to fetch OAuth token: FetchToken (https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken) failed: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr 3 2019 10:26:26.179Z DEBUG [ END ] op=202.164 [vic/lib/imagec.(*ImageC).prepareTransfer:554] [244.027929ms] quasar/webproxy
Apr 3 2019 10:26:26.179Z DEBUG [ END ] op=202.164 [vic/lib/imagec.(*ImageC).PullImage:479] [244.094062ms] quasar/webproxy
Apr 3 2019 10:26:26.179Z DEBUG [ END ] op=202.164 [vic/lib/apiservers/engine/backends.(*ImageBackend).PullImage:350] [287.054229ms] nexus.kiebackpeter.kup:18443/quasar/webproxy
Apr 3 2019 10:26:26.179Z ERROR Handler for POST /v1.25/images/create returned error: FetchToken (https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken) failed: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
I'm experiencing the same issue when authenticating against private Oracle Container Registry (https://container-registry.oracle.com/)
@wjun I revisited VIC and version 1.5.4 is still affected by these problems. Is there going to be any progress? Any conclusion from my Docker image manifest specification analysis?
Summary
Same error as in #6633: "Failed to obtain OAuth endpoint: download failed: www-authenticate header is corrupted" when accessing a private Docker registry (tested against Sonatype Nexus 3 and AWS ECR).
Environment information
Sonatype Nexus Repository Manager OSS 3.13.0-01 AWS ECR
vSphere and vCenter Server version
vCenter Server 6.7.0.11000
VIC version
VIC 1.4.3-6240-deedb985 VIC 1.5.0
VCH configuration
Steps to reproduce
Run docker or docker-compose commands against a docker private registry (such as Nexus 3 or AWS ECR).
Actual behavior
docker
ordocker-compose
commands (like pull) against the docker private registry fail withFailed to obtain OAuth endpoint: download failed: www-authenticate header is corrupted
.Apparently the www-authenticate header requests basic authentication but VIC tries OAuth2 authentication which fails.
Expected behavior
Nexus 3 private registry interaction works flawlessly as with the Docker.inc solution.
Logs