search

vmware / vic

vSphere Integrated Containers Engine is a container runtime for vSphere.
http://vmware.github.io/vic
Other
639 stars 173 forks source link

"Failed to obtain OAuth endpoint: download failed: www-authenticate header is corrupted" with private Docker Registries #8391

Open desolat opened 5 years ago

desolat commented 5 years ago

Summary

Same error as in #6633: "Failed to obtain OAuth endpoint: download failed: www-authenticate header is corrupted" when accessing a private Docker registry (tested against Sonatype Nexus 3 and AWS ECR).

Environment information

Sonatype Nexus Repository Manager OSS 3.13.0-01 AWS ECR

vSphere and vCenter Server version

vCenter Server 6.7.0.11000

VIC version

VIC 1.4.3-6240-deedb985 VIC 1.5.0

VCH configuration
vic-machine-linux create \
--name $MACHINE_NAME \
--debug 1 \
--compute-resource Cluster \
--cpu-reservation 1 \
--cpu-shares normal \
--memory-reservation 1 \
--memory-shares normal \
--endpoint-cpu 1 \
--endpoint-memory 4096 \
--affinity-vm-group \
--image-store QS-Storage \
--base-image-size 8GB \
--volume-store QS-Storage:default \
--bridge-network $MACHINE_NAME-bridge \
--bridge-network-range 172.16.0.0/12 \
--public-network QS\ Net \
--dns-server $DNS \
--http-proxy $PROXY \
--https-proxy $PROXY \
--no-proxy localhost,127.0.0.1,.local,.kp,.kup \
--tls-cname $MACHINE_NAME \
--organization $ORG \
--certificate-key-size 2048 \
--no-tlsverify \
--user $USER \
--thumbprint $THUMBPRINT \
--target $VCENTER/Datacenter \
--insecure-registry $NEXUS_REGISTRY_URL \
--registry-ca $CA_CERT \
--ops-user $VIC_OPS_USER \
--ops-grant-perms
Steps to reproduce

Run docker or docker-compose commands against a docker private registry (such as Nexus 3 or AWS ECR).

Actual behavior

docker or docker-compose commands (like pull) against the docker private registry fail with Failed to obtain OAuth endpoint: download failed: www-authenticate header is corrupted.

Apparently the www-authenticate header requests basic authentication but VIC tries OAuth2 authentication which fails.

Expected behavior

Nexus 3 private registry interaction works flawlessly as with the Docker.inc solution.

Logs

Nov 20 2018 14:14:42.377Z DEBUG Calling POST /v1.25/auth
Nov 20 2018 14:14:42.378Z DEBUG form data: {"password":"*****","serveraddress":"","username":""}
Nov 20 2018 14:14:42.378Z DEBUG [BEGIN]  [vic/lib/apiservers/engine/backends.(*SystemBackend).AuthenticateToRegistry:332]
Nov 20 2018 14:14:42.378Z INFO  running product VM discovery
Nov 20 2018 14:14:42.378Z DEBUG Get category VsphereIntegratedContainers
Nov 20 2018 14:14:42.378Z DEBUG List all categories
Nov 20 2018 14:14:42.399Z DEBUG Get categories failed with status code: %!s(int=-1), error message: call failed: error occurred trying to connect: Get https:///rest/com/vmware/cis/tagging/category: Forbidden
Nov 20 2018 14:14:42.399Z DEBUG Get category failed for: Status code: -1, error: call failed: error occurred trying to connect: Get https:///rest/com/vmware/cis/tagging/category: Forbidden
Nov 20 2018 14:14:42.400Z WARN  could not locate management portal, returning last known config: could not find ova tag: get categories by name VsphereIntegratedContainers failed: Status code: -1, error: call failed: error occurred trying to connect: Get https:///rest/com/vmware/cis/tagging/category: Forbidden
Nov 20 2018 14:14:42.445Z DEBUG header = http.Header{"X-Content-Type-Options":[]string{"nosniff"}, "Www-Authenticate":[]string{"BASIC realm=\"Sonatype Nexus Repository Manager\""}, "Docker-Distribution-Api-Version":[]string{"registry/2.0"}, "Server":[]string{"nginx/1.10.2"}, "Content-Length":[]string{"113"}, "Connection":[]string{"keep-alive"}, "Content-Security-Policy":[]string{"sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation"}, "Date":[]string{"Tue, 20 Nov 2018 14:14:44 GMT"}, "Content-Type":[]string{"application/json"}}
Nov 20 2018 14:14:42.455Z DEBUG Looking up OAuth URL from server https:///v2/
Nov 20 2018 14:14:42.455Z ERROR Looking up OAuth URL failed: download failed: www-authenticate header is corrupted
Nov 20 2018 14:14:42.455Z DEBUG [ END ]  [vic/lib/apiservers/engine/backends.(*SystemBackend).AuthenticateToRegistry:332] [76.9386ms] 
Nov 20 2018 14:14:42.455Z ERROR Handler for POST /v1.25/auth returned error: download failed: www-authenticate header is corrupted
yuyangbj commented 5 years ago

@desolat from the log, I think you delete the registry address, right? And from the response including www-authenticate, VIC assumes it needs to include 'bearer' not BASIC.

Refer to https://docs.docker.com/registry/spec/auth/token/

desolat commented 5 years ago

@yuyangbj I don't get what you want to say or ask. As being already said, VIC has a ongoing problem here which was also described in #6633. I just want to be able to use docker-compose pull against Nexus 3 Docker Repositories also on VIC VCH's. And apparently VIC does not correctly parse the www-authenticate header and does not correctly try basic auth when requested by the registry.

wjun commented 5 years ago

@desolat VIC currently only support Docker registry v2 protocols as Yang posted above: https://docs.docker.com/registry/spec/auth/token/ We will test locally and see if there is a way to talk to this registry through v2 protocols.

wjun commented 5 years ago

@desolat I set up a nexus docker registry locally, and after I tick "Allow anonymous docker pull ( Docker Bearer Token Realm required )", I can login through VCH. This is expected as registry v2 protocol requires bearer token. Then I noticed Nexus images' manifests do not include "Signature" key which is required by VCH to verify the digest during image pull. This is insecure if "Signature" key is ignored in manifest.

desolat commented 5 years ago
  1. @wjun Ok, so how am I expected to authenticate against private Docker registries requiring login such as Nexus or AWS ECR? I found the issues #3044 and vmware/vic-product/issues/1946 on that behalf.
  2. I can confirm that anonymous pulls brings me one step further with the Nexus registry. But as being said, anonymous pull is not possible everywhere.
  3. The Docker image manifest specification v2, schema 2 (which is the one used by Nexus OSS 3.15.1-01 here) does not mention signed manifests anymore (as did schema 1): https://docs.docker.com/registry/spec/manifest-v2-2/. Is this really necessary (anymore)?
  4. With anonymous pull I now get the following error: 
Pulling webproxy ... error

ERROR: for webproxy  FetchToken (https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken) failed: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
FetchToken (https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken) failed: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken

The repository is available on a non-standard port and that port is missing in the token URL! That's probably a separate bug. Should I create another issue for that?

Here's the VCH log:

Apr  3 2019 10:26:25.935Z INFO  PullImage: reference: nexus.kiebackpeter.kup:18443/quasar/webproxy:latest, 127.0.0.1:2377, portlayer: "127.0.0.1:2377"
Apr  3 2019 10:26:25.935Z DEBUG [BEGIN] op=202.164 [vic/lib/imagec.(*ImageC).PullImage:479] quasar/webproxy
Apr  3 2019 10:26:25.935Z DEBUG [BEGIN] op=202.164 [vic/lib/imagec.(*ImageC).prepareTransfer:554] quasar/webproxy
Apr  3 2019 10:26:25.935Z DEBUG [BEGIN] op=202.164 [vic/lib/imagec.(*ImageC).ParseReference:164] quasar/webproxy
Apr  3 2019 10:26:25.935Z DEBUG [ END ] op=202.164 [vic/lib/imagec.(*ImageC).ParseReference:164] [21.468µs] quasar/webproxy
Apr  3 2019 10:26:25.936Z INFO  Using UUID (420204a9-efc8-93b4-346f-bc911ca66876) for imagestore name
Apr  3 2019 10:26:25.936Z DEBUG Running with portlayer
Apr  3 2019 10:26:25.936Z DEBUG [BEGIN] op=202.164 [vic/lib/imagec.PingPortLayer:35] 127.0.0.1:2377
Apr  3 2019 10:26:25.937Z DEBUG [ END ] op=202.164 [vic/lib/imagec.PingPortLayer:35] [1.004ms] 127.0.0.1:2377
Apr  3 2019 10:26:25.937Z DEBUG [BEGIN] op=202.164 [vic/lib/imagec.LearnRegistryURL:74] nexus.kiebackpeter.kup:18443
Apr  3 2019 10:26:25.937Z DEBUG op=202.164: Trying https scheme for &imagec.Options{Reference:(*reference.taggedRef)(0xc4205ddad0), Registry:"nexus.kiebackpeter.kup:18443", Image:"quasar/webproxy", Tag:"latest", Destination:"/tmp", Host:"127.0.0.1:2377", Storename:"420204a9-efc8-93b4-346f-bc911ca66876", Username:"", Password:"", Token:(*fetcher.Token)(nil), Timeout:3600000000000, Outstream:(*ioutils.WriteFlusher)(0xc4214d3180), InsecureSkipVerify:false, InsecureAllowHTTP:true, ImageManifestSchema1:(*imagec.Manifest)(nil), ImageManifestSchema2:(*schema2.DeserializedManifest)(nil), ManifestDigest:"", RegistryCAs:(*x509.CertPool)(0xc4201cc480), Standalone:false, ImageStore:""}
Apr  3 2019 10:26:25.937Z DEBUG [BEGIN] op=202.164 [vic/pkg/registry.Reachable:33] nexus.kiebackpeter.kup:18443
Apr  3 2019 10:26:25.937Z DEBUG URL: https://nexus.kiebackpeter.kup:18443/v2/
Apr  3 2019 10:26:25.937Z DEBUG [BEGIN] op=202.164 [vic/pkg/fetcher.(*URLFetcher).Head:416] https://nexus.kiebackpeter.kup:18443/v2/
Apr  3 2019 10:26:26.049Z DEBUG [ END ] op=202.164 [vic/pkg/fetcher.(*URLFetcher).Head:416] [112.09599ms] https://nexus.kiebackpeter.kup:18443/v2/
Apr  3 2019 10:26:26.049Z DEBUG [ END ] op=202.164 [vic/pkg/registry.Reachable:33] [112.239287ms] nexus.kiebackpeter.kup:18443
Apr  3 2019 10:26:26.049Z DEBUG [ END ] op=202.164 [vic/lib/imagec.LearnRegistryURL:74] [112.521137ms] nexus.kiebackpeter.kup:18443
Apr  3 2019 10:26:26.049Z DEBUG [BEGIN] op=202.164 [vic/lib/imagec.LearnAuthURL:99] nexus.kiebackpeter.kup:18443/quasar/webproxy:latest
Apr  3 2019 10:26:26.049Z DEBUG op=202.164: Pinging https://nexus.kiebackpeter.kup:18443/v2/quasar/webproxy/manifests/latest
Apr  3 2019 10:26:26.049Z DEBUG [BEGIN] op=202.164 [vic/pkg/fetcher.(*URLFetcher).Ping:393] https://nexus.kiebackpeter.kup:18443/v2/quasar/webproxy/manifests/latest
Apr  3 2019 10:26:26.091Z DEBUG header = http.Header{"Server":[]string{"nginx/1.10.2"}, "Connection":[]string{"keep-alive"}, "Www-Authenticate":[]string{"Bearer realm=\"https://nexus.kiebackpeter.kup/v2/token\",service=\"https://nexus.kiebackpeter.kup/v2/token\""}, "Content-Security-Policy":[]string{"sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation"}, "Docker-Distribution-Api-Version":[]string{"registry/2.0"}, "Date":[]string{"Wed, 03 Apr 2019 10:27:53 GMT"}, "Content-Type":[]string{"application/json"}, "Content-Length":[]string{"113"}, "X-Content-Type-Options":[]string{"nosniff"}}
Apr  3 2019 10:26:26.099Z DEBUG [ END ] op=202.164 [vic/pkg/fetcher.(*URLFetcher).Ping:393] [49.315766ms] https://nexus.kiebackpeter.kup:18443/v2/quasar/webproxy/manifests/latest
Apr  3 2019 10:26:26.099Z DEBUG [ END ] op=202.164 [vic/lib/imagec.LearnAuthURL:99] [49.509769ms] nexus.kiebackpeter.kup:18443/quasar/webproxy:latest
Apr  3 2019 10:26:26.099Z DEBUG [BEGIN] op=202.164 [vic/lib/imagec.FetchToken:156] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.099Z DEBUG op=202.164: URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.099Z DEBUG [BEGIN] op=202.164 [vic/pkg/fetcher.(*URLFetcher).FetchAuthToken:211] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.099Z DEBUG 202.165: [OperationFromContext] [vic/pkg/fetcher.(*URLFetcher).Fetch:135]
Apr  3 2019 10:26:26.099Z DEBUG [BEGIN] op=202.165 [vic/pkg/fetcher.(*URLFetcher).Fetch:136] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.099Z DEBUG 202.166: [OperationFromContext] [vic/pkg/fetcher.(*URLFetcher).fetchToString:367]
Apr  3 2019 10:26:26.099Z DEBUG [BEGIN] op=202.166 [vic/pkg/fetcher.(*URLFetcher).fetchToString:368] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.099Z DEBUG 202.167: [OperationFromContext] [vic/pkg/fetcher.(*URLFetcher).fetch:237]
Apr  3 2019 10:26:26.099Z DEBUG [BEGIN] op=202.167 [vic/pkg/fetcher.(*URLFetcher).fetch:238] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.100Z DEBUG Setting user-agent to vic/v1.5.0
Apr  3 2019 10:26:26.179Z DEBUG [ END ] op=202.167 [vic/pkg/fetcher.(*URLFetcher).fetch:238] [79.422828ms] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.179Z ERROR Fetch (https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken) to string error: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.179Z DEBUG [ END ] op=202.166 [vic/pkg/fetcher.(*URLFetcher).fetchToString:368] [79.690368ms] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.179Z DEBUG Error: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.179Z DEBUG [ END ] op=202.165 [vic/pkg/fetcher.(*URLFetcher).Fetch:136] [79.862435ms] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.179Z ERROR Download failed: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.179Z DEBUG [ END ] op=202.164 [vic/pkg/fetcher.(*URLFetcher).FetchAuthToken:211] [80.040588ms] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.179Z ERROR op=202.164: FetchToken (https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken) failed: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.179Z DEBUG [ END ] op=202.164 [vic/lib/imagec.FetchToken:156] [80.257806ms] https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.179Z ERROR Failed to fetch OAuth token: FetchToken (https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken) failed: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
Apr  3 2019 10:26:26.179Z DEBUG [ END ] op=202.164 [vic/lib/imagec.(*ImageC).prepareTransfer:554] [244.027929ms] quasar/webproxy
Apr  3 2019 10:26:26.179Z DEBUG [ END ] op=202.164 [vic/lib/imagec.(*ImageC).PullImage:479] [244.094062ms] quasar/webproxy
Apr  3 2019 10:26:26.179Z DEBUG [ END ] op=202.164 [vic/lib/apiservers/engine/backends.(*ImageBackend).PullImage:350] [287.054229ms] nexus.kiebackpeter.kup:18443/quasar/webproxy
Apr  3 2019 10:26:26.179Z ERROR Handler for POST /v1.25/images/create returned error: FetchToken (https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken) failed: image tag not found: Not found: 404, URL: https://nexus.kiebackpeter.kup/v2/token?service=https%3A%2F%2Fnexus.kiebackpeter.kup%2Fv2%2Ftoken
kkuphal commented 5 years ago

I'm experiencing the same issue when authenticating against private Oracle Container Registry (https://container-registry.oracle.com/)

desolat commented 4 years ago

@wjun I revisited VIC and version 1.5.4 is still affected by these problems. Is there going to be any progress? Any conclusion from my Docker image manifest specification analysis?