[Installer]: report on container registry problems

mrsimonemms commented 2 years ago

This is to be used as a notepad for an issue. Please ignore

this worked with all clouds/container registries before #6681 was merged - why is GCP external not working in Self-Hosted when working for SaaS?
FQDN workspaces use https://github.com/csweichel/test-repo/tree/1c287d4c5ef15f97e12b8316191c22cf47b8c1d3
non-FQDN workspaces use any GP example
also check https://github.com/csweichel/test-repo - this uses an FDQN image with image in GCP container registry

Cloud	Registry	Working?	Notes
GCP	InCluster FQDN	Y
GCP	InCluster non-FQDN	Y
GCP	GCP FQDN	N	As non-FQDN.
GCP	GCP non-FQDN	N	Lots of `502 Bad Gateway` errors, ending in `#5 37.17 error: unexpected status: 502 Bad Gateway ------ error: failed to solve: unexpected status: 502 Bad Gateway {"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","command":"build","error":"exit status 1","level":"error","message":"build failed","serviceContext":{"service":"bob","version":""},"severity":"ERROR","time":"2021-11-19T15:42:35Z"}`
Azure	InCluster FQDN	N	Seems to not get the auth token to validate the `HEAD https://registry-1.docker.io/v2/gitpod/workspace-mongodb/manifests/latest` call. Works fine if using GCR based image
Azure	InCluster non-FQDN	N	As FQDN
Azure	Azure FQDN	N
Azure	Azure non-FQDN	N

Observations:

GCP registry errors appear to be unable to authorize to the gcr.io registry, so not relevant to the FQDN status. The .dockerconfigjson value appears valid when running locally as can push an image with the value set to ~/.docker/config.json

Questions:

~is the issue caused by a different workspace.runtime.containerdRuntimeDir values? In GCP it's /var/lib/containerd/io.containerd.runtime.v2.task/k8s.io, in Azure it's /run/containerd/io.containerd.runtime.v1.linux/k8s.io~
~or containerd versions?~
~Azure uses shiftfs, GCP uses fuse~

when comparing GCP incluster to Azure incluster, there are various differences:

Azure has the following lines before the proxy auth request:

{"alias":"d428f509-7d88-4274-8454-ed1cc99ca470","cmd":"/bin/bash","level":"info","message":"started new terminal","serviceContext":{"service":"supervisor","version":"commit-0be2ad26006e15a151aba93ae5c7b4ea3a3311ef"},"severity":"INFO","time":"2021-11-21T16:45:50Z"}
{"command":"{\nsudo -E /app/bob build\n}; exit","level":"info","message":"task terminal has been started","pid":34,"serviceContext":{"service":"supervisor","version":"commit-0be2ad26006e15a151aba93ae5c7b4ea3a3311ef"},"severity":"INFO","terminal":"d428f509-7d88-4274-8454-ed1cc99ca470","time":"2021-11-21T16:45:50Z"}
{"level":"info","message":"Writing build output to /workspace/.gitpod/prebuild-log-0","pid":34,"serviceContext":{"service":"supervisor","version":"commit-0be2ad26006e15a151aba93ae5c7b4ea3a3311ef"},"severity":"INFO","time":"2021-11-21T16:45:50Z"}

Azure has http: proxy error: HEAD https://registry-1.docker.io/v2/gitpod/workspace-mongodb/manifests/latest which GCP doesn't (is this causal or symptomatic?)

csweichel commented 2 years ago

Generalising: the issue seems to arise when we want to pull a base image from docker hub. The question then becomes: why does it work when testing in a workspace, or more broadly, why does it work at all?

mrsimonemms commented 2 years ago

@csweichel I wonder if this is the key. I've added loads of additional (insecure) logs in a branch (not to be merged) and found something interesting....

I've checked the response from repo.Auth.Authorize.

In GCP, it ALWAYS returns back without an error (even on the very first time). In Azure, it ALWAYS returns back this error:

{
  "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",
  "error": "Get \"https://auth.ipv6.docker.com/token?scope=repository%3Agitpod%2Fworkspace-mongodb%3Apull\u0026service=registry.docker.io\": dial tcp [2600:1f18:2148:bc00:cec:e441:76aa:4346]:443: connect: network is unreachable\nfailed to fetch anonymous token\ngithub.com/containerd/containerd/remotes/docker.(*authHandler).doBearerAuth\n\t/workspace/go/pkg/mod/github.com/containerd/containerd@v1.5.5/remotes/docker/authorizer.go:306\ngithub.com/containerd/containerd/remotes/docker.(*authHandler).authorize\n\t/workspace/go/pkg/mod/github.com/containerd/containerd@v1.5.5/remotes/docker/authorizer.go:228\ngithub.com/containerd/containerd/remotes/docker.(*dockerAuthorizer).Authorize\n\t/workspace/go/pkg/mod/github.com/containerd/containerd@v1.5.5/remotes/docker/authorizer.go:112\ngithub.com/gitpod-io/gitpod/image-builder/bob/pkg/proxy.(*Proxy).reverse.func2\n\t/tmp/build/components-image-builder-bob--app.65626b7c97210e6bd97e4ad3c440d299218b427f/pkg/proxy/proxy.go:173\ngithub.com/hashicorp/go-retryablehttp.(*Client).Do\n\t/workspace/go/pkg/mod/github.com/hashicorp/go-retryablehttp@v0.7.0/client.go:584\ngithub.com/hashicorp/go-retryablehttp.(*RoundTripper).RoundTrip\n\t/workspace/go/pkg/mod/github.com/hashicorp/go-retryablehttp@v0.7.0/roundtripper.go:44\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\t/home/gitpod/go/src/net/http/httputil/reverseproxy.go:298\ngithub.com/gitpod-io/gitpod/image-builder/bob/pkg/proxy.(*Proxy).ServeHTTP\n\t/tmp/build/components-image-builder-bob--app.65626b7c97210e6bd97e4ad3c440d299218b427f/pkg/proxy/proxy.go:127\nnet/http.(*ServeMux).ServeHTTP\n\t/home/gitpod/go/src/net/http/server.go:2424\nnet/http.serverHandler.ServeHTTP\n\t/home/gitpod/go/src/net/http/server.go:2878\nnet/http.(*conn).serve\n\t/home/gitpod/go/src/net/http/server.go:1929\nruntime.goexit\n\t/home/gitpod/go/src/runtime/asm_amd64.s:1581",
  "level": "error",
  "message": "sje auth err",
  "serviceContext": {
    "service": "bob",
    "version": ""
  },
  "severity": "ERROR",
  "time": "2021-11-22T13:28:08Z"
}

So, why is the network unreachable?

mrsimonemms commented 2 years ago

It appears that in Azure, the www-authenticate value that comes back from the initial call to https://registry-1.docker.io/v2/gitpod/workspace-mongodb/manifests/latest is to auth.ipv6.docker.com rather than auth.docker.com.

And I've no idea why

Interestingly, the X-Forwarded-For value in Azure is ::1 and in GCP it's 127.0.0.1 - this appears to set the IPV6 URL

Edit: unbelievably, if I set X-Forwarded-For to 127.0.0.1, it works

gitpod-io / gitpod

[Installer]: report on container registry problems #6810