failed to fetch oauth token: Post "xxx" dial tcp i/o timeout

[renhao-0518]

Occasionally this error occurs，and terminate the build after 30 seconds，Retry and recover. I don't see harbor timeout logs，how can i avoid this erro ？

Here is the buildkit log

time="2023-05-16T12:36:08Z" level=debug msg="serving grpc connection" spanID=87bec976f72e81c2 traceID=0636a35fe8f91f2ab55e02aa4448b230
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 237B done
#2 DONE 0.0s

#3 [auth] kubeone-runtime/golang-runtime:pull token for repoin.harbor.net
#3 DONE 0.0s

#4 [internal] load metadata for repoin.harbor.net/kubeone-runtime/golang-runtime:v1.0-pre
#4 ERROR: failed to authorize: DeadlineExceeded: failed to fetch oauth token: Post "https://repoin.harbor.net/service/token": dial tcp 10.249.179.76:443: i/o timeout
time="2023-05-16T12:36:38Z" level=debug msg="stopping session" spanID=87bec976f72e81c2 traceID=0636a35fe8f91f2ab55e02aa4448b230
------
 > [internal] load metadata for repoin.harbor.net/kubeone-runtime/golang-runtime:v1.0-pre:
------
Dockerfile:1
--------------------
   1 | >>> FROM repoin.harbor.net/kubeone-runtime/golang-runtime:v1.0-pre
   2 |     
   3 |     COPY --chown=ops:ops . /opt/apps/
--------------------
error: failed to solve: DeadlineExceeded: DeadlineExceeded: DeadlineExceeded: repoin.harbor.net/kubeone-runtime/golang-runtime:v1.0-pre: failed to authorize: DeadlineExceeded: failed to fetch oauth token: Post "https://repoin.harbor.net/service/token": dial tcp 10.249.179.76:443: i/o timeout
19 v0.11.5 buildctl --debug --addr=tcp://buildkitd.buildkit.svc.cluster.local:1234 build --progress=plain --frontend=dockerfile.v0 --local context=/workspace/target --local dockerfile=. --output type=image,name=repoin.harbor.net/golang/delivery-report:a40abff4-release-518-fa758fc9,push=true
github.com/moby/buildkit/session/auth/authprovider.(*authProvider).FetchToken.func2
    /src/session/auth/authprovider/authprovider.go:95
github.com/moby/buildkit/session/auth/authprovider.(*authProvider).FetchToken
    /src/session/auth/authprovider/authprovider.go:119
github.com/moby/buildkit/session/auth._Auth_FetchToken_Handler.func1
    /src/session/auth/auth.pb.go:954
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
    /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
    /src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:325
github.com/moby/buildkit/session.filterServer.func1
    /src/session/session.go:164
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
    /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
    /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/session/auth._Auth_FetchToken_Handler
    /src/session/auth/auth.pb.go:956
google.golang.org/grpc.(*Server).processUnaryRPC
    /src/vendor/google.golang.org/grpc/server.go:1340
google.golang.org/grpc.(*Server).handleStream
    /src/vendor/google.golang.org/grpc/server.go:1713
google.golang.org/grpc.(*Server).serveStreams.func1.2
    /src/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1594

Here is the harbor log

[15/May/2023:13:50:14 +0000]:10.249.182.29 - "POST /service/token HTTP/1.1" 405 19 "-" "containerd/1.6.18+unknown" 0.006 0.007 .
[15/May/2023:13:50:14 +0000]:10.249.182.29 - "GET /service/token?scope=repository%3Akubeone-runtime%2Fgolang-runtime%3Apull&service=harbor-registry HTTP/1.1" 200 986 "-" "containerd/1.6.18+unknown" 0.030 0.030 .

Harbor: v2.4.0 Buildkit: v0.11.5

Or how do I use the basic authentication method

[renhao-0518]

this problem occurs frequently in our production，I found that when it timed out, he didn't send a request to harbor at all

[tonistiigi]

I don't see harbor timeout

This looks like a network level timeout. If the packets never reached harbor then it is expected that there is no log.

Not much to go by in here unless there is some reproducer. Seems to be specific to infra between builder and registry.

[tonistiigi]

Or how do I use the basic authentication method

Auth method is chosen by the registry.