The builder hangs when you try to use a custom go image (`1.17-buster` in my case)

[laurentsenta]

Trying to testground run single a test with a different base image (go1.17 for current compact with libp2p, and others), it hangs when retrieving the image:

Apr 25 14:23:19.042508  INFO   go build cache enabled; checking if build cache image exists    {"cache_image": "tg-gobuildcache-ping"}
Apr 25 14:23:19.056698  INFO   build cache image not found {"cache_image": "tg-gobuildcache-ping"}
Step 1/22 : ARG BUILD_BASE_IMAGE
Step 2/22 : ARG RUNTIME_IMAGE=busybox:1.31.1-glibc
Step 3/22 : FROM ${BUILD_BASE_IMAGE} AS builder
1.17-buster: 
85bed84afb9a: 
5fdd409f4b2b: 
fa3069e6cecf: 
4ee16f45eff9: 
21b2aacb9bcb: 
97c4483c1372: 
9ba06393f25d: 
97c4483c1372: 
9ba06393f25d: 
21b2aacb9bcb: 

I've seen this reproduce often, on CI, and on this dedicated server. After 10 minutes the build fails with "context deadline exceeded", Re-running the same command, it will succeed, as if the download succeeded the first time but wasn't signaled.

Second run:

Step 1/22 : ARG BUILD_BASE_IMAGE
Step 2/22 : ARG RUNTIME_IMAGE=busybox:1.31.1-glibc
Step 3/22 : FROM ${BUILD_BASE_IMAGE} AS builder
 ---> bf6c4a990662
Step 4/22 : ENV PLAN_DIR /plan
 ---> Using cache
 ---> 80e8789d1abf
Step 5/22 : ENV SDK_DIR /sdk
 ---> Using cache
 ---> 0064c3f2a3a6
Step 6/22 : RUN rm -rf ${PLAN_DIR} ${SDK_DIR} /testground_dep_list
 ---> Using cache
 ---> f26a11157d11
Step 7/22 : ARG TESTPLAN_EXEC_PKG="."
 ---> Using cache
 ---> 1c39057fc87f
Step 8/22 : ARG GO_PROXY=direct
 ---> Using cache
 ---> 9351235bf76b
Step 9/22 : ARG BUILD_TAGS
 ---> Using cache
 ---> 1fb5e09ca408
Step 10/22 : ENV TESTPLAN_EXEC_PKG ${TESTPLAN_EXEC_PKG}
 ---> Using cache
 ---> ffb3d79348bd
Step 11/22 : ENV GOCACHE /go/cache
 ---> Using cache
 ---> 7dbba2f2dcc2
Step 12/22 : COPY /plan/go.mod ${PLAN_DIR}/go.mod
 ---> Using cache
 ---> fafd1dfed4cf
Step 13/22 : RUN echo "Using go proxy: ${GO_PROXY}"     && cd ${PLAN_DIR}     && go env -w GOPROXY="${GO_PROXY}"     && go mod download
 ---> Running in a14b5f0bee00
Using go proxy: http://testground-goproxy:8081