http_proxy, https_proxy not honored by Dockerfile ADD instruction?

[dancysoft]

At Wikimedia Foundation we build buildkit releases locally to publish to our own registry. We build using buildkitd (currently v0.15.2). The node that buildkitd runs on does not have direct Internet access and must use a proxy. We pass --opt build-arg:http_proxy="$http_proxy" --opt build-arg:https_proxy="$https_proxy" --opt build-arg:no_proxy="$no_proxy" (with variable values coming from the environment) to buildctl build which generally works well. However, buildkit v0.16's Dockerfile started using instructions like ADD --keep-git-dir=true https://github.com/containers/dnsname.git#v1.3.1 . These instructions are failing when we attempt to build buildkit v0.16:

> [dnsname 2/3] ADD --keep-git-dir=true https://github.com/containers/dnsname.git#v1.3.1 .:
#33 129.4 fatal: unable to access 'https://github.com/containers/dnsname.git/': Failed to connect to github.com port 443 after 129401 ms: Couldn't connect to server

Other instructions which use the Internet work fine, for example:

#19 [cni-plugins 5/7] RUN curl -Ls https://github.com/containernetworking/plugins/releases/download/v1.5.1/cni-plugins-linux-amd64-v1.5.1.tgz | tar xzv

It seems like whatever implements ADD (dockerfile-frontend, presumably) isn't honoring the proxy settings.

The transcript of the failing job is available here: https://gitlab.wikimedia.org/repos/releng/buildkit/-/jobs/361358

[dancysoft]

This may be the same thing as #4643.

[tonistiigi]

@thaJeztah Any memory if this might have been intentional? I think we should use them if set with build-arg.

Note that proxy settings do not apply to FROM as well, but I think there it is more expected as registry config/mirrors is in daemon config.

Atm. you should be able to work around it by setting docker buildx create --driver-opt env.HTTP_PROXY=foo

[dancysoft]

Atm. you should be able to work around it by setting docker buildx create --driver-opt env.HTTP_PROXY=foo

In the original description of this ticket I said we run docker build, but we actually use buildctl build. What would be the corresponding workaround for that client?

[tonistiigi]

In the original description of this ticket I said we run docker build, but we actually use buildctl build. What would be the corresponding workaround for that client?

You need to define the proxy env when you are starting the buildkitd daemon, so depends on how you are doing it. The previous buildx command runs buildkitd in container and --driver-opt defines env for that container.

[thaJeztah]

Any memory if this might have been intentional? I think we should use them if set with build-arg.

Hmm.. no, I don't think this was ever discussed, or a conscious decision.

Note that proxy settings do not apply to FROM as well, but I think there it is more expected as registry config/mirrors is in daemon config.

So, that's actually an interesting point; I'm guessing COPY --from=<some image reference> will also take the daemon proxy settings.

I'm considering what's most logical here, and "least surprise";

• Proxy as configured per-build (through --build-arg)
• Proxy as configured for the daemon environment (networking connections made by the daemon itself)

Out of curiousity, I wondered what the classic builder used, and it looks like the classic builder uses the daemon proxy settings;

Start the daemon with some proxy (I used example.com which will fail when used as proxy);

HTTP_PROXY=http://example.com HTTPS_PROXY=http://example.com dockerd --debug

Try a build with a ADD <github URL>;

echo -e 'FROM scratch\nADD https://github.com/containers/dnsname.git#v1.3.1 .' | DOCKER_BUILDKIT=0 docker build -
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM scratch
 --->
Step 2/2 : ADD https://github.com/containers/dnsname.git#v1.3.1 .
ADD failed: Get "https://github.com/containers/dnsname.git#v1.3.1": Bad Request

Starting the daemon without proxy configured, and trying again;

echo -e 'FROM scratch\nADD https://github.com/containers/dnsname.git#v1.3.1 .' | DOCKER_BUILDKIT=0 docker build -
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM scratch
 --->
Step 2/2 : ADD https://github.com/containers/dnsname.git#v1.3.1 .
Downloading  353.2kB

 ---> a014e8d6b763
Successfully built a014e8d6b763

[thaJeztah]

So, from the above, at least it looks like BuildKit behaves different than the classic builder, which could be considered a regression.

Whether the "build-arg" proxy, or "daemon" proxy should be used is still an interesting topic. The daemon may require a specific proxy (if the daemon is running in a different environment than the client).

But a specific build requiring a specific proxy can be a very valid use-case as well. I'm still considering if we need explicit options to allow the user to decide, because either could be considered valid. (e.g. proxy=host or whatever.

If "per build" proxies are considered for this; wondering if allowing different proxies for git would (should?) impact whether those can be shared between builds? i.e. should a docker build --build-arg <proxy-X> share the same git checkout as a build cloning the same repository but without (or with a different) proxy? I think we always considered proxy to not be affecting build-cache, so maybe that's ok; not sure if this specific situation (git clone) is different in that respect.

[dancysoft]

In the original description of this ticket I said we run docker build, but we actually use buildctl build. What would be the corresponding workaround for that client?

You need to define the proxy env when you are starting the buildkitd daemon, so depends on how you are doing it. The previous buildx command runs buildkitd in container and --driver-opt defines env for that container.

Thank you. We run buildkitd using docker. We pass --env-file /etc/default/buildkitd to docker run and /etc/default/buildkitd has all of the proxy variables set (lower and uppercase forms). I can confirm that the variables are set in the buildkitd container using docker exec <containerid> env:

# docker exec 2c9c1f0d9696 env | egrep -i https?_proxy
http_proxy=http://webproxy:8080
https_proxy=http://webproxy:8080
HTTP_PROXY=http://webproxy:8080
HTTPS_PROXY=http://webproxy:8080

[dancysoft]

Hi @thaJeztah and @tonistiigi . I wanted to confirm that you saw my response to the advice about setting http(s)_proxy for buildkitd itself. We do have those environment variables set correctly for buildkitd but they do not seem to have effect for ADD.

[flpajany]

Hi,

I have the same problem :

# cat Dockerfile
FROM scratch
ADD https://github.com/microsoft/secureboot_objects.git#v1.1.3:PreSignedObjects /

# docker buildx build .
[+] Building 0.1s (3/4)                                                                                                                                                                                                                                                                                                                                         docker-container:local1
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                                                                               0.0s
 => => transferring dockerfile: 133B                                                                                                                                                                                                                                                                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                                  0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                                    0.0s
 => ERROR [1/1] ADD https://github.com/microsoft/secureboot_objects.git#v1.1.3:PreSignedObjects /                                                                                                                                                                                                                                                                                  0.0s
------
 > [1/1] ADD https://github.com/microsoft/secureboot_objects.git#v1.1.3:PreSignedObjects /:
0.027 fatal: unable to access 'https://github.com/microsoft/secureboot_objects.git/': Could not resolve host: github.com
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
ERROR: failed to solve: failed to load cache key: failed to fetch remote https://github.com/microsoft/secureboot_objects.git: git stderr:
fatal: unable to access 'https://github.com/microsoft/secureboot_objects.git/': Could not resolve host: github.com
: exit status 128

when I used a curl command :

# cat Dockerfile
FROM alpine/curl
RUN curl -vL  https://github.com/microsoft/secureboot_objects.git#v1.1.3:PreSignedObjects

# docker buildx build .
[+] Building 0.5s (5/5) FINISHED                                                                                                                                                                                                                                                                                                                                docker-container:local1
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                                                                               0.0s
 => => transferring dockerfile: 146B                                                                                                                                                                                                                                                                                                                                               0.0s
 => [internal] load metadata for docker.io/alpine/curl:latest                                                                                                                                                                                                                                                                                                                      0.4s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                                  0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                                    0.0s
 => [1/2] FROM docker.io/alpine/curl:latest@sha256:13f44396bcf020589646a3bafa74b8096b8b97b5714832c322db29af628c9889                                                                                                                                                                                                                                                                0.0s
 => => resolve docker.io/alpine/curl:latest@sha256:13f44396bcf020589646a3bafa74b8096b8b97b5714832c322db29af628c9889                                                                                                                                                                                                                                                                0.0s
 => CACHED [2/2] RUN curl -vL  https://github.com/microsoft/secureboot_objects.git#v1.1.3:PreSignedObjects

Like in the original post, I am setting my proxy configuration in environment, in docker daemon configuration, in driver-opt env.http_proxy... configuration for docker-container driver with my local buildx builder instance.