docker is unable to build private git repository with ssh access using buildkit

stalb commented 5 years ago

Problem:

If you have a private git repository you access using ssh key, you are not able to build it using buildkit.

export DOCKER_BUILDKIT=1 ; docker build git@github.com:xxxx/yyyy.git fails with permission denied, Could not read from remote repository...

Of course the normal build command is working as expected: export DOCKER_BUILDKIT=0 ; docker build git@github.com:xxxx/yyyy.git

Expected behavior:

export DOCKER_BUILDKIT=1 ; docker build git@github.com:xxxx/yyyy.git should be able to use ssh key or ssh-agent in order to download the private project and then build the project.

Docker version:

Client: Docker Engine - Community
 Version:           19.03.0-beta5
 API version:       1.40
 Go version:        go1.12.4
 Git commit:        4a18bf42fb
 Built:             Wed May 29 22:59:42 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.0-beta5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.4
  Git commit:       4a18bf42fb
  Built:            Wed May 29 22:58:17 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Bluesboy commented 4 years ago

I'm also experiencing the same issue but with Concourse CI (vito/oci-build-task). @stalb do you found out workaround?

stalb commented 4 years ago

@Bluesboy no I just don't use Buildkit for this repositories.

tonistiigi commented 4 years ago

The workaround is to do git clone .. x && docker build x. In fact this is exactly what the legacy mode does before the actual builder is invoked.

We do plan to add proper support for this https://github.com/moby/buildkit/pull/1048

Bluesboy commented 4 years ago

It seems to me that error happened because buildkit doesn't interpolates environment variables from host inside build environment. However if I provide variables with ARG directive in Dockerfile and then specify them in build arguments everything works fine. Hope that'll help somebody.

tonistiigi commented 3 years ago

@alexcb I guess this has been addressed but can you update documentation for the extra ~fields~ flags required to forward ssh.

alexcb commented 3 years ago

@tonistiigi I think I might be missing something here, which struct/fields needs documenting?

I'm surprised that this code isn't being triggered automatically: https://github.com/moby/buildkit/blob/master/client/llb/source.go#L292-L310

tonistiigi commented 3 years ago

@alexcb I'd assume at least --ssh default is needed. Although you might be correct that if the context path is already an ssh git URL we could maybe add it automatically on the cli side.

alexcb commented 3 years ago

I just compiled a new version of buildx, and if I specify a socket it works:

./bin/buildx build --ssh default=$(echo $SSH_AUTH_SOCK) git@github.com:xxx/yyy.git

however when I run it without any --ssh option it passes an empty slice to https://github.com/docker/buildx/blob/master/build/ssh.go#L11

I wonder if something like this would be a reasonable fix? https://github.com/docker/buildx/pull/577

edit: this is a simpler fix: https://github.com/docker/buildx/pull/578

playground commented 3 years ago

I'm having the save error. Anyone got this working with buildkit?

tonistiigi commented 2 years ago

https://github.com/docker/buildx/pull/581

moby / buildkit

docker is unable to build private git repository with ssh access using buildkit #1035