[WIP] Support Buildkit in Swarm

[dperny]

Updates Swarm to support Buildkit builds.

Change has two parts:

• The first part updates the vendoring of github.com/docker/docker. To make this work, changes needed to be made to parts of Swarm.
• The second part adds support for the new behavior of the /build and /session endpoints.

It may be helpful to review this PR by looking at the diffs for each commit separately.

Based on work provided to me by @tonistiigi.

This PR is marked WIP because I am having a spot of trouble running the existing tests and getting Swarm running in my environment. However, the code itself should be ready for review.

[GordonTheTurtle]

Please sign your commits following these rules: https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work The easiest way to do this is to amend the last commit:

$ git clone -b "buildkit-support" git@github.com:dperny/swarm.git somewhere
$ cd somewhere
$ git rebase -i HEAD~842354281472
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

[dperny]

sshhhhhh i know gordon i know

[tonistiigi]

@dperny maybe we can split & merge the vendoring update separately to make the new parts easier to review

[dperny]

sure, let me do some gitmagic

[dperny]

@tonistiigi: split vendoring into a separate PR (#2920). To review just the changes in this PR, you can use this link: https://github.com/docker/swarm/pull/2919/commits/fb5c38ba00f77d54f6e86c9807aff5a517b34279

[anshulpundir]

also ping @dani-docker for a review

[tonistiigi]

@dperny First issue is that the reported API version is still the old one. This is what I see with this PR:

Client: Docker Engine - Community
 Version:           18.09.1
 API version:       1.30 (downgraded from 1.39)
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:33:22 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          swarm/1.2.9
  API version:      1.30 (minimum version )
  Go version:       go1.7.1
  Git commit:       fb5c38b
  Built:            2019-01-31T00:45:40+0000
  OS/Arch:          linux/amd64
  Experimental:     false

While the server version should also be 1.39 . WIth the old version cli refuses to even try the new endpoints.

[dperny]

Oh... Now that is going to be a problem... If we're advertising the latest API version, I'm going to need to go updating the swarm api to make sure it conforms... I'm not sure that it does...

[tonistiigi]

Swarm component should be quite transparent and the real updated logic is in the upgraded docker backend. If there are some other new endpoints they need new forwarding as well though. But as new endpoints almost always directly map to the CLI commands it means that without the support a new client is broken in the EE experience for new commands/flags and should be fixed anyway.

[tonistiigi]

https://github.com/tonistiigi/swarm/commits/build-endpoints2 has the patches that fix this.

Far from perfect, eg. builder prune does not work but all system df, image prune etc. commands give just 404 to me so didn't bother to fix that. The build endpoints itself work now. I made a small change deferring session cleanup in case we want to support use cases that use a long-running session with multiple build requests in the future.

Also did another vendoring and switched to 1.40 already so that we don't need to update again to pick up 19.03 functionality.

@tiborvass @dhiltgen

[dperny]

@tonistiigi adding your patch seems to make it work for me, but after the first build it's getting stuck on "build already started"

[tonistiigi]

@dperny Pushed extra commit. This would have happened on building without buildkit. To test buildkit set DOCKER_BUILDKIT=1 on the cli and make sure you see buildkit output.