Swarm is Broken on Docker CE after 17.05 - no services are scheduled.

alexellis commented 7 years ago

Issue:

Docker Swarm is broken in Docker CE 17.07

You cannot schedule ARM containers.

Work needed:

We need to rebuild Docker with the follow PR added to see if it resolves the issue.

https://github.com/moby/moby/pull/34021

Git clone moby
Checkout Moby with the correct 17.07 tag and then run make in the moby directory.
You might want to add a swap file for building too

Update: We've tested the issue on ARMv7 but have to repeat for ARMv6 on a Pi Zero. Bootstrap here - https://github.com/alexellis/docker-arm/issues/19

alexellis commented 7 years ago

Get the change via @nishanttotla

$ git fetch origin pull/34021/head:dont-set-architecture-constraint
$ git checkout dont-set-architecture-constraint

Open a new tux session and make sure you have tuned-up your RPi - #16.

Now invoke a build via make build

alexellis commented 7 years ago

Folks you will need a Docker version here to start with, there's 17.06 in the stretch / edge repo - follow instructions here - https://docs.docker.com/engine/installation/linux/docker-ce/debian/

Change stable to edge in the apt command. Follow instructions for armhf

alexellis commented 7 years ago

This is as far as I got with make tgz

Successfully tagged docker-dev:dont-set-architecture-constraint                                                                  
docker run --rm -i --privileged  -e DOCKER_CROSSPLATFORMS -e BUILD_APT_MIRROR -e BUILDFLAGS -e KEEPBUNDLE -e DOCKER_BUILD_ARGS -e
 DOCKER_BUILD_GOGC -e DOCKER_BUILD_PKGS -e DOCKER_BASH_COMPLETION_PATH -e DOCKER_CLI_PATH -e DOCKER_DEBUG -e DOCKER_EXPERIMENTAL-
e DOCKER_GITCOMMIT -e DOCKER_GRAPHDRIVER -e DOCKER_INCREMENTAL_BINARY -e DOCKER_PORT -e DOCKER_REMAP_ROOT -e DOCKER_STORAGE_OPTS 
-e DOCKER_USERLANDPROXY -e TESTDIRS -e TESTFLAGS -e TIMEOUT -e HTTP_PROXY -e HTTPS_PROXY -e NO_PROXY -e http_proxy -e https_proxy
 -e no_proxy -v "/home/pi/docker/bundles:/go/src/github.com/docker/docker/bundles" -v /home/pi/docker/.git:/go/src/github.com/doc
ker/docker/.git -v "dockerdev-go-pkg-cache-gopath:/go/pkg" -v "dockerdev-go-pkg-cache-goroot-linux_amd64:/usr/local/go/pkg/linux_
amd64" -v "dockerdev-go-pkg-cache-goroot-linux_amd64_netgo:/usr/local/go/pkg/linux_amd64_netgo"    -t "docker-dev:dont-set-archit
ecture-constraint" hack/make.sh dynbinary binary cross tgz                                                                       

---> Making bundle: dynbinary (in bundles/17.06.0-dev/dynbinary)                                                                 
Building: bundles/17.06.0-dev/dynbinary-daemon/dockerd-17.06.0-dev                                                               
Created binary: bundles/17.06.0-dev/dynbinary-daemon/dockerd-17.06.0-dev                                                         

---> Making bundle: binary (in bundles/17.06.0-dev/binary)                                                                       
Building: bundles/17.06.0-dev/binary-daemon/dockerd-17.06.0-dev                                                                  
Created binary: bundles/17.06.0-dev/binary-daemon/dockerd-17.06.0-dev                                                            
Copying nested executables into bundles/17.06.0-dev/binary-daemon                                                                

---> Making bundle: cross (in bundles/17.06.0-dev/cross)                                                                         
Created symlinks: bundles/17.06.0-dev/cross/linux/arm/docker-containerd bundles/17.06.0-dev/cross/linux/arm/docker-containerd-ctr
 bundles/17.06.0-dev/cross/linux/arm/docker-containerd-ctr.md5 bundles/17.06.0-dev/cross/linux/arm/docker-containerd-ctr.sha256 b
undles/17.06.0-dev/cross/linux/arm/docker-containerd-shim bundles/17.06.0-dev/cross/linux/arm/docker-containerd-shim.md5 bundles/
17.06.0-dev/cross/linux/arm/docker-containerd-shim.sha256 bundles/17.06.0-dev/cross/linux/arm/docker-containerd.md5 bundles/17.06
.0-dev/cross/linux/arm/docker-containerd.sha256 bundles/17.06.0-dev/cross/linux/arm/docker-init bundles/17.06.0-dev/cross/linux/a
rm/docker-init.md5 bundles/17.06.0-dev/cross/linux/arm/docker-init.sha256 bundles/17.06.0-dev/cross/linux/arm/docker-proxy bundle
s/17.06.0-dev/cross/linux/arm/docker-proxy.md5 bundles/17.06.0-dev/cross/linux/arm/docker-proxy.sha256 bundles/17.06.0-dev/cross/
linux/arm/docker-runc bundles/17.06.0-dev/cross/linux/arm/docker-runc.md5 bundles/17.06.0-dev/cross/linux/arm/docker-runc.sha256b
undles/17.06.0-dev/cross/linux/arm/dockerd bundles/17.06.0-dev/cross/linux/arm/dockerd-17.06.0-dev bundles/17.06.0-dev/cross/linu
x/arm/dockerd-17.06.0-dev.md5 bundles/17.06.0-dev/cross/linux/arm/dockerd-17.06.0-dev.sha256                                     
Cross building: bundles/17.06.0-dev/cross/linux/amd64                                                                            
Building: bundles/17.06.0-dev/cross/linux/amd64/dockerd-17.06.0-dev                                                              
daemon/graphdriver/overlay2/overlay.go:24:2: no buildable Go source files in /go/src/github.com/docker/docker/daemon/graphdriver/
quota                                                                                                                            
daemon/graphdriver/devmapper/deviceset.go:28:2: no buildable Go source files in /go/src/github.com/docker/docker/pkg/loopback    
daemon/apparmor_default.go:9:2: no buildable Go source files in /go/src/github.com/docker/docker/vendor/github.com/opencontainers
/runc/libcontainer/apparmor                                                                                                      
profiles/seccomp/seccomp.go:13:2: no buildable Go source files in /go/src/github.com/docker/docker/vendor/github.com/seccomp/libs
eccomp-golang                                                                                                                    
Makefile:164: recipe for target 'tgz' failed                                                                                     
make: *** [tgz] Error 1

alexellis commented 7 years ago

make deb with docker/docker repo:

Created binary: bundles/17.06.0-dev/dynbinary-daemon/dockerd-17.06.0-dev                                                         

# ./man/md2man-all.sh runs outside the build container (if at all), since we don't have go-md2man here                           
make[1]: Leaving directory '/usr/src/docker'                                                                                     
   debian/rules override_dh_auto_test                                                                                            
make[1]: Entering directory '/usr/src/docker'                                                                                    
./bundles/17.06.0-dev/dynbinary-daemon/dockerd -v                                                                                
Docker version 17.06.0-dev, build 5fa6df34d                                                                                      
make[1]: Leaving directory '/usr/src/docker'                                                                                     
 debian/rules binary                                                                                                             
dh binary --with=bash-completion --with=systemd                                                                                  
   dh_testroot                                                                                                                   
   dh_prep                                                                                                                       
   debian/rules override_dh_auto_install                                                                                         
make[1]: Entering directory '/usr/src/docker'                                                                                    
mkdir -p debian/docker-engine/usr/bin                                                                                            
cp -aT "$(readlink -f bundles/17.06.0-dev/dynbinary-daemon/dockerd)" debian/docker-engine/usr/bin/dockerd                        
cp -aT /usr/local/bin/docker-proxy debian/docker-engine/usr/bin/docker-proxy                                                     
cp -aT /usr/local/bin/docker-containerd debian/docker-engine/usr/bin/docker-containerd                                           
cp -aT /usr/local/bin/docker-containerd-shim debian/docker-engine/usr/bin/docker-containerd-shim                                 
cp -aT /usr/local/bin/docker-containerd-ctr debian/docker-engine/usr/bin/docker-containerd-ctr                                   
cp -aT /usr/local/bin/docker-runc debian/docker-engine/usr/bin/docker-runc                                                       
cp -aT /usr/local/bin/docker-init debian/docker-engine/usr/bin/docker-init                                                       
mkdir -p debian/docker-engine/usr/lib/docker                                                                                     
make[1]: Leaving directory '/usr/src/docker'                                                                                     
   debian/rules override_dh_install                                                                                              
make[1]: Entering directory '/usr/src/docker'                                                                                    
dh_install                                                                                                                       
cp: cannot stat 'debian/tmp/contrib/completion/fish/docker.fish': No such file or directory                                      
dh_install: cp -a debian/tmp/contrib/completion/fish/docker.fish debian/docker-engine/usr/share/fish/vendor_completions.d// retur
ned exit code 1                                                                                                                  
debian/rules:46: recipe for target 'override_dh_install' failed                                                                  
make[1]: Leaving directory '/usr/src/docker'                                                                                     
make[1]: *** [override_dh_install] Error 2                                                                                       
debian/rules:53: recipe for target 'binary' failed                                                                               
make: *** [binary] Error 2                                                                                                       
dpkg-buildpackage: error: debian/rules binary gave error exit status 2                                                           
The command '/bin/sh -c dpkg-buildpackage -uc -us -I.git' returned a non-zero code: 2                                            
---> Making bundle: .integration-daemon-stop (in bundles/17.06.0-dev/build-deb)                                                  
+++++ cat bundles/17.06.0-dev/build-deb/docker.pid                                                                               
++++ kill 3812                                                                                                                   
Makefile:130: recipe for target 'deb' failed                                                                                     
make: *** [deb] Error 1

alexellis commented 7 years ago

Using the moby/moby PR and make binary I was able to schedule a swarm service:

$ sudo systemctl stop docker
$ sudo cp bundles/latest/binary-daemon/* /usr/bin/
$ sudo systemctl start docker
$ docker version                                                                 Client:
 Version:      17.06.1-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   874a737
 Built:        Thu Aug 17 23:02:18 2017
 OS/Arch:      linux/arm

Server:
 Version:      17.06.0-dev
 API version:  1.31 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   5fa6df34d
 Built:        Sun Aug 20 14:29:31 2017
 OS/Arch:      linux/arm
 Experimental: false

$ docker service create --env fprocess=/bin/cat --publish 8080:8080 functions/alpine:latest-armhf
$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE
         PORTS
8ez9gk3dl0ux        eloquent_snyder     replicated          1/1                 functions/alpine:latest-armhf   *:8080->8080/tcp

$ curl -4 localhost:8080 -d "test"
test

alexellis commented 7 years ago

You can run this as a test work-load before and after:

$ docker swarm init && \
   git clone https://github.com/alexellis/faas && \
   cd faas && \
   ./deploy_stack.armhf.sh

The API gateway for FaaS will be on http://ip.address:8080/ find the ip with ifconfig

Related blog post to give it a thorough test - https://blog.alexellis.io/your-serverless-raspberry-pi-cluster/

bassoman commented 7 years ago

Just got my Pi 3 cluster working with your article on setting up a cluster. I too had the issue of not seeing the Replica column show 1/1 properly. Upgraded my manager Pi 3 OS via the following page and it all works great now! Just do the 'Update to Debian 8 (Jessie)' paragraph and it will all start working properly. There seems to be no need to upgrade the worker nodes...

https://github.com/umiddelb/armhf/wiki/Get-Docker-up-and-running-on-the-RaspberryPi-(ARMv6)-in-four-steps-(Wheezy)

Here are the commands:

sudo sed -i 's/wheezy/jessie/' /etc/apt/sources.list sudo sed -i 's/wheezy/jessie/' /etc/apt/sources.list.d/raspi.list sudo apt-get update && sudo apt-get -y upgrade # answer 'y' to upcoming questions sudo apt-get -y dist-upgrade # answer 'y' to upcoming questions sudo init 6 sudo apt-get -y autoremove sudo apt-get -y purge $(dpkg -l | awk '/^rc/ { print $2 }') sudo init 6

alexellis / docker-arm

Swarm is Broken on Docker CE after 17.05 - no services are scheduled. #14