docker-machine

[kshpytsya]

I am trying to get kubler run in a remote docker. Is it supposed to work? Note that I have repeated those steps several times within a day interval, and the failure is 100% reproducible. On the other hand, running it on local docker on Ubuntu in a VirtualBox seems to work ok (build is currently in progress far beyond the failure point reported here for remote docker). My dev machine is running Gentoo with custom kernel with which docker has always had various problems, so I chose to go with docker-machine/boot2docker/VirtualBox way. Moreover, the paranoiac part of me believes this to be more secure.

uken@kyrylo-thinkpad ~/src-foreign $ git clone https://github.com/edannenberg/kubler.git
uken@kyrylo-thinkpad ~/src-foreign $ cd kubler/
uken@kyrylo-thinkpad ~/src-foreign/kubler (master) $ docker-machine create docker2                                                               Running pre-create checks...
Creating machine...
(docker2) Copying /home/uken/.docker/machine/cache/boot2docker.iso to /home/uken/.docker/machine/machines/docker2/boot2docker.iso...
(docker2) Creating VirtualBox VM...
(docker2) Creating SSH key...
(docker2) Starting the VM...
(docker2) Check network to re-create if needed...
(docker2) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env docker2
uken@kyrylo-thinkpad ~/src-foreign/kubler (master) $ eval $(docker-machine env docker2)
uken@kyrylo-thinkpad ~/src-foreign/kubler (master) $ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
ca4f61b1923c: Pull complete
Digest: sha256:66ef312bbac49c39a89aa9bcc3cb4f3c9e7de3788c944158df3ee0176d32b751
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://cloud.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

uken@kyrylo-thinkpad ~/src-foreign/kubler (master) $ ./kubler.sh build kubler/glibc
*** generate build order
--> required engines:    docker
--> required stage3:     stage3-amd64-hardened+nomultilib stage3-amd64-musl-hardened
--> required builders:   kubler/bob kubler/bob-musl
--> build sequence:      kubler/busybox kubler/glibc
*** gogo!
--2018-02-09 13:38:32--  http://distfiles.gentoo.org/snapshots/portage-latest.tar.xz
Resolving distfiles.gentoo.org... 216.165.129.135, 137.226.34.46, 64.50.236.52, ...
Connecting to distfiles.gentoo.org|216.165.129.135|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 48046052 (46M) [application/x-tar]
Saving to: ‘/home/uken/src-foreign/kubler/tmp/downloads/portage-20180209.tar.xz’

/home/uken/src-foreign/kubler/tmp/do 100%[====================================================================>]  45.82M  10.4MB/s    in 8.8s

2018-02-09 13:38:41 (5.19 MB/s) - ‘/home/uken/src-foreign/kubler/tmp/downloads/portage-20180209.tar.xz’ saved [48046052/48046052]

--> bootstrap kubler-gentoo/portage
Sending build context to Docker daemon  48.05MB
Step 1/5 : FROM busybox:latest
latest: Pulling from library/busybox
57310166fe88: Pull complete
Digest: sha256:1669a6aa7350e1cdd28f972ddad5aceba2912f589f19a090ac75b7083da748db
Status: Downloaded newer image for busybox:latest
 ---> 5b0d59026729
Step 2/5 : LABEL maintainer Erik Dannenberg <erik.dannenberg@xtrade-gmbh.de>
 ---> Running in 5a39323612d9
Removing intermediate container 5a39323612d9
 ---> 1a1faccbbfc2
Step 3/5 : COPY portage-20180209.tar.xz /
 ---> 7cfa4d02229c
Step 4/5 : RUN set -x &&     mkdir -p /var/sync &&     xzcat /portage-20180209.tar.xz | tar -xf - -C /var/sync &&     mkdir -p /var/sync/portage/metadata &&     rm /portage-20180209.tar.xz
 ---> Running in f0f62388ae18
+ mkdir -p /var/sync
+ tar -xf - -C /var/sync
+ xzcat /portage-20180209.tar.xz
+ mkdir -p /var/sync/portage/metadata
+ rm /portage-20180209.tar.xz
Removing intermediate container f0f62388ae18
 ---> b237ca0ecd19
Step 5/5 : VOLUME /var/sync /var/lib/layman /var/cache/eix
 ---> Running in 27de96594997
Removing intermediate container 27de96594997
 ---> 5f0b78b37181
Successfully built 5f0b78b37181
Successfully tagged kubler-gentoo/portage:20180209
--> create portage container, this may take a few moments..
--> import kubler-gentoo/stage3-amd64-hardened-nomultilib:20180104T214501Z using stage3-amd64-hardened+nomultilib-20180206T214502Z.tar.xz
sha256:a9a1951cf327da92650ff443770d67177a792899b622b886d5e654dfda1dba65
tag kubler-gentoo/stage3-amd64-hardened-nomultilib:latest
--> build image kubler/bob-core
--> phase 2: build kubler/bob-core:20180127
Sending build context to Docker daemon  43.01kB
Step 1/10 : FROM kubler-gentoo/stage3-amd64-hardened-nomultilib
 ---> a9a1951cf327
Step 2/10 : LABEL maintainer Erik Dannenberg <erik.dannenberg@xtrade-gmbh.de>
 ---> Running in 0fd092e044bc
Removing intermediate container 0fd092e044bc
 ---> 97b6055754be
Step 3/10 : RUN set -x &&     echo 'GENTOO_MIRRORS="http://distfiles.gentoo.org/"' >> /etc/portage/make.conf &&     mkdir -p /etc/portage/repos.conf /usr/portage &&     sed -e 's|^sync-uri =.*|sync-uri = ${BOB_SYNC_URI}|'         -e 's|^sync-type =.*|sync-type = ${BOB_SYNC_TYPE}|'         /usr/share/portage/config/repos.conf > /etc/portage/repos.conf/gentoo.conf &&     chown -R portage:portage /usr/portage &&     mkdir -p /etc/portage/profile
 ---> Running in ac7fd4f86583
+ echo 'GENTOO_MIRRORS="http://distfiles.gentoo.org/"'
+ mkdir -p /etc/portage/repos.conf /usr/portage
+ sed -e 's|^sync-uri =.*|sync-uri = ${BOB_SYNC_URI}|' -e 's|^sync-type =.*|sync-type = ${BOB_SYNC_TYPE}|' /usr/share/portage/config/repos.conf
+ chown -R portage:portage /usr/portage
+ mkdir -p /etc/portage/profile
Removing intermediate container ac7fd4f86583
 ---> 9c4415236d44
Step 4/10 : ENV DEF_CHOST="x86_64-pc-linux-gnu"     DEF_CFLAGS="-mtune=generic -O2 -pipe"     DEF_CXXFLAGS="-mtune=generic -O2 -pipe"     DEF_BUILDER_CHOST="x86_64-pc-linux-gnu"     DEF_BUILDER_CFLAGS="-mtune=generic -O2 -pipe"     DEF_BUILDER_CXXFLAGS="-mtune=generic -O2 -pipe"     PKGDIR="/packages/x86_64-pc-linux-gnu"
 ---> Running in 9bb8a389195c
Removing intermediate container 9bb8a389195c
 ---> ac362f15cc3d
Step 5/10 : COPY etc/ /etc/
 ---> 81c3e7d01945
Step 6/10 : COPY build-root.sh /usr/local/bin/kubler-build-root
 ---> c3ed0b70a168
Step 7/10 : COPY bashrc.sh /root/.bashrc
 ---> 29cdec940998
Step 8/10 : COPY acserver-push.sh /usr/local/bin/acserver-push
 ---> 837ee1253c7a
Step 9/10 : COPY portage-git-sync.sh /usr/local/bin/portage-git-sync
 ---> 9bbd30c37f8c
Step 10/10 : CMD ["/bin/bash"]
 ---> Running in 8cc4a3bbc9d1
Removing intermediate container 8cc4a3bbc9d1
 ---> 23be3862a195
Successfully built 23be3862a195
Successfully tagged kubler/bob-core:20180127
tag kubler/bob-core:latest
--> build image kubler/bob
--> phase 1: building root fs
using kubler/bob-core:20180127
commit kubler-bob-20640-17204 as kubler/bob:20180127
sha256:58b4d0b84c320de6c747d78feda75087634205c48322d97322209031f00b7f9e
kubler-bob-20640-17204
tag kubler/bob:latest
--> phase 2: build kubler/bob:20180127
Sending build context to Docker daemon   7.68kB
Step 1/3 : FROM kubler/bob
 ---> 58b4d0b84c32
Step 2/3 : LABEL maintainer Erik Dannenberg <erik.dannenberg@xtrade-gmbh.de>
 ---> Running in a096e655f680
Removing intermediate container a096e655f680
 ---> 27d25df0808a
Step 3/3 : CMD ["/bin/bash"]
 ---> Running in 43627985943a
Removing intermediate container 43627985943a
 ---> 42a0b895fa0b
Successfully built 42a0b895fa0b
Successfully tagged kubler/bob:20180127
tag kubler/bob:latest
GPG verification not supported for experimental stage3 tar balls, only checking SHA512
--> import kubler-gentoo/stage3-amd64-musl-hardened:20180106 using stage3-amd64-musl-hardened-20180204.tar.bz2
sha256:1a3a881149dda1ff22199ace40bbc001b199ede7996e677c94b0f47b5e3fb371
tag kubler-gentoo/stage3-amd64-musl-hardened:latest
--> build image kubler/bob-musl-core
--> phase 2: build kubler/bob-musl-core:20180127
Sending build context to Docker daemon  43.01kB
Step 1/10 : FROM kubler-gentoo/stage3-amd64-musl-hardened
 ---> 1a3a881149dd
Step 2/10 : LABEL maintainer Erik Dannenberg <erik.dannenberg@xtrade-gmbh.de>
 ---> Running in 393aa7503557
Removing intermediate container 393aa7503557
 ---> 650dc4a0b886
Step 3/10 : RUN set -x &&     echo 'GENTOO_MIRRORS="http://distfiles.gentoo.org/"' >> /etc/portage/make.conf &&     mkdir -p /etc/portage/repos.conf /usr/portage &&     sed -e 's|^sync-uri =.*|sync-uri = ${BOB_SYNC_URI}|'         -e 's|^sync-type =.*|sync-type = ${BOB_SYNC_TYPE}|'         /usr/share/portage/config/repos.conf > /etc/portage/repos.conf/gentoo.conf &&     chown -R portage:portage /usr/portage &&     mkdir -p /etc/portage/profile
 ---> Running in 234c62d48969
+ echo 'GENTOO_MIRRORS="http://distfiles.gentoo.org/"'
+ mkdir -p /etc/portage/repos.conf /usr/portage
+ sed -e 's|^sync-uri =.*|sync-uri = ${BOB_SYNC_URI}|' -e 's|^sync-type =.*|sync-type = ${BOB_SYNC_TYPE}|' /usr/share/portage/config/repos.conf
+ chown -R portage:portage /usr/portage
+ mkdir -p /etc/portage/profile
Removing intermediate container 234c62d48969
 ---> ccc68ca062dc
Step 4/10 : ENV DEF_CHOST="x86_64-gentoo-linux-musl"     DEF_CFLAGS="-mtune=generic -O2 -pipe"     DEF_CXXFLAGS="-mtune=generic -O2 -pipe"     DEF_BUILDER_CHOST="x86_64-gentoo-linux-musl"     DEF_BUILDER_CFLAGS="-mtune=generic -O2 -pipe"     DEF_BUILDER_CXXFLAGS="-mtune=generic -O2 -pipe"     PKGDIR="/packages/x86_64-gentoo-linux-musl"
 ---> Running in e77a5479edd6
Removing intermediate container e77a5479edd6
 ---> a992fa472d1a
Step 5/10 : COPY etc/ /etc/
 ---> 84516f627d7d
Step 6/10 : COPY build-root.sh /usr/local/bin/kubler-build-root
 ---> 4f79e16305dd
Step 7/10 : COPY bashrc.sh /root/.bashrc
 ---> 46372b2f1df2
Step 8/10 : COPY acserver-push.sh /usr/local/bin/acserver-push
 ---> 6b098e261d66
Step 9/10 : COPY portage-git-sync.sh /usr/local/bin/portage-git-sync
 ---> 323420d6bb6c
Step 10/10 : CMD ["/bin/bash"]
 ---> Running in 9110a86fed04
Removing intermediate container 9110a86fed04
 ---> 574ba87c48a8
Successfully built 574ba87c48a8
Successfully tagged kubler/bob-musl-core:20180127
tag kubler/bob-musl-core:latest
--> build image kubler/bob-musl
--> phase 1: building root fs
using kubler/bob-musl-core:20180127
commit kubler-bob-musl-20640-18498 as kubler/bob-musl:20180127
sha256:9a0d1bd2175118b79316229a3ee2a889c6dc16feb5a7d2459c1d359982a827ff
kubler-bob-musl-20640-18498
tag kubler/bob-musl:latest
--> phase 2: build kubler/bob-musl:20180127
Sending build context to Docker daemon  10.75kB
Step 1/3 : FROM kubler/bob-musl
 ---> 9a0d1bd21751
Step 2/3 : LABEL maintainer Erik Dannenberg <erik.dannenberg@xtrade-gmbh.de>
 ---> Running in 8f208e36089e
Removing intermediate container 8f208e36089e
 ---> 68e5864ad87d
Step 3/3 : CMD ["/bin/bash"]
 ---> Running in e3db05de2c06
Removing intermediate container e3db05de2c06
 ---> bb7b9ff71059
Successfully built bb7b9ff71059
Successfully tagged kubler/bob-musl:20180127
tag kubler/bob-musl:latest
--> build image kubler/busybox
--> phase 1: building root fs
using kubler/bob-musl:20180127
commit kubler-busybox-20640-31396 as kubler/bob-musl-busybox:20180127
sha256:b4b825a626b5838bb8876a11594174158bbf60333baccea4c6652a8d9ab720aa
kubler-busybox-20640-31396
tag kubler/bob-musl-busybox:latest
--> phase 2: build kubler/busybox:20180127
Sending build context to Docker daemon   7.68kB
Step 1/4 : FROM scratch
 --->
Step 2/4 : LABEL maintainer Erik Dannenberg <erik.dannenberg@xtrade-gmbh.de>
 ---> Running in 48f1759aee9f
Removing intermediate container 48f1759aee9f
 ---> 28c688dd0d41
Step 3/4 : ADD rootfs.tar /
ADD failed: stat /mnt/sda1/var/lib/docker/tmp/docker-builder485144939/rootfs.tar: no such file or directory
fatal: failed to build kubler/images/busybox

[edannenberg]

I am trying to get kubler run in a remote docker. Is it supposed to work?

Sorry but using Kubler with a remote Docker daemon is not an option, at least not with some tinkering. The reason for that is that the first phase of the build process writes a rootfs.tar to the host directory, which is then used later by the second phase. With a remote Docker daemon this will not work as expected as the file is written on the remote Docker host and not your local box.

Docker for Mac/Windows work around that problem by transparently mounting (parts of) the local host file system into the "remote" Docker VM on your host. So using sshfs or similiar to mount the Kubler dir on your Docker remote host should do the trick. Note that depending on the images you are building the rootfs.tar may get rather large, so probably best to have the remote host in your LAN.

My dev machine is running Gentoo with custom kernel with which docker has always had various problems, so I chose to go with docker-machine/boot2docker/VirtualBox way.

Just in case you missed it, the Gentoo Docker ebuild checks and lists all missing kernel options upon install. Should be smooth sailing after fixing that.

Moreover, the paranoiac part of me believes this to be more secure.

As Docker access effectively equals root access on the host this is certainly a valid concern, but in that case an isolated VM really is your only option.

[kshpytsya]

Thank you for the detailed explanation. I am aware of the kernel config checker in Docker ebuild, but since I use a non-standard CPU scheduler (PDS, nee BFS), I have never managed to get local Docker running over the course of years. Maybe now that is possible, but considering my security concerns, I am not up to that anyway. The "remote" docker created with "docker machine" is actually running on the same physical machine but inside a VirtualBox VM running boot2docker. Actually, this setup automatically mounts /home directory of the host machine into /hosthome directory inside the VM, so doing docker-machine ssh and creating a symlink ln -s /hosthome/<user> /home has been sufficient to have the build process advance further. I also had to add -userfetch and -userpriv (maybe the latter is unnecessary) to BOB_FEATURES in kubler.conf, as otherwise I was getting "Permission denied" error at the very first fetch done by emerge. With these two fixes, the ./kubler.sh build kubler/glibc build process has successfully ran to completion. Maybe it would be possible to incorporate this logic into your system to make this automagic, or at least document it for the benefit of others trying to tread the same path I did.

[edannenberg]

Thanks for the feedback!

I also had to add -userfetch and -userpriv (maybe the latter is unnecessary) to BOB_FEATURES in kubler.conf, as otherwise I was getting "Permission denied" error at the very first fetch done by emerge

Portage's distfiles/ is also host mounted from <kubler>/tmp/distfiles into build containers, might be an issue with the portage user missing in the Docker VM though I currently don't see how exactly, as even if the user is missing a matching uid/gid should suffice.

Maybe it would be possible to incorporate this logic into your system to make this automagic, or at least document it for the benefit of others trying to tread the same path I did.

No promises on automagic, but yea should at the very least be documented. I'll leave this open until the docs are updated. Soon(tm).

PS: You can also freely edit the project wiki, should be open for all Github user accounts. This particular bit should def. be mentioned in the main install docs though.