docker in rkt with a lkvm stage1

maddyblue commented 9 years ago

I'm attempting to get docker running in rkt with the lkvm stage1. Here's what I've done so far, from a debian stretch install with rkt 0.8.1.

make a docker image with docker installed

FROM debian:stretch

RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D \
    && echo "deb http://apt.dockerproject.org/repo debian-jessie main" > /etc/apt/sources.list.d/docker.list \
    && apt-get update \
    && apt-get install -y docker-engine curl

convert it to a rkt aci

docker build -t mjibson/stretch-docker .
docker save -o sd.docker mjibson/stretch-docker
docker2aci sd.docker

run it

rkt run --debug --stage1-image=stage1-lkvm.aci --private-net \
  --mds-register=false --insecure-skip-verify \
  mjibson-stretch-docker-latest.aci --interactive --exec /bin/bash

attempt to start docker

root@172:/# docker daemon -l debug
DEBU[0000] Registering HEAD, /containers/{name:.*}/archive 
DEBU[0000] Registering GET, /images/search              
DEBU[0000] Registering GET, /containers/{name:.*}/changes 
DEBU[0000] Registering GET, /exec/{id:.*}/json          
DEBU[0000] Registering GET, /containers/{name:.*}/archive 
DEBU[0000] Registering GET, /events                     
DEBU[0000] Registering GET, /images/get                 
DEBU[0000] Registering GET, /images/{name:.*}/get       
DEBU[0000] Registering GET, /images/{name:.*}/history   
DEBU[0000] Registering GET, /images/{name:.*}/json      
DEBU[0000] Registering GET, /containers/{name:.*}/export 
DEBU[0000] Registering GET, /containers/{name:.*}/top   
DEBU[0000] Registering GET, /info                       
DEBU[0000] Registering GET, /containers/json            
DEBU[0000] Registering GET, /containers/{name:.*}/stats 
DEBU[0000] Registering GET, /_ping                      
DEBU[0000] Registering GET, /version                    
DEBU[0000] Registering GET, /images/json                
DEBU[0000] Registering GET, /containers/ps              
DEBU[0000] Registering GET, /containers/{name:.*}/json  
DEBU[0000] Registering GET, /containers/{name:.*}/logs  
DEBU[0000] Registering GET, /containers/{name:.*}/attach/ws 
DEBU[0000] Registering POST, /containers/{name:.*}/rename 
DEBU[0000] Registering POST, /auth                      
DEBU[0000] Registering POST, /images/load               
DEBU[0000] Registering POST, /containers/create         
DEBU[0000] Registering POST, /containers/{name:.*}/unpause 
DEBU[0000] Registering POST, /containers/{name:.*}/resize 
DEBU[0000] Registering POST, /containers/{name:.*}/attach 
DEBU[0000] Registering POST, /containers/{name:.*}/copy 
DEBU[0000] Registering POST, /containers/{name:.*}/exec 
DEBU[0000] Registering POST, /build                     
DEBU[0000] Registering POST, /images/{name:.*}/tag      
DEBU[0000] Registering POST, /containers/{name:.*}/pause 
DEBU[0000] Registering POST, /containers/{name:.*}/start 
DEBU[0000] Registering POST, /exec/{name:.*}/start      
DEBU[0000] Registering POST, /exec/{name:.*}/resize     
DEBU[0000] Registering POST, /images/create             
DEBU[0000] Registering POST, /containers/{name:.*}/kill 
DEBU[0000] Registering POST, /containers/{name:.*}/wait 
DEBU[0000] Registering POST, /commit                    
DEBU[0000] Registering POST, /images/{name:.*}/push     
DEBU[0000] Registering POST, /containers/{name:.*}/restart 
DEBU[0000] Registering POST, /containers/{name:.*}/stop 
DEBU[0000] Registering PUT, /containers/{name:.*}/archive 
DEBU[0000] Registering DELETE, /containers/{name:.*}    
DEBU[0000] Registering DELETE, /images/{name:.*}        
DEBU[0000] Registering OPTIONS,                         
DEBU[0000] devicemapper: driver version is 4.31.0       
WARN[0000] Udev sync is not supported. This will lead to unexpected behavior, data loss and errors. For more information, see https://docs.docker.com/reference/commandline/cli/#daemon-storage-driver-option 
DEBU[0000] Generated prefix: docker-0:15-1557127        
DEBU[0000] Checking for existence of the pool 'docker-0:15-1557127-pool' 
DEBU[0000] Pool doesn't exist. Creating it.             
DEBU[0000] Error retrieving the next available loopback: open /dev/loop-control: no such file or directory 
ERRO[0000] There are no more loopback devices available. 
ERRO[0000] [graphdriver] prior storage driver "devicemapper" failed: loopback mounting failed 
FATA[0000] Error starting daemon: error initializing graphdriver: loopback mounting failed

This ends with an error about the devicemapper driver. I've search for that error and found it a few other places, but they haven't been helpful to me.

Attempting to systemctl start it also doesn't work:

root@172:/# systemctl start docker
Running in chroot, ignoring request.

maddyblue commented 9 years ago

Progress: disabling lots of the IP options allows docker to start. It can also download images. However it cannot actually put the images on disk:

start docker with ip stuff disabled

# docker daemon --ip-forward=false --ip-masq=false --iptables=false -D

pull an image

# docker pull quay.io/coreos/etcd
Using default tag: latest
DEBU[0009] Calling POST /images/create                  
INFO[0009] POST /v1.20/images/create?fromImage=quay.io%2Fcoreos%2Fetcd%3Alatest 
DEBU[0009] hostDir: /etc/docker/certs.d/quay.io         
DEBU[0009] Trying to pull quay.io/coreos/etcd from https://quay.io v2 
DEBU[0009] Error getting v2 registry: endpoint does not support v2 API 
DEBU[0009] Trying to pull quay.io/coreos/etcd from https://quay.io v1 
DEBU[0009] hostDir: /etc/docker/certs.d/quay.io         
DEBU[0009] attempting v2 ping for registry endpoint https://quay.io/v2/ 
DEBU[0015] attempting v1 ping for registry endpoint https://quay.io/v1/ 
DEBU[0015] Error unmarshalling the _ping PingResult: json: cannot unmarshal bool into Go value of type registry.PingResult 
DEBU[0015] Registry version header: '0.6.0'             
DEBU[0015] PingResult.Version: "0.6.0"                  
DEBU[0015] Registry standalone header: '0'              
DEBU[0015] PingResult.Standalone: false                 
Pulling repository quay.io/coreos/etcd
DEBU[0015] [registry] Calling GET https://quay.io/v1/repositories/coreos/etcd/images 
DEBU[0018] Retrieving the tag list                      
DEBU[0018] Got status code 200 from https://quay.io/v1/repositories/coreos/etcd/tags/latest 
DEBU[0018] Registering tags                             
3032497b93b9: Pulling image (latest) from quay.io/coreos/etcd, endpoint: https://quay.io/v1/ 
DEBU[0018] Ancestry: [3032497b93b98d20a1cd48824e65f11a0a88fe687765b974d23e5b5e7156448e 905454660f4dbbffa71003e1915ca8990d65a5e968555b4af2e64699bd178513 8c36a566c7522acfa0765d85728a8620611d7d978ea14b233003000503e3032497b93b9: Pulling dependent layers 
3032497b93b9: Error pulling image (latest) from quay.io/coreos/etcd, operation not supported 
8c36a566c752: Pulling fs layer 
8c36a566c752: Error downloading dependent layers 
DEBU[0019] Not continuing with error: Error pulling image (latest) from quay.io/coreos/etcd, operation not supported 
Error pulling image (latest) from quay.io/coreos/etcd, operation not supported

This same error also occurs when using -s overlay to start the daemon.

jellonek commented 9 years ago

It looks like created image of pod with docker does not have character device (10, 237) as /dev/loop-control. Loop device support is enabled in kernel used by lkvm.

Could You check if manual adding mknod /dev/loop-control c 10 237 before running docker daemon helps a bit?

jellonek commented 9 years ago

Also kernel running inside of vm is not ready to run docker in default configuration, it does not support aufs.

alban commented 9 years ago

It's not currently safe to bring /dev/loop-control inside a container. Kernel patches were discussed to get loop devices safely: https://lwn.net/Articles/600463/

Of course, the lkvm flavor should not have this restriction :)

maddyblue commented 9 years ago

Adding loop-control changed the error:

root@172:/# docker daemon   
WARN[0000] Udev sync is not supported. This will lead to unexpected behavior, data loss and errors. For more information, see https://docs.docker.com/reference/commandline/cli/#daemon-storage-driver-option 
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock) 
ERRO[0000] There are no more loopback devices available. 
ERRO[0000] [graphdriver] prior storage driver "devicemapper" failed: loopback mounting failed 
FATA[0000] Error starting daemon: error initializing graphdriver: loopback mounting failed 
root@172:/# ls -l /dev/loop-control 
crw-r--r-- 1 root root 10, 237 Oct  6 19:05 /dev/loop-control

maddyblue commented 9 years ago

I ran docker/contrib/check-config.sh on rkt's cutdown-config and the output was really similar to the kernel running on my laptop.

$ bash check-config.sh /data/go/src/github.com/coreos/rkt/stage1/usr_from_kvm/kernel/cutdown-config 
info: reading kernel config from /data/go/src/github.com/coreos/rkt/stage1/usr_from_kvm/kernel/cutdown-config ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: missing
- CONFIG_MEMCG: enabled
- CONFIG_MACVLAN: enabled (as module)
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_MEMCG_KMEM: missing
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
- CONFIG_BLK_CGROUP: missing
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_BLK_DEV_THROTTLING: missing
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_EXT3_FS: missing
- CONFIG_EXT3_FS_XATTR: missing
- CONFIG_EXT3_FS_POSIX_ACL: missing
- CONFIG_EXT3_FS_SECURITY: missing
    (enable these ext3 configs if you are using ext3 as backing filesystem)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: missing
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

The notable disabled was CONFIG_CPUSETS. I compiled the rkt lkvm stage1 with that option enabled, and there was no change. (I did that by editing cluster-config and setting that option to y, then clearing the build and running make again.)

Overall still not sure what's going on here. Similar bugs for docker have been fixed by enabling kernel config options that are already enabled here.

jellonek commented 9 years ago

Clearly docker tries to use DM as storage backend (and there is mentioned problem with loop) - maybe better would be use overlayfs?

maddyblue commented 9 years ago

Same errors with -s overlay.

jonboulle commented 9 years ago

@mjibson any more progress in your investigations here?

maddyblue commented 9 years ago

No. I have no plans to investigate further. On Oct 20, 2015 1:08 PM, "Jonathan Boulle" notifications@github.com wrote:

@mjibson https://github.com/mjibson any more progress in your investigations here?

— Reply to this email directly or view it on GitHub https://github.com/coreos/rkt/issues/1539#issuecomment-149635270.

rkt / rkt