search

rancher / os

Tiny Linux distro that runs the entire OS as Docker containers
https://rancher.com/docs/os/v1.x/en/
Apache License 2.0
6.44k stars 656 forks source link

Failed to start up kernel-headers - zfs related. #749

Closed liyimeng closed 7 years ago

liyimeng commented 8 years ago

I try to start kernel-headers service, but keep getting the same failure as below. 

rancher@devops:~$ sudo ros service list
disabled amazon-ecs-agent
disabled debian-console
enabled  kernel-headers
disabled kernel-headers-system-docker
enabled  ubuntu-console
rancher@devops:~$ sudo ros service up -d kernel-headers
INFO[0000] Project [os]: Starting project               
INFO[0000] [0/19] [kernel-headers]: Starting            
ERRO[0000] Failed Starting kernel-headers : API error (500): Could not find container for entity id 827c96fe90d6c36c5cad294fb9efaf305a4030b99c41f54a1376b76982b40153

ERRO[0000] Failed to start: kernel-headers : API error (500): Could not find container for entity id 827c96fe90d6c36c5cad294fb9efaf305a4030b99c41f54a1376b76982b40153

FATA[0000] API error (500): Could not find container for entity id 827c96fe90d6c36c5cad294fb9efaf305a4030b99c41f54a1376b76982b40153

rancher@devops:~$ sudo ros service create  kernel-headers
ERRO[0000] Failed Creating kernel-headers : API error (500): Could not find container for entity id 827c96fe90d6c36c5cad294fb9efaf305a4030b99c41f54a1376b76982b40153

ERRO[0000] Failed to start: kernel-headers : API error (500): Could not find container for entity id 827c96fe90d6c36c5cad294fb9efaf305a4030b99c41f54a1376b76982b40153

FATA[0000] API error (500): Could not find container for entity id 827c96fe90d6c36c5cad294fb9efaf305a4030b99c41f54a1376b76982b40153

rancher@devops:~$ sudo ros os version
v0.4.2
deniseschannon commented 8 years ago

I am unable to reproduce. I am also using v0.4.2.

Can you do system-docker ps -a and system-docker images?

Have you tried to enable kernel headers previously?

liyimeng commented 8 years ago

Hi, the following is the text output. 

rancher@devops:~$ sudo system-docker ps -a
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS                     PORTS               NAMES
4e4993710969        rancher/os-ubuntuconsole:v0.4.0   "/usr/sbin/entry.sh r"   7 weeks ago         Exited (137) 7 weeks ago                       reboot
148e4f1d6463        rancher/os:v0.4.2                 "/scripts/lay-down-os"   7 weeks ago         Exited (0) 7 weeks ago                         os-upgrade
39c0961e6ec2        rancher/os-docker:v0.4.2          "/usr/sbin/entry.sh /"   7 weeks ago         Up 7 weeks                                     docker
f608ba756b43        rancher/os-preload:v0.4.2         "/usr/sbin/entry.sh /"   7 weeks ago         Exited (0) 7 weeks ago                         preload-user-images
3f92b8b3ee7b        rancher/os-ntp:v0.4.2             "/usr/sbin/entry.sh /"   7 weeks ago         Up 7 weeks                                     ntp
b3c24abc52b3        rancher/os-cloudinit:v0.4.2       "/usr/sbin/entry.sh /"   7 weeks ago         Exited (0) 7 weeks ago                         cloud-init
8c7448cac1af        rancher/os-network:v0.4.2         "/usr/sbin/entry.sh w"   7 weeks ago         Exited (0) 7 weeks ago                         wait-for-network
f47bd367ae85        rancher/os-network:v0.4.2         "/usr/sbin/entry.sh n"   7 weeks ago         Up 7 weeks                                     network
f196332c2519        rancher/os-cloudinit:v0.4.2       "/usr/sbin/entry.sh /"   7 weeks ago         Exited (0) 7 weeks ago                         cloud-init-pre
a3f5dfe7a899        rancher/os-udev:v0.4.2            "/usr/sbin/entry.sh /"   7 weeks ago         Up 7 weeks                                     udev
0668082d0f63        rancher/os-state:v0.4.2           "/usr/sbin/entry.sh e"   7 weeks ago         Created                                        all-volumes
5757837ef6f8        rancher/os-preload:v0.4.2         "/usr/sbin/entry.sh /"   7 weeks ago         Exited (0) 7 weeks ago                         preload-system-images
88fd5e4ee02b        rancher/os-udev:v0.4.2            "/usr/sbin/entry.sh /"   7 weeks ago         Exited (0) 7 weeks ago                         udev-cold
cf0307b81760        rancher/os-syslog:v0.4.2          "/usr/sbin/entry.sh /"   7 weeks ago         Up 7 weeks                                     syslog
1db920d8897c        rancher/os-acpid:v0.4.2           "/usr/sbin/entry.sh /"   7 weeks ago         Up 7 weeks                                     acpid
9a1d7a385807        rancher/os-state:v0.4.2           "/usr/sbin/entry.sh e"   7 weeks ago         Created                                        user-volumes
bc0ae8609e08        rancher/os-state:v0.4.2           "/usr/sbin/entry.sh e"   7 weeks ago         Created                                        system-volumes
98235d320dcb        rancher/os-state:v0.4.2           "/usr/sbin/entry.sh e"   7 weeks ago         Created                                        command-volumes
6dec31b35320        rancher/os-state:v0.4.2           "/usr/sbin/entry.sh e"   7 weeks ago         Created                                        container-data-volumes
12190b4b1318        rancher/os-acpid:v0.4.0           "/usr/sbin/entry.sh p"   8 weeks ago         Exited (137) 8 weeks ago                       poweroff
800a7590e9d8        rancher/os-ubuntuconsole:v0.4.0   "/usr/sbin/entry.sh s"   8 weeks ago         Exited (0) 8 weeks ago                         shutdown
ede3dec0e6a5        rancher/os-ubuntuconsole:v0.4.0   "/usr/sbin/entry.sh /"   11 weeks ago        Up 7 weeks                                     console

rancher@devops:~$ sudo system-docker images
REPOSITORY                 TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
rancher/os                 v0.4.2              a31171b37e91        7 weeks ago         191.1 MB
rancher/os-console         v0.4.2              a9cb7aecbce1        7 weeks ago         24.21 MB
rancher/os-ubuntuconsole   v0.4.2              fad710785fe2        8 weeks ago         315.7 MB
rancher/os                 v0.4.1              f5261f101133        12 weeks ago        191 MB
rancher/os                 v0.4.0              29bb895e50e7        3 months ago        189.6 MB
rancher/os-console         v0.4.0              3558fac10012        3 months ago        24.21 MB
rancher/os-ubuntuconsole   v0.4.0              ceff950b10d2        4 months ago        313.9 MB
rancher/os-udev            v0.4.0              779760bac0e3        4 months ago        24.2 MB
rancher/os-udev            v0.4.2              779760bac0e3        4 months ago        24.2 MB
                                   35568319b672        4 months ago        24.2 MB
                                   fd2eb892d4e3        4 months ago        24.2 MB
rancher/os-syslog          v0.4.0              a22cbc26f080        4 months ago        24.2 MB
rancher/os-syslog          v0.4.2              a22cbc26f080        4 months ago        24.2 MB
rancher/os-state           v0.4.0              e209e7cc9c49        4 months ago        24.2 MB
rancher/os-state           v0.4.2              e209e7cc9c49        4 months ago        24.2 MB
                                   6c304e6e5d1e        4 months ago        24.2 MB
                                   2f426dbcfed9        4 months ago        24.2 MB
rancher/os-preload         v0.4.0              40242dfa77b8        4 months ago        24.2 MB
rancher/os-preload         v0.4.2              40242dfa77b8        4 months ago        24.2 MB
                                   ce7ac1ff3b77        4 months ago        24.2 MB
rancher/os-ntp             v0.4.0              77a79238a95f        4 months ago        24.2 MB
rancher/os-ntp             v0.4.2              77a79238a95f        4 months ago        24.2 MB
                                   6c462c29c779        4 months ago        24.2 MB
rancher/os-network         v0.4.0              92fc71defec0        4 months ago        24.2 MB
rancher/os-network         v0.4.2              92fc71defec0        4 months ago        24.2 MB
rancher/os-docker          v0.4.0              3cd1695690b0        4 months ago        24.2 MB
rancher/os-docker          v0.4.2              3cd1695690b0        4 months ago        24.2 MB
                                   155d138933d6        4 months ago        24.2 MB
rancher/os-cloudinit       v0.4.0              9afe2f48e3b0        4 months ago        24.2 MB
rancher/os-cloudinit       v0.4.2              9afe2f48e3b0        4 months ago        24.2 MB
                                   8effca998862        4 months ago        24.2 MB
rancher/os-autoformat      v0.4.0              88f547930594        4 months ago        24.2 MB
rancher/os-autoformat      v0.4.2              88f547930594        4 months ago        24.2 MB
                                   9c68bb77d77d        4 months ago        24.2 MB
rancher/os-acpid           v0.4.2              f778ee110715        4 months ago        24.2 MB
rancher/os-acpid           v0.4.0              f778ee110715        4 months ago        24.2 MB
deniseschannon commented 8 years ago

Sorry, can you also do a docker ps and docker images. I forgot that the kernel headers were actually in user docker.

For example, mine shows:

$ docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
rancher/os-headers   v0.4.2              0a61178c6ca7        4 months ago        24.2 MB
$ docker ps -a
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS                    PORTS               NAMES
3d5f424bf28c        rancher/os-headers:v0.4.2   "/usr/sbin/entry.sh /"   22 hours ago        Exited (0) 22 hours ago                       os_kernel-headers_1
liyimeng commented 8 years ago

No problem, here it is: 

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES
718937d6293a        dfb4481bd21b        "/bin/registry /etc/d"   8 weeks ago         Up 7 weeks          0.0.0.0:80->5000/tcp   registry

docker images
REPOSITORY                        TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
ubuntu                            15.10               d8a164f81acc        2 weeks ago         134.4 MB
debian                            jessie              7a01cc5f27b1        2 weeks ago         125.1 MB
ubuntu                            14.04               6cc0fc2a5ee3        3 weeks ago         187.9 MB
ubuntu                            latest              6cc0fc2a5ee3        3 weeks ago         187.9 MB
ubuntu                            14.04.3             e9ae3c220b23        3 months ago        187.9 MB
rancher/os-headers                v0.4.2              0a61178c6ca7        4 months ago        24.2 MB
deniseschannon commented 8 years ago

How are you launching RancherOS and did you do any additional configuration (like change the user docker version or anything?) Trying to understand what could be different in my setup and yours.

liyimeng commented 8 years ago

I ipxe boot the node, then install it into hard-disk with ubuntu console enabled. After that I actually successfully enable kernel-header once and build zfs. I then enable zfs as docker storage driver 

  docker:
    tls_args: [--tlsverify, --tlscacert=/etc/docker/tls/ca.pem, --tlscert=/etc/docker/tls/server-cert.pem,
      --tlskey=/etc/docker/tls/server-key.pem, '-H=0.0.0.0:2376']
    args: [daemon, --log-opt, max-size=25m, --log-opt, max-file=2, -s, zfs, --storage-opt,
      zfs.fsname=hpool/docker, -G, docker, -H, 'unix:///var/run/docker.sock', --userland-proxy=false,
      --insecure-registry=10.8.9.0/24]

Could it be the case that my zfs dataset is mounted into /var/lib/docker directory, and then covering up the old containers that was previously stored there?

liyimeng commented 8 years ago

@imikushin should we consider to build zfs.ko into kernel now. Ubuntu is doing so http://blog.dustinkirkland.com/2016/02/zfs-is-fs-for-containers-in-ubuntu-1604.html

imikushin commented 8 years ago

@liyimeng We'll look into building in the zfs module.

Meanwhile, RancherOS v0.5.0 (not released yet) should definitely help with ZFS support. I've already updated the storage customization docs for ZFS support for the upcoming version.

In short, you shouldn't mount ZFS to /var/lib/docker. Instead use /your-zpool1/docker as graph storage location in docker args (assuming /your-zpool1/docker is the mountpoint for a ZFS filesystem).

SvenDowideit commented 7 years ago

with RancherOS v.0.8.0 moving to mainline 4.8.* kernels, we'll need to make a separate zfs service that can be added into a cloud-config.

SvenDowideit commented 7 years ago

zfs is now a magical new service that does it all for you (on amd64 only atm)

so closing