search

luxas / kubernetes-on-arm

Kubernetes ported to ARM boards like Raspberry Pi.
MIT License
600 stars 85 forks source link

kube-config enable-master failed #21

Closed gyoho closed 9 years ago

gyoho commented 9 years ago

Hello, Awesome project! Thanks for all your hard work. I am using this for my school project. I have two Raspberry Pi2 Model B and trying to make a cluster. But when I tried to start the master node I got this error:

[root@pi1 ~]# kube-config enable-master
Disabling k8s if it is running
Removed symlink /etc/systemd/system/multi-user.target.wants/k8s-master.service.
Removed symlink /etc/systemd/system/multi-user.target.wants/flannel.service.
Removed symlink /etc/systemd/system/multi-user.target.wants/etcd.service.
Checks so all images are present
Transferring images to system-docker, if necessary
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/flannel.service to /usr/lib/systemd/system/flannel.service.
Job for etcd.service failed because the control process exited with error code. See "systemctl status etcd.service" and "journalctl -xe" for details.
Job for docker.service failed because a configured resource limit was exceeded. See "systemctl status docker.service" and "journalctl -xe" for details.
Starting the master containers
Created symlink from /etc/systemd/system/multi-user.target.wants/k8s-master.service to /usr/lib/systemd/system/k8s-master.service.
Job for k8s-master.service failed because the control process exited with error code. See "systemctl status k8s-master.service" and "journalctl -xe" for details.
Master Kubernetes services enabled

Status

[root@pi1 ~]# systemctl status etcd.service
* etcd.service - Etcd Master Data Store for Kubernetes Apiserver
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2015-11-26 10:06:27 PST; 1min 1s ago
 Main PID: 9893 (code=exited, status=1/FAILURE)

Nov 26 10:06:23 pi1 docker[9893]: operation not permitted
Nov 26 10:06:24 pi1 docker[9893]: Error response from daemon: Cannot start container 30d6de2580837a1d3c647f584d66a6133cdec75538002813ee25834f494a820a: [8] System erro...ot permitted
Nov 26 10:06:24 pi1 systemd[1]: etcd.service: Main process exited, code=exited, status=1/FAILURE
Nov 26 10:06:26 pi1 bash[9894]: operation not permitted
Nov 26 10:06:26 pi1 bash[9894]: Error response from daemon: Cannot start container c7e7913e70fa36783947042a0eeb9a66dfdbb9fe2d0e116565ca2a68a59f5519: [8] System error:...ot permitted
Nov 26 10:06:26 pi1 systemd[1]: etcd.service: Control process exited, code=exited status=1
Nov 26 10:06:27 pi1 docker[9937]: k8s-etcd
Nov 26 10:06:27 pi1 systemd[1]: Failed to start Etcd Master Data Store for Kubernetes Apiserver.
Nov 26 10:06:27 pi1 systemd[1]: etcd.service: Unit entered failed state.
Nov 26 10:06:27 pi1 systemd[1]: etcd.service: Failed with result 'exit-code'.
Hint: Some lines were ellipsized, use -l to show in full.
[root@pi1 ~]# journalctl -xe
Nov 26 10:09:14 pi1 docker[9974]: E1126 18:09:14.496914 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 10:09:15 pi1 docker[9974]: E1126 18:09:15.498819 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 10:09:16 pi1 docker[9974]: E1126 18:09:16.500722 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 10:09:17 pi1 docker[9974]: E1126 18:09:17.502528 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 10:09:18 pi1 docker[9974]: E1126 18:09:18.504369 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 10:09:19 pi1 systemd[1]: k8s-master.service: Service hold-off time over, scheduling restart.
Nov 26 10:09:19 pi1 systemd[1]: Stopped The Master Components for Kubernetes.
-- Subject: Unit k8s-master.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit k8s-master.service has finished shutting down.
Nov 26 10:09:19 pi1 systemd[1]: Starting The Master Components for Kubernetes...
-- Subject: Unit k8s-master.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit k8s-master.service has begun starting up.
Nov 26 10:09:19 pi1 docker[9974]: E1126 18:09:19.506442 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 10:09:19 pi1 docker[11104]: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Nov 26 10:09:19 pi1 docker[11104]: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Nov 26 10:09:19 pi1 docker[11104]: Error: failed to kill containers: [k8s-master k8s-worker-proxy]
Nov 26 10:09:19 pi1 docker[11113]: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Nov 26 10:09:19 pi1 docker[11113]: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Nov 26 10:09:19 pi1 docker[11113]: Error: failed to remove containers: [k8s-master k8s-worker-proxy]
Nov 26 10:09:20 pi1 docker[11124]: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Nov 26 10:09:20 pi1 systemd[1]: k8s-master.service: Control process exited, code=exited status=1
Nov 26 10:09:20 pi1 docker[11137]: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Nov 26 10:09:20 pi1 docker[11137]: Cannot connect to the Docker daemon. Is the docker daemon running on this host?
Nov 26 10:09:20 pi1 docker[11137]: Error: failed to stop containers: [k8s-master k8s-worker-proxy]
Nov 26 10:09:20 pi1 systemd[1]: k8s-master.service: Control process exited, code=exited status=1
Nov 26 10:09:20 pi1 systemd[1]: Failed to start The Master Components for Kubernetes.
-- Subject: Unit k8s-master.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit k8s-master.service has failed.
--
-- The result is failed.
Nov 26 10:09:20 pi1 systemd[1]: k8s-master.service: Unit entered failed state.
Nov 26 10:09:20 pi1 systemd[1]: k8s-master.service: Failed with result 'exit-code'.
Nov 26 10:09:20 pi1 docker[9974]: E1126 18:09:20.509224 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 10:09:21 pi1 docker[9974]: E1126 18:09:21.511007 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 10:09:22 pi1 docker[9974]: E1126 18:09:22.512882 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 10:09:23 pi1 docker[9974]: E1126 18:09:23.514737 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured

Version info

[root@pi1 ~]# docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.5.1
 Git commit:   a34a1d5-dirty
 Built:        Mon Nov 23 14:47:45 UTC 2015
 OS/Arch:      linux/arm
Cannot connect to the Docker daemon. Is the docker daemon running on this host?
[root@pi1 ~]# kube-config info
Architecture: armv7l
Kernel: Linux 4.1.13
CPU: 4 cores x 900 MHz

Used RAM Memory: 45 MiB
RAM Memory: 923 MiB

Used disk space: 3.2GB (3265776 KB)
Free disk space: 3.6GB (3741908 KB)

SD Card was built: 04-11-2015 21:17

kubernetes-on-arm:
Latest commit: ccea1ef
Version: 0.5.8

systemd version: 227

I found similar issue in one of the issue threads, but it was kind of hard to follow. So, I thought it's good to start a new thread.

Thanks for your help in advance.

luxas commented 9 years ago

Thank you for using this for your school project! That is primary what I think this would be very useful for :smile:

Unfortunately, there is a bug in docker v1.9.0 when running on ARM. Some days ago, the Arch Linux ARM team upgraded docker from v1.7.1 (which works fine) to v1.9.0, which fails sporadically: https://github.com/docker/docker/issues/18125 But in the meantime, nobody knows what is causing this to fail.

This isn't a bug in my code right now, but since I (at the moment) depend on the Arch Linux ARM provided docker, this project becomes unusable. This issue is the same as: #19, but don't worry, I have more information now.

What you can do to downgrade docker to v1.7.1: 

pacman -R docker
curl -sSL https://s3.amazonaws.com/docker-armv7/docker-1:1.7.1-2-armv7h.pkg.tar.xz > /var/cache/pacman/pkg/docker-1:1.7.1-2-armv7h.pkg.tar.xz
pacman -U  /var/cache/pacman/pkg/docker-1:1.7.1-2-armv7h.pkg.tar.xz
systemctl daemon-reload
systemctl restart docker

# Test it 
docker version
docker run -it resin/rpi-raspbian /bin/bash

This will get v1.7.1 from an unofficial cache and install it. See: https://github.com/docker/docker/issues/18125#issuecomment-159736200

Hope it works!

gyoho commented 9 years ago

Hi Luxas,

Thanks very much for your quick reply! I downgraded docker to v1.7.1. But when I do systemctl restart docker, it threw me this error message:

[root@pi1 ~]# systemctl restart docker
Job for docker.service failed because a configured resource limit was exceeded. See "systemctl status docker.service" and "journalctl -xe" for details.

Status

[root@pi1 ~]# systemctl status docker.service
* docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/docker.service.d
           `-docker-flannel.conf
   Active: failed (Result: start-limit) since Thu 2015-11-26 10:06:33 PST; 1h 44min ago
     Docs: https://docs.docker.com
 Main PID: 9730 (code=exited, status=0/SUCCESS)

Nov 26 11:48:36 pi1 systemd[1]: docker.service: Failed to run 'start-pre' task: No such file or directory
Nov 26 11:48:36 pi1 systemd[1]: Failed to start Docker Application Container Engine.
Nov 26 11:48:36 pi1 systemd[1]: docker.service: Failed with result 'resources'.
Nov 26 11:48:36 pi1 systemd[1]: docker.service: Failed to load environment files: No such file or directory
Nov 26 11:48:36 pi1 systemd[1]: docker.service: Failed to run 'start-pre' task: No such file or directory
Nov 26 11:48:36 pi1 systemd[1]: Failed to start Docker Application Container Engine.
Nov 26 11:48:36 pi1 systemd[1]: docker.service: Failed with result 'resources'.
Nov 26 11:48:36 pi1 systemd[1]: docker.service: Start request repeated too quickly.
Nov 26 11:48:36 pi1 systemd[1]: Failed to start Docker Application Container Engine.
Nov 26 11:48:36 pi1 systemd[1]: docker.service: Failed with result 'start-limit'.
[root@pi1 ~]# journalctl -xe
-- The result is failed.
Nov 26 11:55:17 pi1 systemd[1]: k8s-master.service: Unit entered failed state.
Nov 26 11:55:17 pi1 systemd[1]: k8s-master.service: Failed with result 'exit-code'.
Nov 26 11:55:17 pi1 docker[9974]: E1126 19:55:17.749191 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 11:55:18 pi1 docker[9974]: E1126 19:55:18.754234 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 11:55:19 pi1 docker[9974]: E1126 19:55:19.756221 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 11:55:20 pi1 docker[9974]: E1126 19:55:20.758095 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 11:55:21 pi1 docker[9974]: E1126 19:55:21.759955 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 11:55:22 pi1 systemd[1]: k8s-master.service: Service hold-off time over, scheduling restart.
Nov 26 11:55:22 pi1 systemd[1]: Stopped The Master Components for Kubernetes.
-- Subject: Unit k8s-master.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit k8s-master.service has finished shutting down.
Nov 26 11:55:22 pi1 systemd[1]: Starting The Master Components for Kubernetes...
-- Subject: Unit k8s-master.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit k8s-master.service has begun starting up.
Nov 26 11:55:22 pi1 docker[18703]: Post http:///var/run/docker.sock/v1.19/containers/k8s-master/kill?signal=KILL: dial unix /var/run/docker.sock: no such file or directory. Are you
Nov 26 11:55:22 pi1 docker[18703]: Post http:///var/run/docker.sock/v1.19/containers/k8s-worker-proxy/kill?signal=KILL: dial unix /var/run/docker.sock: no such file or directory. Ar
Nov 26 11:55:22 pi1 docker[18703]: Error: failed to kill containers: [k8s-master k8s-worker-proxy]
Nov 26 11:55:22 pi1 docker[18710]: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?
Nov 26 11:55:22 pi1 docker[18710]: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?
Nov 26 11:55:22 pi1 docker[18710]: Error: failed to remove containers: [k8s-master k8s-worker-proxy]
Nov 26 11:55:22 pi1 docker[9974]: E1126 19:55:22.761599 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 11:55:22 pi1 docker[18716]: Post http:///var/run/docker.sock/v1.19/containers/create?name=k8s-worker-proxy: dial unix /var/run/docker.sock: no such file or directory. Are you
Nov 26 11:55:22 pi1 systemd[1]: k8s-master.service: Control process exited, code=exited status=1
Nov 26 11:55:22 pi1 docker[18726]: Post http:///var/run/docker.sock/v1.19/containers/k8s-master/stop?t=10: dial unix /var/run/docker.sock: no such file or directory. Are you trying
Nov 26 11:55:22 pi1 docker[18726]: Post http:///var/run/docker.sock/v1.19/containers/k8s-worker-proxy/stop?t=10: dial unix /var/run/docker.sock: no such file or directory. Are you t
Nov 26 11:55:22 pi1 docker[18726]: Error: failed to stop containers: [k8s-master k8s-worker-proxy]
Nov 26 11:55:22 pi1 systemd[1]: k8s-master.service: Control process exited, code=exited status=1
Nov 26 11:55:22 pi1 systemd[1]: Failed to start The Master Components for Kubernetes.
-- Subject: Unit k8s-master.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit k8s-master.service has failed.
--
-- The result is failed.
Nov 26 11:55:22 pi1 systemd[1]: k8s-master.service: Unit entered failed state.
Nov 26 11:55:22 pi1 systemd[1]: k8s-master.service: Failed with result 'exit-code'.
Nov 26 11:55:23 pi1 docker[9974]: E1126 19:55:23.763459 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
[root@pi1 ~]# journalctl -xe
Nov 26 11:55:34 pi1 docker[9974]: E1126 19:55:34.784040 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 11:55:35 pi1 docker[9974]: E1126 19:55:35.785926 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 11:55:36 pi1 docker[9974]: E1126 19:55:36.787829 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 11:55:37 pi1 docker[9974]: E1126 19:55:37.789722 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 11:55:38 pi1 docker[9974]: E1126 19:55:38.791590 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 11:55:39 pi1 systemd[1]: k8s-master.service: Service hold-off time over, scheduling restart.
Nov 26 11:55:39 pi1 systemd[1]: Stopped The Master Components for Kubernetes.
-- Subject: Unit k8s-master.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit k8s-master.service has finished shutting down.
Nov 26 11:55:39 pi1 systemd[1]: Starting The Master Components for Kubernetes...
-- Subject: Unit k8s-master.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit k8s-master.service has begun starting up.
Nov 26 11:55:39 pi1 docker[18788]: Post http:///var/run/docker.sock/v1.19/containers/k8s-master/kill?signal=KILL: dial unix /var/run/docker.sock: no such file or directory. Are you
Nov 26 11:55:39 pi1 docker[18788]: Post http:///var/run/docker.sock/v1.19/containers/k8s-worker-proxy/kill?signal=KILL: dial unix /var/run/docker.sock: no such file or directory. Ar
Nov 26 11:55:39 pi1 docker[18788]: Error: failed to kill containers: [k8s-master k8s-worker-proxy]
Nov 26 11:55:39 pi1 docker[9974]: E1126 19:55:39.793308 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 11:55:39 pi1 docker[18793]: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?
Nov 26 11:55:39 pi1 docker[18793]: dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?
Nov 26 11:55:39 pi1 docker[18793]: Error: failed to remove containers: [k8s-master k8s-worker-proxy]
Nov 26 11:55:39 pi1 docker[18800]: Post http:///var/run/docker.sock/v1.19/containers/create?name=k8s-worker-proxy: dial unix /var/run/docker.sock: no such file or directory. Are you
Nov 26 11:55:39 pi1 systemd[1]: k8s-master.service: Control process exited, code=exited status=1
Nov 26 11:55:40 pi1 docker[18809]: Post http:///var/run/docker.sock/v1.19/containers/k8s-master/stop?t=10: dial unix /var/run/docker.sock: no such file or directory. Are you trying
Nov 26 11:55:40 pi1 docker[18809]: Post http:///var/run/docker.sock/v1.19/containers/k8s-worker-proxy/stop?t=10: dial unix /var/run/docker.sock: no such file or directory. Are you t
Nov 26 11:55:40 pi1 docker[18809]: Error: failed to stop containers: [k8s-master k8s-worker-proxy]
Nov 26 11:55:40 pi1 systemd[1]: k8s-master.service: Control process exited, code=exited status=1
Nov 26 11:55:40 pi1 systemd[1]: Failed to start The Master Components for Kubernetes.
-- Subject: Unit k8s-master.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit k8s-master.service has failed.
--
-- The result is failed.
Nov 26 11:55:40 pi1 systemd[1]: k8s-master.service: Unit entered failed state.
Nov 26 11:55:40 pi1 systemd[1]: k8s-master.service: Failed with result 'exit-code'.
Nov 26 11:55:40 pi1 docker[9974]: E1126 19:55:40.795327 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 11:55:41 pi1 docker[9974]: E1126 19:55:41.797420 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 11:55:42 pi1 docker[9974]: E1126 19:55:42.799302 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured
Nov 26 11:55:43 pi1 docker[9974]: E1126 19:55:43.801210 00001 network.go:53] Failed to retrieve network config: client: etcd cluster is unavailable or misconfigured

Version

[root@pi1 ~]# docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/arm
Cannot connect to the Docker daemon. Is 'docker -d' running on this host?

Docker Daemon

[root@pi1 ~]# docker -d
INFO[0000] [graphdriver] using prior storage driver "overlay"
INFO[0000] Listening for HTTP on unix (/var/run/docker.sock)
INFO[0000] Loading containers: start.
....
INFO[0000] Loading containers: done.
INFO[0000] Daemon has completed initialization
INFO[0000] Docker daemon                                 commit=786b29d execdriver=native-0.2 graphdriver=overlay version=1.7.1
INFO[0000] POST /v1.19/containers/create?name=k8s-worker-proxy
ERRO[0000] Handler for POST /containers/create returned error: Conflict. The name "k8s-worker-proxy" is already in use by container 11c0daa8ce7d. You have to delete (or rename) that container to be able to reuse that name.
ERRO[0000] HTTP Error                                    err=Conflict. The name "k8s-worker-proxy" is already in use by container 11c0daa8ce7d. You have to delete (or rename) that container to be able to reuse that name. statusCode=409
INFO[0001] POST /v1.19/containers/k8s-master/stop?t=10
ERRO[0001] Handler for POST /containers/{name:.*}/stop returned error: no such id: k8s-master
ERRO[0001] HTTP Error                                    err=no such id: k8s-master statusCode=404
INFO[0001] POST /v1.19/containers/k8s-worker-proxy/stop?t=10
INFO[0006] POST /v1.19/containers/k8s-master/kill?signal=KILL
ERRO[0006] Handler for POST /containers/{name:.*}/kill returned error: no such id: k8s-master
ERRO[0006] HTTP Error                                    err=no such id: k8s-master statusCode=404
INFO[0006] POST /v1.19/containers/k8s-worker-proxy/kill?signal=KILL
INFO[0006] DELETE /v1.19/containers/k8s-master
ERRO[0006] Handler for DELETE /containers/{name:.*} returned error: no such id: k8s-master
ERRO[0006] HTTP Error                                    err=no such id: k8s-master statusCode=404
INFO[0006] DELETE /v1.19/containers/k8s-worker-proxy
INFO[0006] POST /v1.19/containers/create?name=k8s-worker-proxy
INFO[0007] POST /v1.19/containers/bf9841c795a93b58bfe7565cc2fc3276759b45c6543f64f0e30c9821a9c2d69c/start
WARN[0007] signal: killed
ERRO[0008] Handler for POST /containers/{name:.*}/start returned error: Cannot start container bf9841c795a93b58bfe7565cc2fc3276759b45c6543f64f0e30c9821a9c2d69c: [8] System error: open /sys/fs/cgroup/cpu,cpuacct/init.scope/system.slice/docker-bf9841c795a93b58bfe7565cc2fc3276759b45c6543f64f0e30c9821a9c2d69c.scope/cpu.shares: no such file or directory
ERRO[0008] HTTP Error                                    err=Cannot start container bf9841c795a93b58bfe7565cc2fc3276759b45c6543f64f0e30c9821a9c2d69c: [8] System error: open /sys/fs/cgroup/cpu,cpuacct/init.scope/system.slice/docker-bf9841c795a93b58bfe7565cc2fc3276759b45c6543f64f0e30c9821a9c2d69c.scope/cpu.shares: no such file or directory statusCode=404

Any thoughts?

luxas commented 9 years ago

Oh, I am so sorry that I forgot to say this. Can you do the following:

nano /etc/kubernetes/dynamic-dropins/docker-flannel.conf
# Add these two lines in the end of the file
Restart=on-failure
RestartSec=2

And reboot your Pi (just to be sure)

This is a known issue in v0.5.8, that sometimes, the docker daemon restarts too quickly (a race condition).

Hopefully I may release v0.6.0 soon, which has many new features and bugfixes.

Running docker -d will never work, because of a systemd related issue in main docker. Instead

# Whenever kube-config enable-* breaks something, you may revert the changes made with
kube-config disable-node

# Now docker will work as usual, with no special k8s config
gyoho commented 9 years ago

Hi Luxas,

It worked and the master node got started now! I also started a worker node too.

Master
[root@pi1 ~]# docker ps
CONTAINER ID        IMAGE                       COMMAND                CREATED             STATUS              PORTS               NAMES
c55fda9483e1        kubernetesonarm/hyperkube   "/hyperkube controll   14 minutes ago      Up 14 minutes                           k8s_controller-manager.7047e990_k8s-master-192.168.10.101_kube-system_447c171dfac8ae64dc585f8e9cbfa7e6_51d04c35
67af5d172430        kubernetesonarm/hyperkube   "/hyperkube schedule   14 minutes ago      Up 14 minutes                           k8s_scheduler.e5efe276_k8s-master-192.168.10.101_kube-system_447c171dfac8ae64dc585f8e9cbfa7e6_4d4e23d8
0e4dc479dff4        kubernetesonarm/hyperkube   "/hyperkube apiserve   14 minutes ago      Up 14 minutes                           k8s_apiserver.c358020f_k8s-master-192.168.10.101_kube-system_447c171dfac8ae64dc585f8e9cbfa7e6_1b3b1185
8575cb0c6a98        kubernetesonarm/pause       "/pause"               15 minutes ago      Up 15 minutes                           k8s_POD.7ad6c339_k8s-master-192.168.10.101_kube-system_447c171dfac8ae64dc585f8e9cbfa7e6_bf3da10b
480b2400e35a        kubernetesonarm/hyperkube   "/hyperkube kubelet    16 minutes ago      Up 16 minutes                           k8s-master
66e7ebfe3a21        kubernetesonarm/hyperkube   "/hyperkube proxy --   16 minutes ago      Up 16 minutes                           k8s-worker-proxy
Worker
[root@pi2 ~]# docker ps
CONTAINER ID        IMAGE                       COMMAND                CREATED             STATUS              PORTS               NAMES
b4848cf67c5e        kubernetesonarm/hyperkube   "/hyperkube kubelet    5 minutes ago       Up 5 minutes                            k8s-worker
8719622a1c63        kubernetesonarm/hyperkube   "/hyperkube proxy --   5 minutes ago       Up 5 minutes                            k8s-worker-proxy

Thanks again for your help!

By the way, for my school project I'm planing to implement an example from the official Kubernetes repo. I'm thinking to setup a master node on my netbook by following this tutorial and add my two worker RPi nodes (setup with your project) to the master node. I see when I do kube-config enable-worker, it asks me the IP addr of the master node. So, I guess just by providing the master node IP they can form a cluster?

Do you think it is possible to do what I'm trying to do? If so, can you tell me how I can change the IP address of master node on a worker node? Because the second time I execute kube-config enable-worker, it didn't ask me the IP addr of the master again...

Sorry for many questions. Many thanks in advance!!

luxas commented 9 years ago

It should be possible (in theory, haven't tested it) to have a master with amd64. The thing you should be aware of is: https://github.com/luxas/kubernetes-on-arm/blob/dev/docs/docker-on-arm.md You will have to find (or build yourself) rabbitmq and celery images. You may probably swap ubuntu against luxas/raspbian

You may change the worker IP in /etc/kubernetes/k8s.conf. That was "bug as a feature". If you have ideas how I could make it more intuitive, please tell me. I just don't want the user to type the same IP all the time.

If you use master-multi.json, remember to checkout to a stable version v1.1.1 first. But there is things that isn't working by default, like serviceAccounts

Remember, that all images you run on the cluster (with amd64 as master and arm as workers) have to be compiled for arm. So, push your customized images to the Hub or a private registry. You will probably have to label your nodes with which arch they have, so the images don't land on the wrong machine. https://github.com/kubernetes/kubernetes/tree/master/docs/user-guide/node-selection

# for the master
kubectl label no {ip-address} arch=amd64

# for all minions
kubectl label no {ip-address} arch=arm

Example rc:

apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    component: rabbitmq
  name: rabbitmq-controller
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: taskQueue
        component: rabbitmq
    spec:
      containers:
      - image: rabbitmq
        name: rabbitmq
        ports:
        - containerPort: 5672
        resources:
          limits:
            cpu: 100m
      nodeSelector:
        arch: amd64
gyoho commented 9 years ago

Hello Luxas,

Thanks for the followup. My notebook is actually i386 running Ubuntu-14.04. I installed Docker by this page. But, it seems there are not many docker images available even for i386.

If all images on the cluster have to be compiled for arm, then it will be much easier to just stick with RPi and use your project. So, I decided to build a 4-RPi cluster.

By the way, to build my own images which are compiled for arm, I'm thinking to take the following steps:

  1. Pull your luxas/raspbian image and start the raspbian container
  2. Install necessary packages (rabbitmq, celery, java, and so on) on the raspbian container
  3. Commit the configured container and publish to the Docker Hub

Do you think I'm on the right track?

luxas commented 9 years ago

Yeah, if you have a 32-bit laptop, it's easiest to use only Pis, because 32-bit k8s isn't supported either.

It's better to write (and much faster) Dockerfiles instead of running interactively and committing. Just as a tip.

Are you sure that docker pull luxas/raspbian doesn't work? It may take some minutes to pull from the Hub.

resin/rpi-raspbian seems to output that strange error on my machine also: 

Error response from daemon: image ID '' is invalid

But if one specifies the tag, it works: resin/rpi-raspbian:jessie

gyoho commented 9 years ago

Sorry I pulled luxas/raspbian not luxas/rpi-raspbian. I successfully started luxas/raspbian image and now I'm installing Python on the container by following this instruction.

However, I got the following error:

root@16ab2eb06084:/# sudo apt-get update
Hit http://archive.raspbian.org jessie InRelease
Get:1 http://archive.raspbian.org jessie/main armhf Packages [12.4 MB]
Get:2 http://archive.raspbian.org jessie/contrib armhf Packages [42.6 kB]
Get:3 http://archive.raspbian.org jessie/non-free armhf Packages [83.5 kB]
Get:4 http://archive.raspbian.org jessie/rpi armhf Packages [1297 B]
Get:5 http://archive.raspbian.org jessie/firmware armhf Packages [1176 B]
Fetched 12.6 MB in 42s (295 kB/s)
Reading package lists... Done
root@16ab2eb06084:/# sudo apt-get install python3-picamera
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package python3-picamera

Any ideas?

luxas commented 9 years ago

Do you really need picamera? That's just an example package and it seems like it doesn't exist.

Check out these images: https://github.com/hypriot?tab=repositories hypriot has prebuilt python and java arm images. At least, use their Dockerfiles as a starting point.

gyoho commented 9 years ago

I pulled hypriot/rpi-python and started the container, and successfully started the ipython-notebook.

Since it's deviating from the original issue, I'm closing this thread. I'm going to build kubernetes-celery-rabbitmq cluster, and I'll let you know if I encounter further issues.

Thanks again for the helps.

v1k0d3n commented 8 years ago

@luxas hate to rehash an "old" issue, but have they figured out what the issue is or is there a ticket that you're following in the Docker repo? i experienced the issue too, but I experienced it prior to upgrading actually. then i upgraded, searched some issues and it let me straight to here.

btw, gave you some major props on my blog: http://www.jinkit.com/k8s-on-rpi/. thanks for the cool project...this is really nice to have!

luxas commented 8 years ago

Hello Brandon! Yep, I have fixed this issue (at least I think so; no new bugs about that have been filed) in v0.6.0 Which version are you using now?

And yes, there is an issue open for this bug https://github.com/docker/docker/issues/18125 So you will have docker v1.7.1 instead of latest until this bug is fixed.

However, I am working on v0.6.2 of this, so in the coming weeks I'll (hopefully) release a new big release with e.g. HypriotOS support.

And thank you for blogging about this :smile:! (Just one thing, did you forgot to say that one should run kube-config install)

v1k0d3n commented 8 years ago

Version: 0.5.8 and it won't upgrade apparently. i hate the idea of starting over because i have the users/groups and everything else set up exactly as i want them.

# kube-config upgrade
Upgrading the system
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
 alarm is up to date
 aur is up to date
:: Starting full system upgrade...
 there is nothing to do

# kube-config info
Architecture: armv7l
Kernel: Linux 4.1.13
CPU: 4 cores x 900 MHz

Used RAM Memory: 35 MiB
RAM Memory: 923 MiB

Used disk space: 7.2GB (7458184 KB)
Free disk space: 21GB (21417332 KB)

SD Card was built: 16-11-2015 14:12

kubernetes-on-arm:
Latest commit: ccea1ef
Version: 0.5.8

systemd version: 228

I'll look over the write-up again. I can't believe I forgot that! that's a pretty important part, and I thought I captured everything. I'll fix it later today though...thanks for catching that!!

luxas commented 8 years ago

OK, I fully understand that you don't want to upgrade, I know the SD Card method isn't practical in this scenario. In future versions, I want kube-config upgrade to be able to fetch latest code from GitHub and install it.

kube-config disable-node

pacman -R docker
curl -sSL https://s3.amazonaws.com/docker-armv7/docker-1:1.7.1-2-armv7h.pkg.tar.xz > /var/cache/pacman/pkg/docker-1:1.7.1-2-armv7h.pkg.tar.xz
pacman -U  /var/cache/pacman/pkg/docker-1:1.7.1-2-armv7h.pkg.tar.xz

cat > /etc/kubernetes/dynamic-dropins/docker-flannel.conf <<EOF
Restart=on-failure
RestartSec=2
EOF

systemctl daemon-reload
systemctl restart docker

kube-config enable-master

Also I didn't get why you ran hostnamectl explicitely. That's handled by kube-config install. I think kubectl should be in $PATH automatically, if not, notify me :smile:

v1k0d3n commented 8 years ago

no, you're right...kube-config install does set the hostname. i was writing down my notes as i was installing and i guess i need to clean up the document a little bit. running through that a little bit more this week. thanks for reading through it and spotting the corrections!