search

hypriot / image-builder-rpi

SD card image for Raspberry Pi with Docker: HypriotOS
http://blog.hypriot.com/post/how-to-get-docker-working-on-your-favourite-arm-board-with-hypriotos/
MIT License
1.07k stars 168 forks source link

Kernel error when running Docker service #166

Closed gesellix closed 5 years ago

gesellix commented 7 years ago

Please see https://github.com/docker/example-voting-app/pull/64#issuecomment-288873621 and https://github.com/docker/docker/issues/27833#issuecomment-282596696 for details.

In essence: I'm trying to run the Docker example voting app stack on Raspberry Pi, but Kernel errors arise:

...
[ 2102.873233] BUG: using smp_processor_id() in preemptible [00000000] code: node/3348
[ 2102.873250] caller is debug_smp_processor_id+0x18/0x24
[ 2102.873269] CPU: 2 PID: 3348 Comm: node Not tainted 4.4.50-hypriotos-v7+ #1
[ 2102.873279] Hardware name: BCM2709
[ 2102.873306] [<80019468>] (unwind_backtrace) from [<80014a14>] (show_stack+0x20/0x24)
[ 2102.873331] [<80014a14>] (show_stack) from [<803362dc>] (dump_stack+0xbc/0x108)
[ 2102.873358] [<803362dc>] (dump_stack) from [<80350e34>] (check_preemption_disabled+0x104/0x134)
[ 2102.873381] [<80350e34>] (check_preemption_disabled) from [<80350e7c>] (debug_smp_processor_id+0x18/0x24)
[ 2102.873443] [<80350e7c>] (debug_smp_processor_id) from [<7f40bb90>] (ip_vs_in.part.2.constprop.9+0x2fc/0x75c [ip_vs])
[ 2102.873573] [<7f40bb90>] (ip_vs_in.part.2.constprop.9 [ip_vs]) from [<7f40c074>] (ip_vs_local_request4+0x40/0x44 [ip_vs])
[ 2102.873656] [<7f40c074>] (ip_vs_local_request4 [ip_vs]) from [<8050e014>] (nf_iterate+0x80/0x90)
...

I checked the same setup on Hypriot 1.4.0 (produces errors), and Raspbian Lite 2017-03-02 (no errors). I can trigger the error by running the commands below. It's not relevant whether to deploy the stack on a single node or on several nodes. Enabling swap (disabled on Hypriot by default) doesn't change anything. I also suspect that it's enough to only start a single service only.

Details for the mentioned Raspbian setup:

Kernel Version: 4.4.50-v7+
Operating System: Raspbian GNU/Linux 8 (jessie)

 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 11:32:23 2017
 OS/Arch:      linux/arm
 Experimental: true

Commands to reproduce:

git clone https://github.com/gesellix/example-voting-app.git
cd example-voting-app/
git checkout armhf 
docker swarm init
docker stack deploy -c docker-stack.armhf.yml example

Then watch the Kernel messages (e.g. via dmesg --follow).

If you need other details to reproduce the error, I'm happy to help!

kalgecin commented 7 years ago

can also confirm this issue

Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 1330.101398] ip_set: protocol 6
[ 1330.394270] openvswitch: Open vSwitch switching datapath
[ 1330.401109] device datapath entered promiscuous mode
[ 1330.430902] vxlan: Cannot bind port 47294, err=-98
[ 1330.431270] device vxlan-47294 entered promiscuous mode
[ 1330.431868] device vxlan-47294 left promiscuous mode
[ 1330.667749] device vethwedu entered promiscuous mode
[ 1330.672287] device vethwedu left promiscuous mode
[ 1330.672336] weave: port 1(vethwedu) entered disabled state
[ 1330.754503] IPv6: ADDRCONF(NETDEV_UP): vethwe-bridge: link is not ready
[ 1330.759527] IPv6: ADDRCONF(NETDEV_CHANGE): vethwe-bridge: link becomes ready
[ 1330.798908] device vethwe-datapath entered promiscuous mode
[ 1330.804823] device vethwe-bridge entered promiscuous mode
[ 1331.010595] weave: port 1(vethwe-bridge) entered forwarding state
[ 1331.010664] weave: port 1(vethwe-bridge) entered forwarding state
[ 1346.044984] weave: port 1(vethwe-bridge) entered forwarding state
[ 1406.516672] ------------[ cut here ]------------
[ 1406.521583] WARNING: CPU: 2 PID: 16467 at kernel/sched/core.c:2966 preempt_count_add+0xfc/0x118()
[ 1406.530932] DEBUG_LOCKS_WARN_ON((preempt_count() & PREEMPT_MASK) >= PREEMPT_MASK - 10)
[ 1406.539107] Modules linked in:
[ 1406.542341]  xt_NFLOG nfnetlink_log xt_tcpudp veth dummy vport_vxlan vxlan udp_tunnel ip6_udp_tunnel openvswitch nf_defrag_ipv6 ip_set_hash_ip xt_set ip_set xt_comment xt_mark nfsd nf_conntrack_netlink nfnetlink ipt_MASQUERADE nf_nat_masquerade_ipv4 bnep iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables hci_uart btbcm bluetooth xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc overlay brcmfmac brcmutil cfg80211 rfkill snd_bcm2835 snd_pcm snd_timer snd bcm2835_gpiomem bcm2835_wdt uio_pdrv_genirq uio ipv6
[ 1406.594605] CPU: 2 PID: 16467 Comm: kube-proxy Not tainted 4.4.50-hypriotos-v7+ #1
[ 1406.602536] Hardware name: BCM2709
[ 1406.606144] [<80019468>] (unwind_backtrace) from [<80014a14>] (show_stack+0x20/0x24)
[ 1406.614277] [<80014a14>] (show_stack) from [<803362dc>] (dump_stack+0xbc/0x108)
[ 1406.621945] [<803362dc>] (dump_stack) from [<8002672c>] (warn_slowpath_common+0x8c/0xc8)
[ 1406.630426] [<8002672c>] (warn_slowpath_common) from [<800267a8>] (warn_slowpath_fmt+0x40/0x48)
[ 1406.639549] [<800267a8>] (warn_slowpath_fmt) from [<8005005c>] (preempt_count_add+0xfc/0x118)
[ 1406.648480] [<8005005c>] (preempt_count_add) from [<805bf608>] (_raw_spin_lock+0x20/0x60)
[ 1406.742586] [<805bf608>] (_raw_spin_lock) from [<7f2f145c>] (nf_conntrack_set_hashsize+0xa4/0x200 [nf_conntrack])
[ 1406.840782] [<7f2f145c>] (nf_conntrack_set_hashsize [nf_conntrack]) from [<80043530>] (param_attr_store+0x6c/0xc4)
[ 1406.938944] [<80043530>] (param_attr_store) from [<80042864>] (module_attr_store+0x30/0x3c)
[ 1407.035188] [<80042864>] (module_attr_store) from [<801dfd1c>] (sysfs_kf_write+0x54/0x58)
[ 1407.131415] [<801dfd1c>] (sysfs_kf_write) from [<801df4f4>] (kernfs_fop_write+0xc8/0x1c8)
[ 1407.227684] [<801df4f4>] (kernfs_fop_write) from [<8016ad94>] (__vfs_write+0x34/0xe8)
[ 1407.323713] [<8016ad94>] (__vfs_write) from [<8016b658>] (vfs_write+0xa0/0x1a8)
[ 1407.419786] [<8016b658>] (vfs_write) from [<8016bf78>] (SyS_write+0x4c/0xa0)
[ 1407.471273] [<8016bf78>] (SyS_write) from [<8000fc40>] (ret_fast_syscall+0x0/0x1c)
[ 1407.566662] ---[ end trace 28174df7e9a94279 ]---
[ 1407.620012] BUG: scheduling while atomic: kube-proxy/16467/0x00000401
[ 1407.670158] Modules linked in: xt_nat xt_recent xt_NFLOG nfnetlink_log xt_tcpudp veth dummy vport_vxlan vxlan udp_tunnel ip6_udp_tunnel openvswitch nf_defrag_ipv6 ip_set_hash_ip xt_set ip_set xt_comment xt_mark nfsd nf_conntrack_netlink nfnetlink ipt_MASQUERADE nf_nat_masquerade_ipv4 bnep iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables hci_uart btbcm bluetooth xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc overlay brcmfmac brcmutil cfg80211 rfkill snd_bcm2835 snd_pcm snd_timer snd bcm2835_gpiomem bcm2835_wdt uio_pdrv_genirq uio ipv6
[ 1408.071840] Preemption disabled at:[<  (null)>]   (null)

[ 1408.164585] CPU: 2 PID: 16467 Comm: kube-proxy Tainted: G        W       4.4.50-hypriotos-v7+ #1
[ 1408.258163] Hardware name: BCM2709
[ 1408.304621] [<80019468>] (unwind_backtrace) from [<80014a14>] (show_stack+0x20/0x24)
[ 1408.393730] [<80014a14>] (show_stack) from [<803362dc>] (dump_stack+0xbc/0x108)
[ 1408.482245] [<803362dc>] (dump_stack) from [<8010c4d4>] (__schedule_bug+0xac/0xd0)
[ 1408.571165] [<8010c4d4>] (__schedule_bug) from [<805bbc60>] (__schedule+0x6a0/0x750)
[ 1408.664937] [<805bbc60>] (__schedule) from [<805bbf1c>] (schedule+0x58/0xb8)
[ 1408.713230] [<805bbf1c>] (schedule) from [<80014210>] (do_work_pending+0x3c/0xd4)
[ 1408.801336] [<80014210>] (do_work_pending) from [<8000fc68>] (slow_work_pending+0xc/0x20)
[ 1440.137173] device vxlan-6784 entered promiscuous mode