Docker exits in host network mode

lilveg commented 4 years ago

Hello!

I want to use OpenWRT on an embedded device running docker. I have observed the following behaviour both on my development laptop and on the target device:

$ time docker run --network host openwrtorg/rootfs:x86-64-19.07.2
/etc/preinit: line 6: can't create /sys/devices/system/cpu/microcode/reload: Read-only file system
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level

real    0m3,011s
user    0m0,045s
sys 0m0,028s

I am using time to give additional data, the same thing happens if I run without it. Also, I believe I need to use the --cap-add NET_ADMIN parameter to allow OpenWRT to make changes, but it does not seem to make a difference for now.

If I start the container using docker run --network host -it openwrtorg/rootfs:x86-64-19.07.2 ash I get a shell and I'm able to do some things, but important services like ubus are not running.

Is this a problem with init? /etc/init.d/rcS doesn't exist even though it's referenced by /etc/inittab. Any help locating the source of the problem would be appreciated!

shivarammysore commented 4 years ago

Any guidance on what is wrong?

$ docker run --privileged --network="host" -it openwrtorg/rootfs:x86-64-19.07.3 ash
BusyBox v1.30.1 () built-in shell (ash)
/ # /sbin/init
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
1
/ # /sbin/procd
^C/ # exit

aparcar commented 4 years ago

@shivarammysore I don't really understand your error message.

shivarammysore commented 4 years ago

@aparcar I am trying to run OpenWRT on a docker container. When I run it as:

$ sudo docker run -d -t --name openwrt  --privileged  --network="host" openwrtorg/rootfs:x86-64-19.07.3
$ docker ps -a
CONTAINER ID        IMAGE                              COMMAND             CREATED             STATUS                      PORTS               NAMES
6bf827da90ca        openwrtorg/rootfs:x86-64-19.07.3   "/sbin/init"        20 seconds ago      Exited (1) 16 seconds ago                       openwrt

$ docker logs openwrt
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level

Question is if the above is not the right way to run OpenWRT in a container, then what is the correct procedure?

Thanks

aparcar commented 4 years ago

Why would you want to run it as privileged? OpenWrt does some interface resets on init, so it likely hangs somewhere there

shivarammysore commented 4 years ago

@aparcar If you don't use privileged option, you will get:

Failed to resize receive buffer: Operation not permitted
/etc/preinit: line 6: can't create /sys/devices/system/cpu/microcode/reload: Read-only file system
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level

aparcar commented 4 years ago

Maybe you're missing -i? Or did you try to login via SSH?

ser@dawn:~$ sudo docker run -it --rm openwrtorg/rootfs:x86-64-19.07.3
Unable to find image 'openwrtorg/rootfs:x86-64-19.07.3' locally
x86-64-19.07.3: Pulling from openwrtorg/rootfs
a5a5938fa80f: Pull complete 
Digest: sha256:2a7eea411a12c882705bc8fecb2810cd7c0e795e3eca30a3e546dc2bebda547d
Status: Downloaded newer image for openwrtorg/rootfs:x86-64-19.07.3
Failed to resize receive buffer: Operation not permitted
/etc/preinit: line 6: can't create /sys/devices/system/cpu/microcode/reload: Read-only file system
ip: RTNETLINK answers: Operation not permitted
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
ip: can't send flush request: Operation not permitted
ip: SIOCSIFFLAGS: Operation not permitted
Please press Enter to activate this console.

BusyBox v1.30.1 () built-in shell (ash)

/bin/ash: can't access tty; job control turned off
  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 19.07.3, r11063-85e04e9f46
 -----------------------------------------------------
=== WARNING! =====================================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------
root@f92852f80cae:/# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=59 time=5.135 ms
64 bytes from 1.1.1.1: seq=1 ttl=59 time=5.843 ms
64 bytes from 1.1.1.1: seq=2 ttl=59 time=4.069 ms
^C64 bytes from 1.1.1.1: seq=3 ttl=59 time=5.371 ms

shivarammysore commented 4 years ago

@aparcar thanks for the hint. -i is the interactive shell mode. Yes it does work:

sudo docker run --privileged -p 80:80 -p 443:443 -it --rm openwrtorg/rootfs:x86-64-19.07.3
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level

Please press Enter to activate this console.

BusyBox v1.30.1 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 19.07.3, r11063-85e04e9f46
 -----------------------------------------------------
=== WARNING! =====================================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------
root@OpenWrt:/# netstat -lpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      559/uhttpd
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      347/dnsmasq
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      384/dropbear
tcp        0      0 :::80                   :::*                    LISTEN      559/uhttpd
tcp        0      0 fe80::42:acff:fe11:2:53 :::*                    LISTEN      347/dnsmasq
tcp        0      0 ::1:53                  :::*                    LISTEN      347/dnsmasq
tcp        0      0 :::22                   :::*                    LISTEN      384/dropbear
udp        0      0 127.0.0.1:53            0.0.0.0:*                           347/dnsmasq
udp        0      0 fe80::42:acff:fe11:2:53 :::*                                347/dnsmasq
udp        0      0 ::1:53                  :::*                                347/dnsmasq
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     26470130 75/ubusd            /var/run/ubus.sock
root@OpenWrt:/# halt

Now, If I run

$ curl http://localhost 
curl: (56) Recv failure: Connection reset by peer

I also tried to reach the OpenWRT console via browser and it does not work. What I also confirmed is that --network=host is also not working. The instance just exists.

shivarammysore commented 4 years ago

I see that

root@OpenWrt:/# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
ping: sendto: Network unreachable
root@OpenWrt:/# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
root@OpenWrt:/#

did you change any network settings when you configured the docker instance?

Thanks

aparcar commented 4 years ago

No. But I don't let it use the host network. What is you use case? Running docker privileged and using host networks sound you should install OpenWrt in the first place?

shivarammysore commented 4 years ago

My use case is to run OpenWRT on Fedora CoreOS.

/

On Jul 16, 2020, at 11:41 PM, Paul Spooren notifications@github.com wrote:

No. But I don't let it use the host network. What is you use case? Running docker privileged and using host networks sound you should install OpenWrt in the first place?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

shivarammysore commented 4 years ago

@aparcar When we run OpenWRT as a container, without the --network=host option, we will not have access to the physical interfaces on the box. In a standard mode without host networking, we will have just the eth0 interface on the container and hence, services cannot be used or accessed. I have more detailed documentation on how host networking works @ https://github.com/shivarammysore/ovs/blob/master/docs/docker_networking.rst

root@OpenWrt:/etc/config# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether e6:ee:52:08:bf:15 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e4ee:52ff:fe08:bf15/64 scope link 
       valid_lft forever preferred_lft forever

I have on my host system a host of physical network interfaces.

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0d:48:59:00:c0 brd ff:ff:ff:ff:ff:ff
    altname enp2s0
    inet 10.20.30.208/16 brd 10.20.255.255 scope global dynamic noprefixroute eth0
       valid_lft 4999sec preferred_lft 4999sec
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0d:48:59:00:c1 brd ff:ff:ff:ff:ff:ff
    altname enp3s0
4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0d:48:59:00:c2 brd ff:ff:ff:ff:ff:ff
    altname enp4s0
5: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0d:48:59:00:c3 brd ff:ff:ff:ff:ff:ff
    altname enp5s0
6: eth4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0d:48:59:00:c4 brd ff:ff:ff:ff:ff:ff
    altname enp10s0f0
7: eth5: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0d:48:59:00:c5 brd ff:ff:ff:ff:ff:ff
    altname enp10s0f1
8: eth6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0d:48:59:00:c6 brd ff:ff:ff:ff:ff:ff
    altname enp11s0f0
9: eth7: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0d:48:59:00:c7 brd ff:ff:ff:ff:ff:ff
    altname enp11s0f1
10: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b4:82:c5:58:0b:70 brd ff:ff:ff:ff:ff:ff
    altname wlp9s0

I want to use these network interfaces with OpenWRT. This will be similar to the work I have done for Open vSwitch - https://github.com/shivarammysore/ovs

openwrt / docker

Docker exits in host network mode #56