search

nosada / mkosi-files

Configs and files for creating basic Arch Linux container image on systemd-nspawn using mkosi
Do What The F*ck You Want To Public License
6 stars 0 forks source link

No internet inside container? #6

Closed tmpm697 closed 3 years ago

tmpm697 commented 3 years ago

Steps to produce:

git clone .../mkosi-files
cd mkosi-files
sudo ./update-nspawn-images buildspawn/
sudo systemd-nspawn -UM buildspawn
passwd
exit

# start to container
sudo machinectl start buildspawn
# will failed to start due to # Bind=/ramdisk/scratch/:/scratch/
# I have to comment out Bind=/ramdisk/scratch/:/scratch/ in /etc/systemd/nspawn/buildspawn.nspawn to make it work

# login to container
sudo machinectl login buildspawn
root
<password.

ip ad
ping google.com
# failed

on host I have ve-* interface, and inside container I have another interface without IP address, I would expect the non-lo interface should have and ip address assigned to it and I'll be able to ping google.com from container.

all setup are mostly default and I haven't changed anything.

I use archlinux latest lts-linux, systemd-networkd to manage my wifi connection via wpa_supplicant profile.

nosada commented 3 years ago

@tmpm697 Argh, my approgies, I've not documented about veth... Try below rules for iptables:

-A INPUT -i ve-buildspawn -j ACCEPT

From my experience (sorry not to show you any sources), you must add rule INPUT for veth with -j ACCEPT to communitcate outside container with veth. (i.e. VirtualEthernet=yes in /etc/systemd/nspawn/*.nspawn)

I added above to README.md at https://github.com/nosada/mkosi-files/commit/18bed45f29f3b0cf744446072d2db92b1ad386f7.

Thanks for your reporting :relaxed:

tmpm697 commented 3 years ago

This is weird. I don't have iptables or nftables services running.

How to setup that container will have static IP address?

My setup:

00-mynet.network in /etc/systemd/network that match against a wifi nic wlxxx. (On host).

I use archlinux latest with linux-lts, systemd-networkd to manage network for both host and container. I have wpa-supplicant@wlxxx.service to start wifi profile and the profile in /etc/wpa_supplicant on host. Everything else has default config.

I want build and start buildspawn with a specific ipadress but failed. I followed this article: https://wiki.archlinux.org/index.php/Systemd-networkd#Usage_with_containers

Basically: On host: Add Bridge=br0 to 00-mynet.network Create br0.netdev:

#/etc/systemd/network/MyBridge.netdev
[NetDev]
Name=br0
Kind=bridge

Create bridge network: br0.network

#/etc/systemd/network/MyBridge.network
[Match]
Name=br0

[Network]
DNS=192.168.1.254
Address=192.168.1.87/24
Gateway=192.168.1.1
#192.169.1.1 is also my host's gateway, 192.168.1.0/24 is my host's LAN network.

I build and start buildspawn container as default, the add static ip adress as in article: On container:

#/etc/systemd/network/80-container-host0.network
[Match]
Name=host0

[Network]
DNS=192.168.1.254
Address=192.168.1.94/24
Gateway=192.168.1.1

Ip route on host:

wlxxx (container 192.168.1.xx)
ve-buidspawn@f13 status DOWN, no ipadress

Ip route on container:

lo
host0 DOWN, no ipadress

I think there's something thing wrong with my setup, do you have same issue before?

P/s I'll put code block when in desktop.

Thanks.

nosada commented 3 years ago

I see. I haven't ever used bridge interface with nspawn + networkd, so there's no tips I can show you...

I'll try to use bridge + nspawn + networkd later when I have time.

nosada commented 3 years ago

I haven't tried bridge + nspawn + networkd yet, but found my mistaken (or forgot) point for using veth. If you use veth and then go outside from container, you must set up NAT with iptables (or nftables), as described in https://wiki.archlinux.org/index.php/systemd-nspawn#Use_a_virtual_Ethernet_link.

Below's an example for iptables I used before (cf. https://wiki.archlinux.org/index.php/Simple_stateful_firewall) :

Install this to /etc/iptables/iptables.rules

``` *filter :INPUT DROP [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :TCP - [0:0] :UDP - [0:0] -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -i ve-+ -j ACCEPT -A INPUT -m conntrack --ctstate INVALID -j DROP -A INPUT -p icmp -m icmp --icmp-type 8 -m conntrack --ctstate NEW -j ACCEPT -A INPUT -p udp -m conntrack --ctstate NEW -j UDP -A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j TCP -A INPUT -j REJECT --reject-with icmp-proto-unreachable -A INPUT -p tcp -m recent --set --name TCP-PORTSCAN --mask 255.255.255.255 --rsource -j REJECT --reject-with tcp-reset -A INPUT -p udp -m recent --set --name UDP-PORTSCAN --mask 255.255.255.255 --rsource -j REJECT --reject-with icmp-port-unreachable -A TCP -p tcp -m recent --update --seconds 60 --name TCP-PORTSCAN --mask 255.255.255.255 --rsource -j REJECT --reject-with tcp-reset -A UDP -p udp -m recent --update --seconds 60 --name UDP-PORTSCAN --mask 255.255.255.255 --rsource -j REJECT --reject-with icmp-port-unreachable COMMIT ```

and below's for nftables I'm using now (cf. https://wiki.archlinux.org/index.php/Nftables#Simple_stateful_firewall):

Install this to /etc/nftables.conf

``` table inet filter { chain input { type filter hook input priority filter; policy drop; ct state established,related accept iifname "lo" accept iifname "ve-*" accept ct state invalid drop ip6 nexthdr ipv6-icmp icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, mld-listener-query, mld-listener-report, mld-listener-done, nd-router-solicit, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, ind-neighbor-solicit, ind-neighbor-advert, mld2-listener-report } accept ip protocol icmp icmp type { destination-unreachable, router-advertisement, router-solicitation, time-exceeded, parameter-problem } accept ip protocol igmp accept ip protocol udp ct state new jump UDP ip protocol tcp tcp flags & (fin | syn | rst | ack) == syn ct state new jump TCP ip protocol udp reject ip protocol tcp reject with tcp reset meta nfproto ipv4 counter packets 0 bytes 0 reject with icmp type prot-unreachable } chain forward { type filter hook forward priority filter; policy accept; } chain output { type filter hook output priority filter; policy accept; } chain TCP { } chain UDP { } } ```

tmpm697 commented 3 years ago

I don't use iptables or nftables in my system.

I have ve-xxxx on host and host0 in container but as the archwiki:

"When you start the container, an IP address has to be assigned to both interfaces (on the host and in the container). If you use systemd-networkd on the host as well as in the container, this is done out-of-the-box"

This is not true in my case, both use systemd-networkd but there's no ip under container.

ve-xxx and host0 both in DOWN state.

I use wifi profile with wpa-supplicant but I think it doesn't relate to this case.

nosada commented 3 years ago

Ah, sorry, I misread your comment.

I could maybe reproduce your issue (no IP address on ve-* in host, host0 in container) by doing below (seems to be identical inside systemd) in host:

"networkctl" result

``` # In Host $ networkctl IDX LINK TYPE OPERATIONAL SETUP 1 lo loopback carrier unmanaged 2 enp0s25 ether no-carrier configuring 4 wlan0 wlan routable configured 8 ve-buildspawn ether off unmanaged <- veth for nspawn container 4 links listed. $ networkctl status ve-buildspawn ● 8: ve-buildspawn Link File: /usr/lib/systemd/network/99-default.link Network File: n/a Type: ether State: off (unmanaged) Driver: veth HW Address: 8a:41:5e:XX:XX:XX MTU: 1500 (min: 68, max: 65535) QDisc: noop IPv6 Address Generation Mode: eui64 Queue Length (Tx/Rx): 1/1 Auto negotiation: no Speed: 10Gbps Duplex: full Port: tp Oct 13 00:07:30 HOST systemd-networkd[4744]: ve-buildspawn: Link DOWN Oct 13 00:07:30 HOST systemd-networkd[4744]: ve-buildspawn: Lost carrier Oct 13 00:07:30 HOST systemd-networkd[4744]: ve-buildspawn: DHCPv6 lease lost Oct 13 00:12:10 HOST systemd-networkd[23786]: ve-buildspawn: IPv6 successfully enabled Oct 13 00:12:10 HOST systemd-networkd[23786]: ve-buildspawn: Link UP Oct 13 00:12:12 HOST systemd-networkd[23786]: ve-buildspawn: Gained carrier Oct 13 00:12:14 HOST systemd-networkd[23786]: ve-buildspawn: Gained IPv6LL Oct 13 00:12:24 HOST systemd-networkd[23786]: ve-buildspawn: Link DOWN Oct 13 00:12:24 HOST systemd-networkd[23786]: ve-buildspawn: Lost carrier Oct 13 00:12:24 HOST systemd-networkd[23786]: ve-buildspawn: DHCPv6 lease lost ``` ``` # In container $ networkctl IDX LINK TYPE OPERATIONAL SETUP 1 lo loopback carrier unmanaged 2 host0 ether no-carrier configuring 2 links listed. $ networkctl status host0 ● 2: host0 Link File: n/a Network File: /usr/lib/systemd/network/80-container-host0.network Type: ether State: no-carrier (configuring) HW Address: 7a:4d:b2:YY:YY:YY MTU: 1500 (min: 68, max: 65535) QDisc: noqueue IPv6 Address Generation Mode: eui64 Queue Length (Tx/Rx): 1/1 Auto negotiation: no Speed: 10Gbps Duplex: full Port: tp DHCP6 Client DUID: DUID-EN/Vendor:0000ab11fe986dfda77474780000 Oct 13 00:12:51 buildspawn systemd-networkd[19]: host0: IPv6 successfully enabled Oct 13 00:12:51 buildspawn systemd-networkd[19]: host0: Link UP ```

"ip address show" result

``` # In host $ ip address show dev ve-buildspawn 8: ve-buildspawn@if2: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 8a:41:5e:XX:XX:XX brd ff:ff:ff:ff:ff:ff link-netnsid 0 ``` ``` # In container $ ip address show dev host0 2: host0@if8: mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000 link/ether 7a:4d:b2:YY:YY:YY brd ff:ff:ff:ff:ff:ff link-netnsid 0 ```

Both could do them. After reproduced, I restored 80-container-ve.network then restart systemd-networkd.service and things recovered.

Here's /etc/systemd/nspawn/buildspawn.nspawn I used:

[Exec]
PrivateUsers=true
NotifyReady=true

[Files]
# Commented out below also works fine
Bind=/ramdisk/scratch/:/scratch/

[Network]
Private=yes
VirtualEthernet=yes

Maybe you couldn't treat veth by systemd-networkd properly. Could you try checking your *.network in /etc/systemd/network/ and /usr/lib/systemd/network/?

P.S. I use iwd to manage wifi profile instead of wpa_supplicant, I've used before. But when wpa_supplicant I haven't experience issue like you reported. So I think so too that wpa_supplicant doesn't relate to this issue.

tmpm697 commented 3 years ago

ALL of my configs:

ON HOST:

cat /etc/systemd/network/00-wireless.network
# NOTE: I use static ip address.
# /etc/systemd/network/00-wireless.network
[Match]
Name=wlp1sX
SSID=<my wifi ssid>

[Network]
Address=192.168.Y.XXX/24
Gateway=192.168.E.F
DNS=8.8.8.8 8.8.4.4
cat /etc/wpa_supplicant/wpa_supplicant-wlp1sX.conf
network={
        ssid="<my wifi ssid>"
        scan_ssid=1
        psk=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
}
networkctl
IDX LINK          TYPE     OPERATIONAL SETUP
  1 lo            loopback carrier     unmanaged
  # ...
  8 ve-buildspawn ether    no-carrier  configuring
ip address show dev ve-buildspawn
8: ve-buildspawn@if2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default qlen 1000
    link/ether ee:bc:06:a5:c1:fc brd ff:ff:ff:ff:ff:ff link-netnsid 0
cat /etc/systemd/nspawn/buildspawn.nspawn
[Exec]
PrivateUsers=true
NotifyReady=true

[Files]
# Commented out below also works fine
# Bind=/ramdisk/scratch/:/scratch/

[Network]
Private=yes
VirtualEthernet=yes
networkctl status ve-buildspawn
● 8: ve-buildspawn
                     Link File: /usr/lib/systemd/network/99-default.link
                  Network File: /usr/lib/systemd/network/80-container-ve.network
                          Type: ether
                         State: no-carrier (configuring)
                        Driver: veth
                    HW Address: ee:bc:06:a5:c1:fc
                           MTU: 1500 (min: 68, max: 65535)
                         QDisc: noqueue
  IPv6 Address Generation Mode: eui64
          Queue Length (Tx/Rx): 1/1
              Auto negotiation: no
                         Speed: 10Gbps
                        Duplex: full
                          Port: tp
             DHCP6 Client DUID: DUID-EN/Vendor:0000ab11127bcb58b9aa75fa0000
           Offered DHCP leases: none

Oct 13 14:11:43 localhost systemd-networkd[389]: ve-buildspawn: IPv6 successfully enabled
Oct 13 14:11:43 localhost systemd-networkd[389]: ve-buildspawn: Link UP
cat /usr/lib/systemd/network/99-default.link
#  SPDX-License-Identifier: LGPL-2.1+
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Match]
OriginalName=*

[Link]
NamePolicy=keep kernel database onboard slot path
AlternativeNamesPolicy=database onboard slot path
MACAddressPolicy=persistent

ON CONTAINER:

networkctl
WARNING: systemd-networkd is not running, output will be incomplete.

IDX LINK  TYPE     OPERATIONAL SETUP
  1 lo    loopback n/a         unmanaged
  2 host0 ether    n/a         unmanaged
cat /usr/lib/systemd/network/80-container-ve.network
#  SPDX-License-Identifier: LGPL-2.1+
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# This network file matches the host-side of the virtual Ethernet link
# created by systemd-nspawn's --network-veth switch. See systemd-nspawn(1) for
# details.

[Match]
Name=ve-*
Driver=veth

[Network]
# Default to using a /28 prefix, giving up to 13 addresses per container.
Address=0.0.0.0/28
LinkLocalAddressing=yes
DHCPServer=yes
IPMasquerade=yes
LLDP=yes
EmitLLDP=customer-bridge
cat /usr/lib/systemd/network/80-container-host0.network
#  SPDX-License-Identifier: LGPL-2.1+
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# This network file matches the container-side of the virtual Ethernet link
# created by systemd-nspawn's --network-veth switch. See systemd-nspawn(1) for
# details.

[Match]
Virtualization=container
Name=host0

[Network]
DHCP=yes
LinkLocalAddressing=yes
LLDP=yes
EmitLLDP=customer-bridge

[DHCP]
UseTimezone=yes
ip address show host0
2: host0@if9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a2:32:d8:4f:17:cf brd ff:ff:ff:ff:ff:ff link-netnsid 0
networkctl status host0
WARNING: systemd-networkd is not running, output will be incomplete.

Failed to query link bit rates: The name org.freedesktop.network1 was not provided by any .service files
Failed to query link DHCP leases: The name org.freedesktop.network1 was not provided by any .service files
● 2: host0
                     Link File: n/a
                  Network File: n/a
                          Type: ether
                         State: n/a (unmanaged)
                    HW Address: a2:32:d8:4f:17:cf
                           MTU: 1500 (min: 68, max: 65535)
                         QDisc: noop
  IPv6 Address Generation Mode: eui64
          Queue Length (Tx/Rx): 1/1
              Auto negotiation: no
                         Speed: 10Gbps
                        Duplex: full
                          Port: tp
systemctl status systemd-networkd
● systemd-networkd.service - Network Service
     Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2020-10-13 14:25:08 UTC; 14s ago
TriggeredBy: ● systemd-networkd.socket
       Docs: man:systemd-networkd.service(8)
    Process: 488 ExecStart=/usr/lib/systemd/systemd-networkd (code=exited, status=200/CHDIR)
   Main PID: 488 (code=exited, status=200/CHDIR)

Oct 13 14:25:08 buildspawn systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
Oct 13 14:25:08 buildspawn systemd[1]: Stopped Network Service.
Oct 13 14:25:08 buildspawn systemd[1]: systemd-networkd.service: Start request repeated too quickly.
Oct 13 14:25:08 buildspawn systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Oct 13 14:25:08 buildspawn systemd[1]: Failed to start Network Service.
cat /etc/systemd/network/all-ethernet.network
[Match]
Type=ether

[Network]
DHCP=yes

It's oposite of yours as my host0 is completely in down state and ve-* is in configuring state. I have no idea why it just don't work. Note that I'm using static ip for nic on host but it shouldn't be problem.

nosada commented 3 years ago

How about /usr/lib/systemd/network/80-container-ve.network in host? You showed it in container, not host.

And your log shows systemd-networkd not running in container. Would you show me your journalctl -u systemd-nspawn@buildspawn after starting container?

Also, any changes when you use kernel providedlinux instead of linux-lts?

Updated: last line may not related for this issue because of https://wiki.archlinux.org/index.php/Linux_Containers#Privileged_containers_or_unprivileged_containers...:

The Arch linux, linux-lts and linux-zen kernel packages currently provide out-of-the-box support for unprivileged containers.

tmpm697 commented 3 years ago

on HOST:

cat /usr/lib/systemd/network/80-container-ve.network
#  SPDX-License-Identifier: LGPL-2.1+
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# This network file matches the host-side of the virtual Ethernet link
# created by systemd-nspawn's --network-veth switch. See systemd-nspawn(1) for
# details.

[Match]
Name=ve-*
Driver=veth

[Network]
# Default to using a /28 prefix, giving up to 13 addresses per container.
Address=0.0.0.0/28
LinkLocalAddressing=yes
DHCPServer=yes
IPMasquerade=yes
LLDP=yes
EmitLLDP=customer-bridge

vi /etc/systemd/nspawn/buildspawn.nspawn

[Exec]
PrivateUsers=true
NotifyReady=true

[Files]
# Commented out below also works fine
# Bind=/ramdisk/scratch/:/scratch/

[Network]
Private=yes
VirtualEthernet=yes

journalctl -u systemd-nspawn@buildspawn log here

I use linux-lts, not tried with linux yet.

iptables.service and nftables.service are disabled.

nosada commented 3 years ago

Thank you.

Your 80-container-ve.network seems to be fine, so I think there's no fault for your configuration. It seems there are some problems in networkd & resolved in nspawn container with error: Failed to connect stdout to the journal socket, ignoring: Permission denied. I googled the error message, but there's no result for nspawn.

I don't have any answer / solution for you now... This may not be caused by mkosi or this repository, I think.

You might want to consider asking for your issue in ArchWiki or https://github.com/systemd/systemd/issues.