[Feature Request] Podman containers within macvlan

databreach commented 3 years ago

Currently, unifi-systemd-units runs the nested podman containers in a bridged network. For a more secure and flexible solution, I would like the podman containers to run within a completely isolated network stack using macvlan.

Similar functionality is provided by @boostchicken , such as for Adguard Home. However, I can’t seem to figure out how to achieve the same with unifi-systemd-units.

It would be great if this feature could be added to unifi-systemd-units.

ntkme commented 3 years ago

The unifi-systemd container already has macvlan driver installed.

You will need to write a systemd unit to create a macvlan interface. You can use the macvlan script by @boostchicken if you like, just write a OneShot type systemd unit that call the script.

databreach commented 3 years ago

If I understand correct, I need to do the following:

Ignore the 05-install-cni-plugins.sh script as unifi-systemd already includes the CNI plugins.
Trigger the 10-dns.sh script with a OneShottype systemd unit.
Copy the 20-dns.sh parameters to the ExecStart command in the service unit (e.g. container-adguardhome@.service)

I wasn't expecting it to be that simple, but will give a try.

ntkme commented 3 years ago

More or less like that, but you will probably need to modify 10-dns.sh a little bit:

https://github.com/boostchicken/udm-utilities/blob/b4b66b9701c39058ea7e96f3fd5e46d27b0c8c45/dns-common/on_boot.d/10-dns.sh#L34-L38

The cni plugin in fedora is installed at /usr/libexec/cni/, not /opt/cni/bin, so it will fail the check above. Make sure you modify the script to remove this section. Also remove the container start part from the script as we can use systemd to manage containers.

You will need to copy 20-dns.conflist to /etc/cni/net.d/, this put this in the same oneshot unit with a second ExecStart. You can have multiple ExecStart in one unit to run multiple command.

Something like this:

[Unit]
# Make sure we have the right order of units
Before=container-adguardhome@.service

[Service]
# 10-dns.sh should be executed on host like below as our fedora-minimum container is missing a few commands.
ExecStart=/usr/bin/podman run --rm --net host --volume /mnt/data/ssh/id_rsa:/root/.ssh/id_rsa:ro --volume /var/run/ssh_proxy_port:/etc/unifi-os/ssh_proxy_port:ro ghcr.io/ntkme/unifi-ssh-proxy /bin/sh /mnt/data/10-dns.sh
# Now install the conflist
ExexStart=/usr/bin/cp /mnt/data/etc/cni/net.d/20-dns.conflist /etc/cni/net.d/20-dns.conflist

If you look at 20-dns.conflist, you will see that the name of the network is dns, so:

systemctl enable --now container-adguardhome@dns.service

There might be small details that I missed, but this is the overall idea. Let me know if you have questions.

databreach commented 3 years ago

Thank you very much for the support so far. Unfortunately, I am unable to get it to work. As such, I have started from scratch and used your test files for validation. Your help would be appreciated.

Installed unifi-systemd
Installed unifi-systemd-units (rebooted afterwards)
Executed test_cni-dhcp.sh (success)
Executed test_cni-podman-bridge.sh (success)
Executed test_container-adguardhome.sh (failure see message below)
Executed test_container-boringtun.sh (failure see message below)
Executed test_container-homebridge.sh (failure see message below)

Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded.
level=error msg="error unmounting /var/lib/containers/storage/overlay/10d9d474fea4da4671b56431a5659728dbc29178778c89c6ac581c86762f8faf/merged: invalid argument"
Error: error mounting storage for container 26e746738c5e74ffdf82dc1610ca5cf5d8ae88df78ec10535df614a7aac6d412: error creating overlay mount to /var/lib/containers/storage/overlay/10d9d474fea4da4671b56431a5659728dbc29178778c89c6ac581c86762f8faf/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/3E35GQOKVFWPRMZYLGIADPFBO5:/var/lib/containers/storage/overlay/l/XV4HOR6G7AQFC6ULKKMOFIDRAS:/var/lib/containers/storage/overlay/l/2P3ORZOK2VZ4FHDKW77XW3DYIC:/var/lib/containers/storage/overlay/l/GPOPKCCZBRVFFP2CA7LLTEXYZQ,upperdir=/var/lib/containers/storage/overlay/10d9d474fea4da4671b56431a5659728dbc29178778c89c6ac581c86762f8faf/diff,workdir=/var/lib/containers/storage/overlay/10d9d474fea4da4671b56431a5659728dbc29178778c89c6ac581c86762f8faf/work": invalid argument
container-adguardhome@host.service: Control process exited, code=exited, status=126/n/a

Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded.
level=error msg="error unmounting /var/lib/containers/storage/overlay/c321084097a08cf41ca3a096e5653a8183957b32007e2d9f4a5d7793534030aa/merged: invalid argument"
Error: error mounting storage for container 7d671eebaa06f6ec36357895be15ec0e340e2e9b851ced91262b54a32ae854e3: error creating overlay mount to /var/lib/containers/storage/overlay/c321084097a08cf41ca3a096e5653a8183957b32007e2d9f4a5d7793534030aa/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/UFAQJ5O4KJ2NHNFIX5PF7RVNEG:/var/lib/containers/storage/overlay/l/LNAONTSBTRMTP4ZKL5Z4TO2UWA,upperdir=/var/lib/containers/storage/overlay/c321084097a08cf41ca3a096e5653a8183957b32007e2d9f4a5d7793534030aa/diff,workdir=/var/lib/containers/storage/overlay/c321084097a08cf41ca3a096e5653a8183957b32007e2d9f4a5d7793534030aa/work": invalid argument
container-boringtun@wg0.service: Control process exited, code=exited, status=126/n/a

Your kernel does not support pids limit capabilities or the cgroup is not mounted. PIDs limit discarded.
level=error msg="error unmounting /var/lib/containers/storage/overlay/fc665ad4b3712ec3cb3aa91ce1db4096af77eb636cb6bab1a32dfb80bb96bae0/merged: invalid argument"
Error: error mounting storage for container 0ed55305dd967740df32c9912df517f44527e4501946a4ad10d43f4b927d6149: error creating overlay mount to /var/lib/containers/storage/overlay/fc665ad4b3712ec3cb3aa91ce1db4096af77eb636cb6bab1a32dfb80bb96bae0/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/3N63MJEYE2I64WJGRVMEKMAHQX:/var/lib/containers/storage/overlay/l/B4P26QGIUSCWQ7VYGW4BAMNWHY:/var/lib/containers/storage/overlay/l/RY2KRG7L6KDNOM26WIIZGJLROO:/var/lib/containers/storage/overlay/l/5DCA35VRA5NLCKFOBYWVLV4AE3:/var/lib/containers/storage/overlay/l/OXE4NQVSNL22AC6A3UIRHAJHNC:/var/lib/containers/storage/overlay/l/AKWACRUJTLGG5NAWCN6D2OBA4H:/var/lib/containers/storage/overlay/l/YRECEAOQHUPD52IOP7TI7DDBBQ:/var/lib/containers/storage/overlay/l/TRSBNF5FVKMCTMKJMI2GMQRT7R:/var/lib/containers/storage/overlay/l/4LBD3Y2OEPNULO5J2CH5IIM3GS:/var/lib/containers/storage/overlay/l/GPOPKCCZBRVFFP2CA7LLTEXYZQ,upperdir=/var/lib/containers/storage/overlay/fc665ad4b3712ec3cb3aa91ce1db4096af77eb636cb6bab1a32dfb80bb96bae0/diff,workdir=/var/lib/containers/storage/overlay/fc665ad4b3712ec3cb3aa91ce1db4096af77eb636cb6bab1a32dfb80bb96bae0/work": invalid argument
container-homebridge@host.service: Control process exited, code=exited, status=126/n/a

ntkme commented 3 years ago

What firmware version do you have? If you are not on 1.10.0 beta, you need to upgrade to the beta, or run the command below before you install unifi-systemd. This is also in README:

For UniFi Dream Machine firmware <1.10.0, pull the legacy container with the support for legacy kernel.

if podman exec unifi-os dpkg --compare-versions "$(uname -r)" lt 4.19 2>/dev/null; then
  podman pull ghcr.io/ntkme/systemd-podman:legacy && podman tag ghcr.io/ntkme/systemd-podman:legacy ghcr.io/ntkme/systemd-podman:latest
fi

If you have already installed. You can also do:

unifi-systemd stop
podman image rm ghcr.io/ntkme/systemd-podman:latest
podman pull ghcr.io/ntkme/systemd-podman:legacy
podman tag ghcr.io/ntkme/systemd-podman:legacy ghcr.io/ntkme/systemd-podman:latest
unifi-systemd start

databreach commented 3 years ago

Upgrading to the beta firmware (1.10.0-12) did indeed fix it!

I guess that I misread your README, thinking it was referring to <1.1.0 or something... not realizing you were actually referring to beta firmware <1.10.0.

I will recreate the macvlan configuration and files again, and let you know how it goes.

databreach commented 3 years ago

It is working as expected. For those who are interested to know how, here is how I do it.

Create CNI Macvlan Instead of dedicating a macvlan to a single service (e.g. Adguard Home), I make use of a services macvlan. This macvlan can host multiple podman containers, such as Adguard Home and a SNI proxy.

Add a network to your UDM Pro with a VLAN ID and set its gateway and DHCP range. I am using VLAN ID 130 with IP addresses 192.168.130.x.
```
UDM Pro > Settings > Networks > Add New Network
```
Create a file cni-podman-macvlan.sh in the /mnt/data/etc directory
```
vi /mnt/data/etc/cni-podman-macvlan.sh
```
Insert the following code and change VLAN and IP addresses to your needs

#!/bin/sh

## configuration variables:
VLAN=130
IPV4_IP="192.168.130.0/24"
# This is the IP address range of the macvlan. You may want to set it to match
# your own network structure such as 10.0.5.0/24 or similar.
IPV4_GW="192.168.130.1"
# As above, this should match the gateway of the VLAN for the container
# network as above which is usually the .1 address of the IPV4_IP range

# if you want IPv6 support, generate a ULA, select an IP range for the macvlan
# and an appropriate gateway address on the same /64 network. Make sure that
# the conflist is updated appropriately. It will need the IP range and GW
# added along with a ::/0 route. Also make sure that additional --dns options
# are passed to podman with your IPv6 DNS IPs when deploying the container for
# the first time. You will also need to configure your VLAN to have a static
# IPv6 block.

# IPv6 Also works with Prefix Delegation from your provider. The gateway is the
# IP of br(VLAN) and you can pick any ip address within that subnet that dhcpv6
# isn't serving
IPV6_IP=""
IPV6_GW=""

if ! podman exec -it unifi-systemd sh -c "test -f /usr/libexec/cni/macvlan"; then
    echo "Error: CNI plugins not found. Please check if you have installed unifi-systemd."
    exit 1
fi

# set VLAN bridge promiscuous
ip link set br${VLAN} promisc on

# create macvlan bridge and add IPv4 IP
ip link add br${VLAN}.mac link br${VLAN} type macvlan mode bridge
ip addr add ${IPV4_GW}/24 dev br${VLAN}.mac noprefixroute

# (optional) add IPv6 IP to VLAN bridge macvlan bridge
if [ -n "${IPV6_GW}" ]; then
  ip -6 addr add ${IPV6_GW} dev br${VLAN}.mac noprefixroute
fi

# set macvlan bridge promiscuous and bring it up
ip link set br${VLAN}.mac promisc on
ip link set br${VLAN}.mac up

# add IPv4 route to macvlan
ip route del ${IPV4_IP}
ip route add ${IPV4_IP} dev br${VLAN}.mac src ${IPV4_GW}

# (optional) add IPv6 route to macvlan
if [ -n "${IPV6_IP}" ]; then
  ip route del ${IPV6_IP}
  ip -6 route add ${IPV6_IP} dev br${VLAN}.mac src ${IPV6_GW}
fi

# Make DNSMasq listen to the container network for split horizon or conditional forwarding
if ! grep -qxF interface=br$VLAN.mac /run/dnsmasq.conf.d/custom.conf; then
    echo interface=br$VLAN.mac >> /run/dnsmasq.conf.d/custom.conf
    kill -9 `cat /run/dnsmasq.pid`
fi

Give the file cni-podman-macvlan.sh execution permissions
```
chmod a+x /mnt/data/etc/cni-podman-macvlan.sh
```
Create a file cni-podman-macvlan@.service in the /mnt/data/etc/systemd/system directory
```
vi /mnt/data/etc/systemd/system/cni-podman-macvlan@.service
```
Insert the following code and change MAC and IP addresses to your needs

[Unit]
Description=Configure CNI %j-%i Network
DefaultDependencies=no
Conflicts=shutdown.target
Before=sysinit.target shutdown.target

[Service]
# macvlan.sh should be executed on host like below as our fedora-minimum container is missing a few commands.
ExecStart=/usr/bin/podman run --rm --net host --volume /mnt/data/ssh/id_rsa:/root/.ssh/id_rsa:ro --volume /var/run/ssh_proxy_port:/etc/unifi-os/ssh_proxy_port:ro ghcr.io/ntkme/unifi-ssh-proxy /bin/sh /mnt/data/etc/cni-podman-macvlan.sh
# Now install the conflist
ExecStart=/bin/sh -c 'echo "$1" | tee /etc/cni/net.d/%j-%i.conflist' -- '{"cniVersion": "0.4.0","name":"%j-%i","plugins":[{"type":"%j","mode":"bridge","master":"%i","mac":"PUT YOUR GENERATED OWN MAC HERE","ipam":{"type":"host-local","ranges":[[{"subnet":"192.168.130.0/24","rangeStart":"192.168.130.2","rangeEnd":"192.168.130.254","gateway":"192.168.130.1"}]],"routes":[{"dst":"0.0.0.0/0"}]}},{"type":"tuning","capabilities":{"mac":true}}]}'
ExecStop=/bin/rm -f /etc/cni/net.d/%j-%i.conflist
Type=oneshot
RemainAfterExit=yes

[Install]
WantedBy=sysinit.target

Enable the CNI Macvlan container

podman exec unifi-systemd systemctl enable --now cni-podman-macvlan@br130.service

Deploy Adguard Home

Create the Adguard Home data and config directories

mkdir -p /mnt/data/var/lib/adguardhome /mnt/data/etc/adguardhome

Edit the content of file container-adguardhome@.service in the /mnt/data/etc/systemd/system directory
```
vi /mnt/data/etc/systemd/system/container-adguardhome@.service
```
Insert the following code and change MAC and IP addresses to your needs

[Unit]
Description=Podman container-adguardhome@.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target
RequiresMountsFor=/var/lib/containers/storage /var/run/containers/storage
ConditionPathIsDirectory=/mnt/data/var/lib/adguardhome
ConditionPathIsDirectory=/mnt/data/etc/adguardhome

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm -f %t/container-adguardhome.pid %t/container-adguardhome.ctr-id
ExecStart=/usr/bin/podman run --conmon-pidfile %t/container-adguardhome.pid --cidfile %t/container-adguardhome.ctr-id --cgroups=no-conmon --replace -d --rm --name adguardhome-%i --net %i --ip 192.168.130.2 --mac-address PUT YOUR GENERATED OWN MAC HERE --volume /mnt/data/var/lib/adguardhome:/opt/adguardhome/work --volume /mnt/data/etc/adguardhome:/opt/adguardhome/conf docker.io/adguard/adguardhome
ExecStop=/usr/bin/podman stop --ignore --cidfile %t/container-adguardhome.ctr-id -t 10
ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile %t/container-adguardhome.ctr-id
PIDFile=%t/container-adguardhome.pid
Type=forking

[Install]
WantedBy=multi-user.target

Enable the Adguard Home container

podman exec unifi-systemd systemctl enable --now container-adguardhome@macvlan-br130.service

Configure Firewall Don't forget to set your desired firewall rules in the UDM Pro to secure traffic to/from the macvlan and its containers

   UDM Pro > Settings > Security > Internet Threat Management > Firewall

ntkme commented 3 years ago

Glad to hear that it worked for you.

I will consider packing it as a few predefined systemd services to make it easier to use when I got time.

Thanks for being an pioneer on this!

ntkme / unifi-systemd-units

[Feature Request] Podman containers within macvlan #1