Port forwarding with firewalld is not persistent

[p3lim]

Steps to reproduce:

Run a container that publishes a port, e.g. nginx:

podman run --rm -p 8000:80 docker.io/nginx:latest

This yield's the following firewalld policy:

netavark_portfwd (active)
  priority: -1
  target: CONTINUE
  ingress-zones: ANY
  egress-zones: ANY
  services:
  ports:
  protocols:
  masquerade: no
  forward-ports:
        port=8000:proto=tcp:toport=80:toaddr=10.88.0.8
  source-ports:
  icmp-blocks:
  rich rules:

If I now run firewall-cmd --reload/systemctl reload firewalld because I did some other firewall changes (not necessarily related to this container), the port forwarding is now gone:

netavark_portfwd (active)
  priority: -1
  target: CONTINUE
  ingress-zones: ANY
  egress-zones: ANY
  services:
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

I can remedy this myself by running podman network reload --all, but this should be handled by netavark/podman itself.

Tested with:

• firewalld 1.1.1 (from Ubuntu 22.04 repos)
• podman 4.5.1 (from kubic)
• netavark 1.3.0 (from kubic)

Let me know if this should be raised in the podman repo instead.

[Luap99]

The problem here is that we can either mark it persistent or not. And if it is marked persistent it mean firewalld will apply them after a reboot again right? Podman has no deamon, we cannot ever have things staying around after a possible reboot so we do not use persistent AFAIK.

And while it would be cool to have this fixed when using the firewalld backend, iptables and a possible direct nft implementation would still suffer from that.

In my ideal world we could say firewalld please keep the rules on reaload/restart but do not add them back, e.g. on reboot.

[p3lim]

You do have full autonomy with the policies you create yourself, nothing is stopping you from pruning them before you add anything, e.g. after a reboot and after a container is started.

And this would be firewalld-specific, direct nft/iptables should not be considered as a part of this issue.

[Luap99]

You do have full autonomy with the policies you create yourself, nothing is stopping you from pruning them before you add anything, e.g. after a reboot and after a container is started.

Of course we can add/remove any rules but that is not the problem, the problem is we have no process to do it. Podman is not a daemon, it will not be started after a boot unless someone runs a podman command.

Which means if we were to mark them persistent and the system crashes firewalld will add the rules back after a boot. However there is exactly 0 guarantee that any podman command was started after the boot so we have no way of removing the old rules. The only way do that would be to create a new systemd unit and a special command to flush all rules on startup. But this is extra work for us and then we have nice race conditions to take care of as this would need to be run after firewalld created the rules but before podman starts containers and calls netavark to create rules for the container.

[p3lim]

Alternatively - listen for firewalld reload events and add them back after

[robbmanes]

Alternatively - listen for firewalld reload events and add them back after

Noting we do tap dbus for firewalld rulesets but as mentioned, only when a podman command is ran. So actively listening for dbus notifications of a firewalld reload, unless you had some other mechanism in mind, isn't possible because there's no podman process still alive at that point. The network configuration, to the best of my knowledge, configures and exits, meaning the changes on firewalld can't be notified to any existing processes really because anything with the power to make the change has already exited. Sans some kind of hook in firewalld that prevents the rules we did not persistently add via netavark from being removed during reload, I don't know how this would be possible, in agreement with:

In my ideal world we could say firewalld please keep the rules on reaload/restart but do not add them back, e.g. on reboot.

Just my 2 cents; I can't think of a way currently to make this work as at the time of reload we'd have no processes alive to handle re-adding rulesets.

[Luap99]

This was fixed in https://github.com/containers/netavark/pull/840, in netavark v1.9.

See https://blog.podman.io/2023/11/new-netavark-firewalld-reload-service/ for info on how to use it.