Closed chrisportela closed 3 years ago
thatsmydoing
commented
3 years ago I'm not really a maintainer and just contributed a bug fix here so I know as much as you do.
Have you tried it on a non-hardened kernel config? Did this work before and broke recently?
Could you put the contents of the file returned by this command?
nix-instantiate --eval '<nixpkgs/nixos>' -A config.systemd.services.firewall.serviceConfig.ExecStart
chrisportela
commented
3 years ago Yea I realize that. I do still appreciate the help. From what I could see the file looks like it's using the argument correctly. I had similar issues trying to setup cake which required a specific kernel module.
I will try not using the hardened version. I think that has been a source of a ton of problems. If possible it'd be nice to figure things out while still using that.
Here is what was in that file for me
#! /nix/store/9ywr69qi622lrmx5nn88gk8jpmihy0dz-bash-4.4-p23/bin/bash -e
# Helper command to manipulate both the IPv4 and IPv6 tables.
ip46tables() {
iptables -w "$@"
ip6tables -w "$@"
}
# Flush the old firewall rules. !!! Ideally, updating the
# firewall would be atomic. Apparently that's possible
# with iptables-restore.
ip46tables -D INPUT -j nixos-fw 2> /dev/null || true
for chain in nixos-fw nixos-fw-accept nixos-fw-log-refuse nixos-fw-refuse; do
ip46tables -F "$chain" 2> /dev/null || true
ip46tables -X "$chain" 2> /dev/null || true
done
# The "nixos-fw-accept" chain just accepts packets.
ip46tables -N nixos-fw-accept
ip46tables -A nixos-fw-accept -j ACCEPT
# The "nixos-fw-refuse" chain rejects or drops packets.
ip46tables -N nixos-fw-refuse
ip46tables -A nixos-fw-refuse -j DROP
# The "nixos-fw-log-refuse" chain performs logging, then
# jumps to the "nixos-fw-refuse" chain.
ip46tables -N nixos-fw-log-refuse
ip46tables -A nixos-fw-log-refuse -p tcp --syn -j LOG --log-level info --log-prefix "refused connection: "
ip46tables -A nixos-fw-log-refuse -m pkttype ! --pkt-type unicast -j nixos-fw-refuse
ip46tables -A nixos-fw-log-refuse -j nixos-fw-refuse
# The "nixos-fw" chain does the actual work.
ip46tables -N nixos-fw
# Clean up rpfilter rules
ip46tables -t raw -D PREROUTING -j nixos-fw-rpfilter 2> /dev/null || true
ip46tables -t raw -F nixos-fw-rpfilter 2> /dev/null || true
ip46tables -t raw -X nixos-fw-rpfilter 2> /dev/null || true
# Perform a reverse-path test to refuse spoofers
# For now, we just drop, as the raw table doesn't have a log-refuse yet
ip46tables -t raw -N nixos-fw-rpfilter 2> /dev/null || true
ip46tables -t raw -A nixos-fw-rpfilter -m rpfilter --validmark -j RETURN
# Allows this host to act as a DHCP4 client without first having to use APIPA
iptables -t raw -A nixos-fw-rpfilter -p udp --sport 67 --dport 68 -j RETURN
# Allows this host to act as a DHCPv4 server
iptables -t raw -A nixos-fw-rpfilter -s 0.0.0.0 -d 255.255.255.255 -p udp --sport 68 --dport 67 -j RETURN
ip46tables -t raw -A nixos-fw-rpfilter -j DROP
ip46tables -t raw -A PREROUTING -j nixos-fw-rpfilter
# Accept all traffic on the trusted interfaces.
ip46tables -A nixos-fw -i lo -j nixos-fw-accept
# Accept packets from established or related connections.
ip46tables -A nixos-fw -m conntrack --ctstate ESTABLISHED,RELATED -j nixos-fw-accept
# Accept connections to the allowed TCP ports.
ip46tables -A nixos-fw -p tcp --dport 53 -j nixos-fw-accept -i br0
ip46tables -A nixos-fw -p tcp --dport 80 -j nixos-fw-accept -i br0
ip46tables -A nixos-fw -p tcp --dport 443 -j nixos-fw-accept -i br0
ip46tables -A nixos-fw -p tcp --dport 6789 -j nixos-fw-accept -i br0
ip46tables -A nixos-fw -p tcp --dport 8080 -j nixos-fw-accept -i br0
ip46tables -A nixos-fw -p tcp --dport 8443 -j nixos-fw-accept -i br0
ip46tables -A nixos-fw -p tcp --dport 8843 -j nixos-fw-accept -i br0
ip46tables -A nixos-fw -p tcp --dport 8880 -j nixos-fw-accept -i br0
ip46tables -A nixos-fw -p tcp --dport 13443 -j nixos-fw-accept -i br0
ip46tables -A nixos-fw -p tcp --dport 22 -j nixos-fw-accept
ip46tables -A nixos-fw -p tcp --dport 13443 -j nixos-fw-accept
ip46tables -A nixos-fw -p tcp --dport 13443 -j nixos-fw-accept -i enp1s0
# Accept connections to the allowed TCP port ranges.
# Accept packets on the allowed UDP ports.
ip46tables -A nixos-fw -p udp --dport 53 -j nixos-fw-accept -i br0
ip46tables -A nixos-fw -p udp --dport 3478 -j nixos-fw-accept -i br0
ip46tables -A nixos-fw -p udp --dport 10001 -j nixos-fw-accept -i br0
ip46tables -A nixos-fw -p udp --dport 41641 -j nixos-fw-accept
# Accept packets on the allowed UDP port ranges.
ip46tables -A nixos-fw -p udp --dport 67:68 -j nixos-fw-accept -i br0
# Accept IPv4 multicast. Not a big security risk since
# probably nobody is listening anyway.
#iptables -A nixos-fw -d 224.0.0.0/4 -j nixos-fw-accept
# Optionally respond to ICMPv4 pings.
iptables -w -A nixos-fw -p icmp --icmp-type echo-request -j nixos-fw-accept
# Accept all ICMPv6 messages except redirects and node
# information queries (type 139). See RFC 4890, section
# 4.4.
ip6tables -A nixos-fw -p icmpv6 --icmpv6-type redirect -j DROP
ip6tables -A nixos-fw -p icmpv6 --icmpv6-type 139 -j DROP
ip6tables -A nixos-fw -p icmpv6 -j nixos-fw-accept
# Allow this host to act as a DHCPv6 client
ip6tables -A nixos-fw -d fe80::/64 -p udp --dport 546 -j nixos-fw-accept
# Helper command to manipulate both the IPv4 and IPv6 tables.
ip46tables() {
iptables -w "$@"
ip6tables -w "$@"
}
ip46tables -w -t nat -D PREROUTING -j nixos-nat-pre 2>/dev/null|| true
ip46tables -w -t nat -F nixos-nat-pre 2>/dev/null || true
ip46tables -w -t nat -X nixos-nat-pre 2>/dev/null || true
ip46tables -w -t nat -D POSTROUTING -j nixos-nat-post 2>/dev/null || true
ip46tables -w -t nat -F nixos-nat-post 2>/dev/null || true
ip46tables -w -t nat -X nixos-nat-post 2>/dev/null || true
ip46tables -w -t nat -D OUTPUT -j nixos-nat-out 2>/dev/null || true
ip46tables -w -t nat -F nixos-nat-out 2>/dev/null || true
ip46tables -w -t nat -X nixos-nat-out 2>/dev/null || true
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i br0 -o enp1s0 -j ACCEPT
tc qdisc replace dev enp1s0 root cake besteffort autorate-ingress bandwidth 1000mbit lan ingress dual-dsthost ethernet
# tc qdisc replace dev br0 root cake besteffort autorate-ingress bandwidth 1000mbit lan dual-dsthost ethernet
# tc qdisc add dev enp1s0 handle 1: root cake besteffort autorate-ingress bandwidth $BANDWIDTH internet ingress dual-dsthost ethernet
# Helper command to manipulate both the IPv4 and IPv6 tables.
ip46tables() {
iptables -w "$@"
ip6tables -w "$@"
}
# Create subchain where we store rules
ip46tables -w -t nat -N nixos-nat-pre
ip46tables -w -t nat -N nixos-nat-post
ip46tables -w -t nat -N nixos-nat-out
# We can't match on incoming interface in POSTROUTING, so
# mark packets coming from the internal interfaces.
iptables -w -t nat -A nixos-nat-pre \
-i 'br0' -j MARK --set-mark 1
iptables -w -t nat -A nixos-nat-pre \
-i 'tailscale0' -j MARK --set-mark 1
# NAT the marked packets.
iptables -w -t nat -A nixos-nat-post -m mark --mark 1 \
-o enp1s0 -j MASQUERADE
# NAT packets coming from the internal IPs.
# NAT from external ports to internal ports.
iptables -w -t nat -A nixos-nat-pre \
-i enp1s0 -p tcp \
--dport 13443 \
-j DNAT --to-destination 192.168.3.10:13443
# Append our chains to the nat tables
ip46tables -w -t nat -A PREROUTING -j nixos-nat-pre
ip46tables -w -t nat -A POSTROUTING -j nixos-nat-post
ip46tables -w -t nat -A OUTPUT -j nixos-nat-out
# Reject/drop everything else.
ip46tables -A nixos-fw -j nixos-fw-log-refuse
# Enable the firewall.
ip46tables -A INPUT -j nixos-fwI switched to not using the hardened kernel package and seems to not have made a difference. I did reboot to and reapply the config to forward ports, but I get the same error.
Apr 25 13:40:37 cmp-nix1 systemd[1]: Reloading Firewall.
Apr 25 13:40:41 cmp-nix1 firewall-reload[16462]: iptables v1.8.5 (legacy): unknown option "--to-destination"
Apr 25 13:40:41 cmp-nix1 firewall-reload[16462]: Try `iptables -h' or 'iptables --help' for more information.
Apr 25 13:40:41 cmp-nix1 firewall-reload[16303]: Failed to reload firewall... Stopping
Apr 25 13:40:42 cmp-nix1 systemd[1]: firewall.service: Control process exited, code=exited, status=1/FAILURE
Apr 25 13:40:42 cmp-nix1 systemd[1]: Reload failed for Firewall.
Apr 25 13:41:14 cmp-nix1 systemd[1]: Reloading Firewall.
Apr 25 13:41:15 cmp-nix1 firewall-reload[16740]: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 25 13:41:15 cmp-nix1 firewall-reload[16741]: iptables: Bad rule (does a matching rule exist in that chain?).
Apr 25 13:41:18 cmp-nix1 systemd[1]: Reloaded Firewall.
-- Reboot --
Apr 25 13:41:55 cmp-nix1 systemd[1]: Starting Firewall...
Apr 25 13:41:57 cmp-nix1 systemd[1]: Finished Firewall.
Apr 25 13:45:00 cmp-nix1 systemd[1]: Reloading Firewall.
Apr 25 13:45:03 cmp-nix1 firewall-reload[2416]: iptables v1.8.5 (legacy): unknown option "--to-destination"
Apr 25 13:45:03 cmp-nix1 firewall-reload[2416]: Try `iptables -h' or 'iptables --help' for more information.
Apr 25 13:45:03 cmp-nix1 firewall-reload[2256]: Failed to reload firewall... Stopping
Apr 25 13:45:03 cmp-nix1 systemd[1]: firewall.service: Control process exited, code=exited, status=1/FAILURE
Apr 25 13:45:03 cmp-nix1 systemd[1]: Reload failed for Firewall.Following is my full configuration.nix. I'm posting it cause last time it was the kernel module I needed and what I had posted would never have shown that. Maybe someone can see something obviously wrong with my config.
# your system. Help is available in the configuration.nix(5) man page
# and in the NixOS manual (accessible by running ‘nixos-help’).
{ config, pkgs, ... }:
{
imports =
[ # Include the results of the hardware scan.
./hardware-configuration.nix
];
# Use the systemd-boot EFI boot loader.
boot.loader.systemd-boot.enable = true;
boot.loader.efi.canTouchEfiVariables = true;
nixpkgs.config.allowUnfree = true;
boot.kernel.sysctl."net.ipv4.ip_forward" = 1;
boot.kernel.sysctl."net.core.default_qdisc" = "cake";
boot.kernel.sysctl."net.ipv4.tcp_ecn" = 1;
boot.kernel.sysctl."net.ipv4.tcp_sack" = 1;
boot.kernel.sysctl."net.ipv4.tcp_dsack" = 1;
#boot.kernel.sysctl."net.ipv6.conf.all.forwarding" = 1;
networking = {
hostName = "cmp-nix1"; # Define your hostname.
wireless.enable = false; # Enables wireless support via wpa_supplicant.
useDHCP = false; # Disabled because it's on each interface
bridges = {
br0.interfaces = ["enp2s0" "enp3s0" "enp4s0"];
};
interfaces = {
# WAN Port
enp1s0.useDHCP = true;
# LAN Port
br0 = {
useDHCP = false;
ipv4 = {
addresses = [
{ address = "192.168.3.1"; prefixLength = 24;}
];
};
};
};
nat = {
enable = true;
externalInterface = "enp1s0";
#externalInterface = "enp2s0";
internalInterfaces = [ "br0" "tailscale0" ];
forwardPorts = [{ sourcePort = 13443; destination = "192.168.3.10:13443"; proto = "tcp"; }];
};
firewall = {
allowPing = true;
allowedTCPPorts = [ 22 13443 ];
allowedUDPPorts = [ 41641 ];
interfaces = {
enp1s0 = {
allowedTCPPorts = [ 13443 ];
allowedUDPPorts = [ ];
};
br0 = {
allowedTCPPorts = [ 53 80 443 13443 8443 8080 8880 8843 6789 ];
allowedUDPPorts = [ 53 3478 10001 ];
allowedUDPPortRanges = [ { from = 67; to = 68; }];
};
};
extraPackages = [ pkgs.iproute ];
extraCommands = ''
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i br0 -o enp1s0 -j ACCEPT
tc qdisc replace dev enp1s0 root cake besteffort autorate-ingress bandwidth 1000mbit lan ingress dual-dsthost ethernet
# tc qdisc replace dev br0 root cake besteffort autorate-ingress bandwidth 1000mbit lan dual-dsthost ethernet
# tc qdisc add dev enp1s0 handle 1: root cake besteffort autorate-ingress bandwidth $BANDWIDTH internet ingress dual-dsthost ethernet
'';
extraStopCommands = ''
iptables -D FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT || true
iptables -D FORWARD -i br0 -o enp1s0 -j ACCEPT || true
'';
};
};
services.tailscale.enable = true;
services.dhcpd4 = {
enable = true;
#authoritative = true;
interfaces = [ "br0" ];
machines = [
<snip>
];
extraConfig = ''
subnet 192.168.3.0 netmask 255.255.255.0 {
range 192.168.3.100 192.168.3.199;
option routers 192.168.3.1;
option domain-name-servers 192.168.3.1;
option broadcast-address 192.168.3.255;
}
'';
};
services.bind = {
enable = true;
cacheNetworks = [
"127.0.0.0/24"
"192.168.3.0/24"
"1.1.1.1/32"
"1.0.0.1/32"
];
#listenOn = ["enp2s0"];
#listenOnIpv6 = ["enp2s0"];
forwarders = [
"1.1.1.1"
"1.0.0.1"
];
# extraOptions = ''
#recursion yes;
#allow-recursion { cachenetworks; };
#'';
};
services.unifi = {
enable = true;
openPorts = false;
};
# Select internationalisation properties.
i18n.defaultLocale = "en_US.UTF-8";
console = {
font = "Lat2-Terminus16";
keyMap = "us";
};
# Set your time zone.
time.timeZone = "America/New_York";
services.chrony.enable = true;
# List packages installed in system profile. To search, run:
# $ nix search wget
environment.systemPackages = with pkgs; [
wget vim inetutils traceroute
];
# Some programs need SUID wrappers, can be configured further or are
# started in user sessions.
# programs.mtr.enable = true;
# programs.gnupg.agent = {
# enable = true;
# enableSSHSupport = true;
# pinentryFlavor = "gnome3";
# };
# List services that you want to enable:
# Enable the OpenSSH daemon.
services.openssh.enable = true;
services.openssh.passwordAuthentication = false;
services.openssh.challengeResponseAuthentication = false;
services.openssh.permitRootLogin = "no";
# Open ports in the firewall.
# Enable CUPS to print documents.
services.printing.enable = false;
# Enable sound.
# sound.enable = true;
# hardware.pulseaudio.enable = true;
# Enable the X11 windowing system.
services.xserver.enable = false;
# services.xserver.layout = "us";
# services.xserver.xkbOptions = "eurosign:e";
# Enable touchpad support.
services.xserver.libinput.enable = false;
# Enable the KDE Desktop Environment.
services.xserver.displayManager.sddm.enable = false;
services.xserver.desktopManager.plasma5.enable = false;
# Define a user account. Don't forget to set a password with ‘passwd’.
users.users.cmp = {
isNormalUser = true;
extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
openssh.authorizedKeys.keys = [
<snip>
];
};
# Security Settings from secure profile in nixos repo
#boot.kernelPackages = pkgs.linuxPackages_hardened;
# environment.memoryAllocator.provider = "scudo";
# environment.variables.SCUDO_OPTIONS = "ZeroContents=1";
security.hideProcessInformation = true;
security.lockKernelModules = true;
security.protectKernelImage = true;
security.allowSimultaneousMultithreading = false;
security.forcePageTableIsolation = true;
security.virtualisation.flushL1DataCache = "always";
security.apparmor.enable = true;
boot.kernelParams = [
# Slab/slub sanity checks, redzoning, and poisoning
"slub_debug=FZP"
# Overwrite free'd memory
"page_poison=1"
# Enable page allocator randomization
"page_alloc.shuffle=1"
];
boot.kernelModules = [
"sch_cake"
"ipt_mark"
"iptable_nat"
"iptable_filter"
];
boot.blacklistedKernelModules = [
# Obscure network protocols
"ax25"
"netrom"
"rose"
# Old or rare or insufficiently audited filesystems
"adfs"
"affs"
"bfs"
"befs"
"cramfs"
"efs"
"erofs"
"exofs"
"freevxfs"
"f2fs"
"hfs"
"hpfs"
"jfs"
"minix"
"nilfs2"
"qnx4"
"qnx6"
"sysv"
"ufs"
];
# Restrict ptrace() usage to processes with a pre-defined relationship
# (e.g., parent/child)
boot.kernel.sysctl."kernel.yama.ptrace_scope" = "1";
# Hide kptrs even for processes with CAP_SYSLOG
boot.kernel.sysctl."kernel.kptr_restrict" = "2";
# Disable bpf() JIT (to eliminate spray attacks)
boot.kernel.sysctl."net.core.bpf_jit_enable" = false;
# Disable ftrace debugging
boot.kernel.sysctl."kernel.ftrace_enabled" = false;
# Enable strict reverse path filtering (that is, do not attempt to route
# packets that "obviously" do not belong to the iface's network; dropped
# packets are logged as martians).
boot.kernel.sysctl."net.ipv4.conf.all.log_martians" = true;
boot.kernel.sysctl."net.ipv4.conf.all.rp_filter" = "1";
boot.kernel.sysctl."net.ipv4.conf.default.log_martians" = true;
boot.kernel.sysctl."net.ipv4.conf.default.rp_filter" = "1";
# Ignore broadcast ICMP (mitigate SMURF)
boot.kernel.sysctl."net.ipv4.icmp_echo_ignore_broadcasts" = true;
# This value determines the NixOS release from which the default
# settings for stateful data, like file locations and database versions
# on your system were taken. It‘s perfectly fine and recommended to leave
# this value at the release version of the first install of this system.
# Before changing this value read the documentation for this option
# (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
system.stateVersion = "20.03"; # Did you read the comment?
}
nixos-discourse
commented
3 years ago This issue has been mentioned on NixOS Discourse. There might be relevant details there:
thatsmydoing
commented
3 years ago I'd recommend you unlock kernel modules and if it succeeds, check the difference in iptables modules.
Just a wild guess, but have you tried adding ipt_DNAT / xt_nat?
thatsmydoing
commented
3 years ago Your logs say
Apr 25 13:41:55 cmp-nix1 systemd[1]: Starting Firewall...
Apr 25 13:41:57 cmp-nix1 systemd[1]: Finished Firewall.so it does look like it works? Did you only add your forward config after you've rebooted? After doing a nixos-rebuild boot with the forward config and rebooting does it not work?
Looking at how lockKernelModules works, it only locks it after the firewall has loaded so you shouldn't even have to explicitly load anything IIUC. https://github.com/NixOS/nixpkgs/blob/d600f006643e074c2ef1d72e462e218b647a096c/nixos/modules/security/lock-kernel-modules.nix
chrisportela
commented
3 years ago Yea I basically remove the forwardPorts option and the firewall works fine. I rollback to get internet again when I'm still connected to that network.
I have unlocked the kernel modules and added the other two modules. I thought those were being loaded somehow cause I saw them in the list of modules, but maybe I needed to be more explicit.
chrisportela
commented
3 years ago ok! I think that was the fix. Pretty sure combo of not locking the modules and also explicitly adding them made it work. Now I can access the NAS I was trying to access fine.
eljojo
commented
2 years ago Hey everyone, I had a similar issue. The following solved the problem and helped me keep the locked modules settings enabled:
config.boot.kernelModules = [
"iptable_nat"
"iptable_filter"
"xt_nat"
];
Describe the bug Trying to use this type of config creates the expected firewall config, but that config fails
The errors
I am currently using the Hardened Kernel with the following extra kernel modules
These are all the modules with "ip" loaded
To Reproduce Steps to reproduce the behavior:
Expected behavior Firewall reload succeeds and port forwarding enabled
Notify maintainers Unclear who are maintainer to me for NAT cause to me it seems the issue is a middle module. Just tagging last two committers for now.
@valeriangalliat @thatsmydoing
Metadata
Maintainer information: