Firewall Logs

[opustecnica]

After being spoiled by many years of customizable and efficient EdgeRouter use, ... I fell for the UDM/P. Rather than complaining and hoping my needs make it to the top of UNIFI's development team priorities, I tend to fix, in a more or less efficient way, my problems with a bit of code glue. To that end, there are not enough thank you I can express for your simple and elegant work on the udm-utilities package.

One item that always bothered me was the lack of tagging of the UDM/P firewall logs. These scripts attempt to address that issue.

There are a few things I would like to improve and I am wondering if anyone has suggestions:

1) Currently, in my implementation, the script is called at reboot time. Unfortunately, the IPTABLES rules are "reset" to default logging - or lack thereof - behavior every time a change is performed on the firewall. To correct the issue without rebooting, one has to execute the "correction" script manually. Does anyone know of a "hook" that can be used to automatically call the correction script on firewall change? Lacking a more elegant way to achieve that, I am thinking of monitoring /mnt/data/udapi-config/ubios-udapi-server/ubios-udapi-server.state for changes and trig the script.

2) In the current implementation of the "tagging" script, I was able to add the "ACTION" and the "CHAIN" generating the log. It would even be more useful if a reference (e.g. rule id) to the originating rule could be added. Does anyone have any idea on how to gather that piece of info? (direct access to Postgres RO, API, some specific JSON?)

[spali]

Not a simple solution, but maybe auditd could help. Available as debian package in unifi-os. i.e. https://unix.stackexchange.com/questions/206891/audit-on-changes-to-the-running-iptables-configuration

@boostchicken This could also be interesting for other situations for some scripts in this repo ;) As always... I think far ... this combined with systemd event listeners would allow to notify other services (or scripts) on iptables changes and restart them. 🤤

[opustecnica]

Will definitively take a look at it later today. In the meanwhile I have developed a small service (unifi-os) that monitors the /mnt/data/udapi-config/ubios-udapi-server/ubios-udapi-server.state file and executes one or more actions (add logs to iptables in this case) on alert. Will push this bit later today.

[opustecnica]

Used the on_boot manual install as a template for a watchdog that, on configuration change, can run any command like the one to add tags to the firewall logs. https://github.com/opustecnica/public/blob/master/UDM/install-udmwatchdog.sh

[spali]

also found another possible way:

the "official" api from within unifi-os shell:

curl --include \
     --no-buffer \
     --header "Connection: Upgrade" \
     --header "Upgrade: websocket" \
     --header "Sec-WebSocket-Key: anyvalue" \
     --header "Sec-WebSocket-Version: 13" \
     --header "X-UserId: $(mongo 127.0.0.1:27117/ace --quiet --eval 'db.admin.findOne({is_owner: true}).uid')" \
     http://localhost:8081/wss/s/default/events

can also run outside of unifi-os if you get your user id elsewhere or type it static in the header ;) then just filter for the firewallrule:* events

[boostchicken]

@opustecnica could you modify your files and put them in the repo or here? Or made a version for people using this and we can add a link to your repo?

[bluewalk]

@opustecnica You can also use inotifywait (part of the inotify-tools package) to prevent polling every second in your script

LOG=/mnt/data/udapi-config/ubios-udapi-server/ubios-udapi-server.state

while inotifywait -e modify $LOG; do
  # execute scripts in e.g. /mnt/data/on_firewall.d
done

@boostchicken This would actually be a very nice addition to on-boot-script, sort of a on-firewallchange-script?

[noaboa97]

@opustecnica so I found your this conversation and also had a look at your notes on your repo. I was wondering if this currently works for you?

Have you found a "hook" to run the script? Have you implemented the "hook" with the "ubios-udapi-server.state". If yes where did you publish it?

Also did you see the comment from @spali and did you managed to do so?

I am very interested in the whole project unfortunatly I don't know linux that well and also don't know much about bash or sh scripting.

[opustecnica]

noaboa97, Sincere apologies for the delay. Unfortunately, due to work commitments, I have been busy on other things and have not fully explored @spali's suggestion/hint. In the meanwhile, look at the watch.sh. When launched at boot, it periodically (currently 1 second but modifiable) checks for changes affecting /mnt/data/udapi-config/ubios-udapi-server/ubios-udapi-server.state file (... something changed in the configuration) and executes the "ipt-enable-logs-launch.sh" script.

PS: Most/all of these script will likely be relocated to boostchicken's repo together with some explanatory comments in the near future. Likely more convenient for end-users.

[pedropombeiro]

For anyone interested, I've added a suggestion here to replace the bash script that is being run in the unifi-os container, to instead use a Go script that is built locally on the UDM Pro and allows generating rules that include the comment ID in the log prefix, making it possible to trace back the log entry to an iptables rule (unfortunately not yet to the Network Application firewall rule number).

[boostchicken]

@opustecnica what is the best way to re-direct people to your stuff, you gonna PR to here or is there some edit I can make?

cc @pedropombeiro

[martintoreilly]

Thanks @opustecnica and @pedropombeiro for your work on this and @pedropombeiro for adding your go scripts as the ipt-enable-logs module/folder! 🎉 🙌

Do you know what the numbers in the iptables firewall rule --comment fields reference? I've been trying to link them back to the firewall rules set in the UI with no success.

I've looked at what looks like the database my UDM stores its configuration data in, including firewall rules, and compared these with the iptables rules. I have managed to link the various --match-set references in the iptables rules to entries in the settings database, but have not had any luck deciphering what the numbers in the iptables --comments fields refer to.

Notes on my explorations so far below (mainly for my future self, but also for anyone else who wants to help explore).

Explore iptables rules

All commands below run from the native shell (i.e. SSH'd in as root without launching a Unifi OS shell).

• View iptables firewall rules: iptables-save
• View port and IP address sets used in --match-set references: ipset list <reference>. IP address sets appear to hold references to two child IP address sets, one for IPv4 and one for IPv6 that actually contain the IP address range data. Port sets appear to directly contain the port list data.

I can match all the --match-set references in my iptables data to data in the settings database as follows:

• Those of the form UBIOS_NETv4_br<VLAN-ID> reference ipset entries with the IP address ranges associated with the network configuration with a vlan field matching that VLAN-ID in the settings database networkconf collection.
• Those of the form UBIOS_<GUID> reference ipset entries with the IP address or port ranges associated with the firewall address or port group with an _id field matching that GUID in the settings database firewallgroup collection.

Explore "ace" MongoDB database

⚠️ WARNING: The commands below run mongodb with full admin rights on the settings database, so be super careful not to change anything ⚠️

It looks like the UDM settings are stored in this "ace" database, with the underlying MongoDB files stored in /data/unifi/data/db I think. This can be explored as follows.

• SSH into UDM as root
• Launch Unifi OS shell: unifi-os shell
• Launch mongo database client: mongo 127.0.0.1:27117
• Switch to "ace" database: use ace
• List collections in database: db.getCollectionNames()
• List firewall rules: db.firewallrule.find().pretty()
• List firewall port/address groups: db.firewallgroup.find().pretty()
• List network configurations: db.networkconf.find().pretty()

[pedropombeiro]

Great job @martintoreilly! Unfortunately, I also have no idea what the numbers used for comments refer to.

[pgorod]

Well, let's see 🕵️ ...

I can figure out some parts of the comment numbers, they are concatenated numbers.

(I added a few spaces for clarity)

The final digit which looks more sequential is the sequence number within a rule group (so, it's the Rule Index units), with a static 48 in the middle. The previous digit before the 48 is the Rule index thousands (so, 2 for 200x, and 6 for 600x).

The initial part seems to be stable (in my case, always 0000000109), the next block is quite stable (521666) but seems to change a few times, maybe for the logging rules that don't show up in the UI tables (736414, and the 48 further ahead changes to 12 - to me this suggests bit flags, 32+16, 8+4).

I didn't go look in the DB, but I expect some of those other numbers are row ids from stuff in the DB...

[martintoreilly]

@pgorod @pedropombeiro I've been doing some experimenting and I think the comment is actually in two parts. Each comment is 40-bits long and I think each one splits into an 8-bit part and 32-bit part. I'm pretty confident that the final 32-bits are the rule number within the rule set. I'm not 100% sure what the first 8-bits represent, but I think they are a bit mask of some kind. I see the following 3 values in a range of rules I've configured.

bit mask	min comment no.	max comment no.	notes
00010000	4294967296	70866960383	all rules with tcp protocol set via "tcp" or "tcp and udp" radio selection or via preset rule
00100000	137438953472	139586437119	all rules with udp protocol set via "udp" or tcp and udp" radio selection or via preset rule
11111111	1095216660480	1097364144127	set for every other rule I've created and preset rules with protocol other than tcp and/or udp

I had first thought the initial 8-bits might be the IP protocol number (in the range 0-255). However, I no longer think this is true as (1) the mask is 11111111 for most protocols; (2) the 00010000 "tcp" mask is 16, in decimal rather than 6 (the IP protocol number for tcp); (3) the 00100000 "udp" mask is 32, in decimal rather than 17 (the IP protocol number for udp).

@pedropombeiro Even though I'm not totally sure what the first 8-bits is encoding, I'm pretty confident the last 32-bits is the rule number, so I think you should be able to reliably add the rule number in place of the comment number in the rule logging prefix just by taking the last 32-bits of the comment number and parsing it as a signed integer.

IP rule properties to iptables comment mapping investigation

protocol (set via)	port	rule no.	iptables comment no.	first 8-bits of comment (binary)	last 32-bits of comment (decimal)	rule type
all ( 0 in no. box)	any-any	2000	00000001095216662480	11111111	2000	user
icmp (1 in no. box)	any-any	2001	00000001095216662481	11111111	2001	user
st (2 in no. box)	any-any	2002	00000001095216662482	11111111	2002	user
10 (10 in no. box)	any-any	2003	00000001095216662483	11111111	2003	user
esp (50 in no. box)	any-any	2004	00000001095216662484	11111111	2004	user
254 (254 in via no. box)	any-any	2005	00000001095216662485	11111111	2005	user
AH (AH in name dropdown)	any-any	2006	00000001095216662486	11111111	2006	user
tcp (6 in no. box)	any-any	2007	00000001095216662487	11111111	2007	user
tcp (tcp radio selection)	any-any	2008	00000000004294969304	00010000	2008	user
udp (udp radio selection)	any-any	2009	00000000008589936601	00100000	2009	user
tcp + ucp (tcp and ucp radio selection)	all	2010	00000000004294969306 and 00000000008589936602	00010000 and 00100000	2010 and 2010	user
icmp (icmp radio selection)	any-any	2011	00000001095216662491	11111111	2011	user
all (all radio selection)	any-any	2012	00000001095216662492	11111111	2012	user
tcp + udp (DNS preset)	any-53	3001	00000000004294970297 and 00000000008589937593	00010000 and 00100000	3001	preset
icmp (ICMP preset)	any-any	3002	00000001095216663482	11111111	3002	preset
udp (DHCP server preset)	68-67	3003	00000000008589937595	00100000	3003	preset
udp (RADIUS authentication preset)	any-1812	3004	00000000008589937596	00100000	3004	preset
udp (RADIUS accounting preset)	any-1813	3005	00000000008589937597	00100000	3005	preset
all (Guest portal preset)	any-8843 and any-8880	3006	00000001095216663486	11111111	3006	preset
tcp (redirector preset)	any-39080	3007	00000000004294970303	00010000	3007	preset
all (invisible default)	any-any	2147483647	00000001097364144127	11111111	2147483647	default

[martintoreilly]

@pedropombeiro Any chance of also making the prefix start with "FW-"? This would make it much easier to filter just for firewall rules in my remote log server (it's a Synology DSM and the log view is not customisable at all).

[pgorod]

@martintoreilly did you read my comment?

The last part, the rule number, is not a bitmask, or an encoded integer, as I have shown and your entries confirm.

[martintoreilly]

@pgorod Yes, I did read your comment, but I think we have suggested differing theories on how the rule number/index is encoded in the iptables comment number.

In your comment you split the digits of the comment number in the decimal format it appears in the comment, suggesting interpretations for various groups of digits. If I'm reading your comment correctly, you suggest that the last digit is the sequence number of the rule within the rule group and the 4th-from-last digit is the rule index/number thousands. However, this pattern does not hold for several of my rules (e.g. rules 2011 and 2012 don't match the within rule group sequence number pattern and those with iptables comments starting "0000000000429" or "0000000000858" don't match either on the rule index/number thousands pattern or the within rule group sequence number pattern).

My suggestion is to split the binary representation of the iptables comment number, rather than its decimal digits. If I convert the comment number to its binary representation, remove the upper (first) 8 bits and convert the remaining 32 digits back to a (signed) integer, I consistently get the firewall rule number/index for all the rules I explored above.

I'm not totally sure how to interpret the upper (first) 8 bits of the iptables comment number, but I suspect it's an 8-bit mask of some kind as the only three values I observe for these 8 bits are 00010000, 00100000 and 11111111.

[pedropombeiro]

Nice findings, thanks for the hard work! Interestingly, I see a scenario where the rule number is 65535 (all bits in the lower word are set to 1):

-A UBIOS_LAN_IN_USER -j LOG --log-prefix "[FW-A-LAN_IN_U-65535] "
-A UBIOS_LAN_IN_USER -m comment --comment 00000001097364144127 -j RETURN
-A UBIOS_LAN_LOCAL_USER -j LOG --log-prefix "[FW-A-LAN_LOCAL_U-65535] "
-A UBIOS_LAN_LOCAL_USER -m comment --comment 00000001097364144127 -j RETURN

I don't have any rule for LAN LOCAL, so maybe this is just logging by default that it was accepted?

[martintoreilly]

Nice findings, thanks for the hard work! Interestingly, I see a scenario where the rule number is 65535 (all bits in the lower word are set to 1):

-A UBIOS_LAN_IN_USER -j LOG --log-prefix "[FW-A-LAN_IN_U-65535] "
-A UBIOS_LAN_IN_USER -m comment --comment 00000001097364144127 -j RETURN
-A UBIOS_LAN_LOCAL_USER -j LOG --log-prefix "[FW-A-LAN_LOCAL_U-65535] "
-A UBIOS_LAN_LOCAL_USER -m comment --comment 00000001097364144127 -j RETURN

I don't have any rule for LAN LOCAL, so maybe this is just logging by default that it was accepted?

No worries, I'm glad to help. My thanks to you and @pgorod for doing the heavy lifting on this.

Yes, I have that rule too. This rule is the final "invisible default" rule in my explorations table. It looks like each rule set has a 00000001097364144127 default ~"deny all"~ rule. 00000001097364144127 actually has one zero in the 9th bit (1111111101111111111111111111111111111111), so it splits as 11111111 and 01111111111111111111111111111111, which equates to a rule number of 2147483647 in decimal, which is the maximum positive value for a 32-bit signed integer (the highest bit is the sign bit, and is 0 for positive numbers and 1 for negative numbers).

These default ~"deny all"~ rules are not visible in the firewall rules UI screens or the mongodb database firewallrule collection (which only seems to contain user-defined rules), but they are present in the /mnt/data/udapi-config/ubios-udapi-server/ubios-udapi-server.state server state file, with the rule number set to 2147483647.

Interestingly the state file has a second default rule with rule number 2147483646 (one less) that looks like the equivalent default ~"deny all"~ rule for IPv6. I'm assuming the reason it doesn't end up in my iptables configuration is because I don't have IPv6 active on my router. In fact, each rule group seems to have IPv6 versions of each of the preset IPv4 rules in the database and state file, but these also don't show in the UI or the mongodb database or make it to the iptables config.

Edit: Updated to reflect the fact that these default rules are not aways "deny all". Some are "accept all" (see next comment).

[martintoreilly]

@pedropombeiro Actually, I'm not sure what I said in my previous comment matches completely with the rule snippet you have in your last comment.

1. I see the rules you show are all ACCEPT. Looking at my state file, I think the action for the last default rule (rule number 2147483647) depends on the rule set rather than always being a "deny all" rule. In my state file the default final IPv4 rules for each rule set are as follows, and the default IPv4 LAN rules are all RETURN (ACCEPT), matching your rules.

   • AUTHORIZED_GUESTS : DROP
   • GUEST_IN : RETURN (ALLOW)
   • GUEST_LOCAL : DROP
   • GUEST_OUT : RETURN (ALLOW)
   • LAN_IN : RETURN (ALLOW)
   • LAN_LOCAL : RETURN (ALLOW)
   • LAN_OUT : RETURN (ALLOW)
   • WAN_IN : DROP
   • WAN_LOCAL : DROP
   • WAN_OUT : RETURN (ALLOW)

2. You extract the rule number for this final rule as 65535. I've double checked and 00000001097364144127 is definitely 1111111101111111111111111111111111111111 in binary and the last 32 bits (01111111111111111111111111111111) are decimal 2147483647, whether treated as a signed or unsigned integer, which matches the rule number of the last rule in the rule set in the state file. I think the 0xFFFF bit mask you apply to commentNr on line 59 of main.go is too short and only selects the lower 16-bits (65535 is the maximum value for an unsigned 16-bit integer). You need two hexadecimal characters for each of the 4 bytes in the 32-bit rule number, so this mask should be 0xFFFFFFFF. I've made a suggestion with this edit in the pull request.

[martintoreilly]

@pedropombeiro By the way, if the "Default Action Logging" toggle for the WAN/LAN/GUEST rules is toggled on, a new rule does appear in the iptables rules for those rule sets (e.g. -A UBIOS_GUEST_LOCAL_USER -j LOG), just before the final 00000001097364144127 rule. However, it has no associated comment in the iptables entry. In the state file, it is not represented as a new rule, but via setting the "log" fields of the final 2147483647 (IPv4) and 2147483646 (IPv6) rules to true.

[pedropombeiro]

Thanks for looking into it @martintoreilly. After I submitted the PR last night and closed the laptop, I started remembering that there seemed to be a default logging setting somewhere. I'm glad there's an explanation for that log rule. I've fixed the PR to take the 32-bits instead of 16-bits (although it shouldn't have any real-world effect other than the number used for the default action).