Support for more than 65,536 IPs?

[userisnotavaliable]

Hi, I have ufw-blocklist installed and it's working really well, thanks.

I'd like to use the Level 1 IPSUM blocklist (currently about 255K IP's) and I tried editing the script to achieve this, but it seems that only 65,536 IPs can be loaded?

Assuming i'm not doing something wrong, is that a limit that can be extended?

Cheers

[alzeih]

It certainly appears to be possible, but perhaps inadvisable.

The limit of 65,536 seems to be a default value for maxelem in ipset. The ipsets are created here:

https://github.com/poddmo/ufw-blocklist/blob/00dbe79712d236d41f6eb6e51b5e1f5fbbba61fb/after.init#L73

https://github.com/poddmo/ufw-blocklist/blob/00dbe79712d236d41f6eb6e51b5e1f5fbbba61fb/ufw-blocklist-ipsum#L68

You could change them from the default of $2^{16}$ to set a higher limit of $2^{18}$ by adding maxelem 262144

    $IPSET_EXE create  $ipsetname hash:net -exist maxelem 262144

You'll want to update this line to load the level 1 list updates:

https://github.com/poddmo/ufw-blocklist/blob/00dbe79712d236d41f6eb6e51b5e1f5fbbba61fb/ufw-blocklist-ipsum#L14

And perhaps these lines too, after creating /etc/ipsum.1.txt:

https://github.com/poddmo/ufw-blocklist/blob/00dbe79712d236d41f6eb6e51b5e1f5fbbba61fb/after.init#L29-L31

It takes a while, but appears to work:

May 15 10:04:35 linux ufw-blocklist-ipsum[33792]: starting update of ufw-blocklist-ipsum with 8606 entries from https://raw.githubusercontent.com/stamparm/ipsum/master/levels/1.txt
May 15 10:23:19 linux ufw-blocklist-ipsum[427780]: finished updating ufw-blocklist-ipsum. Old entry count: 8606 New count: 201254 of 201254

No idea if this is a good idea. Let the world know :)

[alzeih]

Although it's against the spirit of this repository to write state to disk, the hashing for these longer lists is now a significant performance hit. ipset can save the hash to a file in the hashed order, which makes restoring it much quicker.

I added a "restore file" with the output of ipset save for the given ipsetname. This is updated every time with the cron script. I then use this restore file, when available, in after.init start.

I also added a save and restore to after.init to manually save the ipset hash to this restore file. Intended use is with flush-all.

I don't expect this to be accepted as a pull request, so I've included the diff here in case that's helpful.

diff --git a/after.init b/after.init
index 8d6df2b..d0cecec 100644
--- a/after.init
+++ b/after.init
@@ -26,9 +26,12 @@ set -e
 export ipsetname=ufw-blocklist-ipsum

 # seed file containing the list of IP addresses to be blocked, one per line
-# curl -sS -f --compressed 'https://raw.githubusercontent.com/stamparm/ipsum/master/levels/4.txt' > /etc/ipsum.4.txt
+# curl -sS -f --compressed 'https://raw.githubusercontent.com/stamparm/ipsum/master/levels/1.txt' > /etc/ipsum.1.txt
 # ipset is updated daily by /etc/cron.daily/ufw-blocklist-ipsum
-export seedlist=/etc/ipsum.4.txt
+export seedlist=/etc/ipsum.1.txt
+
+# restore file containing the saved ipset hash for quicker loading
+export restorefile="/etc/${ipsetname}.ipset"

 export IPSET_EXE="/sbin/ipset"
 # check ipset exists and is executable
@@ -70,7 +73,7 @@ start)
     fi

     # create an empty ipset
-    $IPSET_EXE create  $ipsetname hash:net -exist
+    $IPSET_EXE create  $ipsetname hash:net -exist maxelem 262144
     $IPSET_EXE flush   $ipsetname

     ## Insert firewall rules to take precedence, removing them and adding them back if they already existed
@@ -106,6 +109,12 @@ start)
     iptables -A ufw-blocklist-forward -j DROP -m comment --comment "ufw-blocklist-forward"
     iptables -I FORWARD -m set --match-set $ipsetname dst -j ufw-blocklist-forward

+    # check if blocklist restore file exists
+    if [ -f "$restorefile" ]; then
+        $IPSET_EXE restore -exist < "$restorefile"
+        exit 0
+    fi
+
     # add members to the ipset
     # start this in a subshell and then disown the job so we return quickly.
     (
@@ -145,6 +154,24 @@ status)
     # show the last 10 lines from the logs
     journalctl | grep -i blocklist | tail
     ;;
+save)
+    if set_exists $ipsetname; then
+       $IPSET_EXE save -output save $ipsetname > "$restorefile"
+    fi
+    ;;
+restore)
+    # check that blocklist restore file exists
+    if [ ! -f "$restorefile" ]; then
+        echo "ufw after.init: $restorefile does not exist."
+        exit 1
+    fi
+
+    if set_exists $ipsetname; then
+       $IPSET_EXE flush $ipsetname
+    fi
+
+    $IPSET_EXE restore -exist < "$restorefile"
+    ;;
 flush-all)
     # flush sets created above. Use /etc/cron.daily/ufw-blocklist-ipsum to repopulate
     $IPSET_EXE flush  $ipsetname
@@ -163,6 +190,6 @@ flush-all)
     ;;
 *)
     echo "'$1' not supported"
-    echo "Usage: /etc/ufw/after.init {start|stop|flush-all|status}"
+    echo "Usage: /etc/ufw/after.init {start|stop|flush-all|status|save|restore}"
     ;;
 esac
diff --git a/ufw-blocklist-ipsum b/ufw-blocklist-ipsum
index 0efb978..368c6ef 100644
--- a/ufw-blocklist-ipsum
+++ b/ufw-blocklist-ipsum
@@ -11,7 +11,7 @@
 # install this file into /etc/cron.daily/ufw-blocklist-ipsum

 ## URL must return a text file with one IP address per line
-ipsumurl='https://raw.githubusercontent.com/stamparm/ipsum/master/levels/3.txt'
+ipsumurl='https://raw.githubusercontent.com/stamparm/ipsum/master/levels/1.txt'
 # reject the new list if there are fewer than minlen number of ip addresses
 minlen=1000

@@ -19,6 +19,9 @@ ipsetname=ufw-blocklist-ipsum
 ipset_exe=/usr/sbin/ipset
 logger="/usr/bin/logger -t ${ipsetname}"

+## restore file containing the saved ipset hash for quicker loading
+restorefile="/etc/${ipsetname}.ipset"
+
 ## Check if ipsetname exists. exit if not - ie no set to update
 ipsetstatus=$("${ipset_exe}" -t list "${ipsetname}" 2>/dev/null )
 RET=$?
@@ -65,7 +68,7 @@ isvalidip () {

 ## create a temporary ipset
 tmpsetname="$(mktemp -u | cut -f2 -d'.')-tmp"
-$ipset_exe -q create "$tmpsetname" hash:net
+$ipset_exe -q create "$tmpsetname" hash:net maxelem 262144
 RET=$?
 if [ $RET -ne 0 ]; then
         $logger -s "error code $RET creating temporary ipset $tmpsetname"
@@ -109,3 +112,6 @@ if [ $RET -ne 0 ]; then
 fi

 $logger "finished updating $ipsetname. Old entry count: $ipsetcount New count: $cnt of $scrublistlen"
+
+## save the ipset to $restorefile
+$ipset_exe save -output save "$ipsetname" > "$restorefile"