search

multiduplikator / mikrotik_blocklist

Aggregated blocklist for mikrotik (and others)
37 stars 10 forks source link

Add the 194000 IPs from https://raw.githubusercontent.com/stamparm/ipsum/master/levels/1.txt ? #8

Closed TomHGitHub closed 3 months ago

TomHGitHub commented 3 months ago

Hi, Could you add an option to add the 194000 IPs from https://raw.githubusercontent.com/stamparm/ipsum/master/levels/1.txt ? Much appreciated. Thank you!

multiduplikator commented 3 months ago

Will look into it next week.

multiduplikator commented 3 months ago

I added your ipsum l1 list as xl variant. It contains about 173k entries, and as such is at risk to be too large for many routers.

Enjoy.

TomHGitHub commented 3 months ago

Awesome. Thank You!

multiduplikator commented 3 months ago

Just as a sidenote, I added ipsum l3 to all variants of the list.

njumaen commented 3 months ago

 Looks good, if they are already in…. Had fears that the lists gets far too large for some devices...

Just as a sidenote, I added ipsum l3 to all variants of the list.

TomHGitHub commented 3 months ago

From a performance standpoint, would it make sense to break the blocklists into 10 or 25 sublists, e.g. based on the first IP byte and then search for IPs in the corresponding sublists only? My Mikrotik HexS starts getting issues at 35k+ entries, already. That way you could update one list after the other.

multiduplikator commented 3 months ago

Hmm. Depends on where the performance issues come from.

If you are willing to endure a moment of higher exposure, you could load the new blocklist in full, simply replacing the old one. That way, you skip all the comparison/delete/add cycles. That will give the maximum performance.

Then, you are only left with the actual blocklist lookup during rules processing. Here you can experiment with doing the split manually first, and then see if that does any good.

theQuikest commented 3 months ago

I used the blocklist on a RB750gr3 (HEX).

If you load the blocklist into memory instead on flash/hdd, it runs considerably faster.

If you go this way, don't forget to run this script on startup!

Added bonus: a lot more friendly on the flash hdd and no issues on hdd getting full.