borestad / blocklist-abuseipdb

Aggregated AbuseIPDB blocklists with worst IPv4 & IPv6 offenders (~100% confidence)
111 stars 10 forks source link

Thanks #2

Closed elandorr closed 10 months ago

elandorr commented 1 year ago

This is pretty handy. And probably pretty illegal. But for broke admins, the only (?) way to a decent blacklist. As far as I know, it even used to be donation-powered in the past. Nice tooling too.

Thanks for sharing!

Edit: What is your intent with the 'all' file? With the speed at which capitalists breeze through IPs, we'll end up having half the non-retail net in it. And that array is going to get huge as well. Am curious.

How about adding 60, 90, 120? That seems sustainable indefinitely.

This is such a 4D move, I'm still in awe. This possibility didn't even cross my mind.

Edit2: This is such a fun thing, I had to look into it. Check this out:

According to this post:

For example bash uses 1.3 GB of virtual memory for an array size of 18074340.

Your setting of 9999 aka 27 years is 'only' 239976. So that actually is doable just fine.

I've moved to a future-proof py solution as it handles IPv6, but iprange is wicked fast for v4. Orders of magnitude faster. But even then, at the current speed our system can handle, this means it'd run for 3 hours straight. In 27 years, that is.

Guess I'm stupid, but I always like to think of the resources.

As long as you're keeping this repo up, I'll just be pragmatic and use it. :) Cheers

borestad commented 1 year ago

Hi @elandorr

Late reply, sorry! ๐Ÿ˜

I've added 60,90 & 120 days now to the list. I've thought to add them earlier, but will update this repository to a much better/faster solution asap (have it running in another experimental repository)

Regarding the bash/array/memory. Yes - I know, but this script isn't creating an array. It's merely concatinating (extremely fast) to a temporary file with a modulus 100 to avoid huge numbers.

Edit: What is your intent with the 'all' file? With the speed at which capitalists breeze through IPs, we'll end up having half the non-retail net in it. And that array is going to get huge as well. Am curious.

Mainly for statistical analysis, but as said - my other experimental repository solves this in a much more efficient way.

Anyway, thank you for the kind words! ๐Ÿ™‚

elandorr commented 1 year ago

Yes, I saw, I re-implemented it myself in the meantime using a different optimizer, as iprange didn't handle some edge cases, and I somehow insisted on handling those which may never occur. (forgot which cases by now, sorry). iprange is hugely faster, and I can see how it's efficient. But presumably every tool will eat more memory with size. The newer one handles more, but explodes rather quickly, iprange is far less hungry. I'd sure like to see your experimental repo where you got it even faster! Thanks again for the idea!

borestad commented 10 months ago

@elandorr FYI: I've now updated the repository to the much much faster alternative logic ๐Ÿ™ƒ