search

WaterByWind / edgeos-bl-mgmt

Automated updating of EdgeOS firewall network-group to be used as source address blacklist
MIT License
196 stars 40 forks source link

Issue Parsing Talos Feed #23

Open ntopnguser opened 1 year ago

ntopnguser commented 1 year ago

Not sure if anyone still maintains this, but there appears to be an issue pulling the Cisco Talos feed (https://www.talosintelligence.com/documents/ip-blacklist). My guess is the fact that the URL redirects to an Amazon S3 bucket is the issue.

chessmck commented 1 year ago

Not sure if anyone still maintains this, but there appears to be an issue pulling the Cisco Talos feed (https://www.talosintelligence.com/documents/ip-blacklist). My guess is the fact that the URL redirects to an Amazon S3 bucket is the issue.

Not a list I use, but in testing just now, I have no issue accessing that URL which directs to (as you mentioned) to https://snort-org-site.s3.amazonaws.com/production/document_files/files/000/019/195/original/ip_filter.blf

91.228.167.128 31.41.8.66 27.31.180.123 91.109.184.7 83.220.172.27 95.214.107.21 and on

I show the default list having it commented out, which is why I may not be using it...

Talos Reputation Center IP Blacklist

Also see https://www.talosintelligence.com/reputation

https://www.talosintelligence.com/documents/ip-blacklist

Updated - Interesting read here on snort blocking and only used for testing, the moved part is old.. https://www.reddit.com/r/pfBlockerNG/comments/iclh0y/talos_blocklist_seems_to_have_moved/

ntopnguser commented 1 year ago

It may be an artifact of how my implementation is parsing the site (I'm using VyOS). When I manually run the script, I can see where the Talos site is fetched, but it is not processed into the final list. I was actually able to get it to work by adding a "-L" option on line 458 of updBLackList.sh which informs cURL to follow redirects.

WaterByWind commented 1 year ago

This seems to be a semi-recent change in that list location. This doesn't appear to be a traditional redirect either (but isn't broken). This had been a separate Cisco Talos list but is now just a redirect to a snort list.

I intentionally did not include a -L option to curl. The ultimate target URL for this list appears to be dynamic so if this is is desired a -L would be required unfortunately.

I'll probably look to add a comment to an updated reference list after looking into this one further.