search

blacklanternsecurity / bbot

A recursive internet scanner for hackers.
https://www.blacklanternsecurity.com/bbot/
GNU General Public License v3.0
4.19k stars 381 forks source link

Better deduplication for SOCIAL events #1311

Closed TheTechromancer closed 3 months ago

TheTechromancer commented 3 months ago

Running into some duplicate SOCIAL events:

# trailing slash
{"platform": "facebook", "profile_name": "evilcorp", "url": "https://www.facebook.com/evilcorp/"}
{"platform": "facebook", "profile_name": "evilcorp", "url": "https://www.facebook.com/evilcorp"}

# upper/lower case
{"platform": "twitter", "profile_name": "EvilCorp", "url": "https://twitter.com/EvilCorp"}
{"platform": "twitter", "profile_name": "evilcorp", "url": "https://twitter.com/evilcorp"}

# optional "www"
{"platform": "github", "profile_name": "elliot", "url": "https://github.com/elliot/"}
{"platform": "github", "profile_name": "elliot", "url": "http://www.github.com/elliot"}
TheTechromancer commented 3 months ago

Fixed in https://github.com/blacklanternsecurity/bbot/pull/1313.