search

blacklanternsecurity / bbot

A recursive internet scanner for hackers.
https://www.blacklanternsecurity.com/bbot/
GNU General Public License v3.0
4.65k stars 420 forks source link

URL_UNVERIFIED dupe detection / spider-danger conflict #471

Closed liquidsec closed 1 year ago

liquidsec commented 1 year ago

If excavate finds a URL_UNVERIFIED, and adds spider-danger tag, and FFUF (or another module) finds the same URL_UNVERIFIED, it will be marked as a dupe and will not be visited by HTTPX

TheTechromancer commented 1 year ago

We can fix this by overriding the _data_id() method on URL_UNVERIFIED events to include spider-danger.

The _data_id() method returns the event data that's used to calculate the id hash.

We would need to notify @SpamFaux because this might create some duplicate URLs in his Neo4j data.

TheTechromancer commented 1 year ago

Fixed in https://github.com/blacklanternsecurity/bbot/pull/480