stryker2k2
commented
3 weeks ago I'm tinkering around with it now. @TheTechromancer would know better than me. But... looking at your debug log, there seems to be a big lack of 'URL_UNVERIFIED' from the excavate module to feed everything else; which would allowing everything else to spider out.
An hour ago, I ran bbot -t <target> -m httpx to keep it very simple. An hour ago, I got nothing.
But now when I run the same command, I get:
[INFO] aggregate: +-----------+-----------------------------------+---------------------------------------+
[INFO] aggregate: | Module | Produced | Consumed |
[INFO] aggregate: +===========+===================================+=======================================+
[INFO] aggregate: | excavate | 54 (54 URL_UNVERIFIED) | 2 (2 HTTP_RESPONSE) |
[INFO] aggregate: +-----------+-----------------------------------+---------------------------------------+
[INFO] aggregate: | speculate | 3 (1 OPEN_TCP_PORT, 1 ORG_STUB, 1 | 78 (9 DNS_NAME, 2 HTTP_RESPONSE, 1 |
[INFO] aggregate: | | URL_UNVERIFIED) | IP_ADDRESS, 2 URL, 64 URL_UNVERIFIED) |
[INFO] aggregate: +-----------+-----------------------------------+---------------------------------------+
[INFO] aggregate: | httpx | 2 (2 URL) | 3 (2 OPEN_TCP_PORT, 1 URL_UNVERIFIED) |
[INFO] aggregate: +-----------+-----------------------------------+---------------------------------------+
[INFO] aggregate: | host | 1 (1 DNS_NAME) | 0 |
[INFO] aggregate: +-----------+-----------------------------------+---------------------------------------+I could absolutely do something with this now if I added more modules and config settings.
I don't know what the answer is. But, I'm guessing either normal network hiccup... or something fishy with excavate and/or httpx.
I'll keep looking into it.
On BBOT v1.1.7.3251rc , on this site, bbot is not crawling at all.
debug.log