Stdout dies mid-scan

[TheTechromancer]

During large scans, stdout sometimes stops working. Status messages are still printed via stderr, and output.txt continues to have events written to it. But no stdout is output to the console.

I was able to reliably reproduce this with the following command:

bbot -t ea.com -f safe passive -p spider subdomain-enum -em gowitness hunt

Stdout dies roughly thirty seconds into this scan.

[TheTechromancer]

Did some troubleshooting and found some even stranger behavior. Stdout doesn't die suddenly; it has "gaps" where twenty or thirty lines will mysteriously disappear into nowhere, and then it will resume again normally. This seems to happen several times, losing roughly half of the lines, before stdout completely dies.

Here's what I've tried so far:

• disabled this line in cli.py:

  # fix tee buffering
  sys.stdout.reconfigure(line_buffering=True)

• disabled sslcert (I noticed some strange behavior with sslcert timeouts possibly holding up the scan)
• neutered all subprocesses in case of any stdout conflicting weirdness
• disabled keyboard interaction loop
• turned down dns.max_event_handlers to 1 (and reduced resolve_batch threads to 1)

[TheTechromancer]

It has to be a module, right?

[TheTechromancer]

• anubisdb,asn,azure_realm,azure_tenant,baddns,baddns_zone,bevigil,binaryedge,bucket_amazon,bucket_azure,bucket_file_enum,bucket_firebase,bucket_google,builtwith,c99,censys,certspotter,chaos,columbus,crt,digitorus,dnscommonsrv,dnsdumpster,fullhunt,github_codesearch,github_org,hackertarget: BAD
  • anubisdb,asn,azure_realm,azure_tenant,baddns,baddns_zone,bevigil,binaryedge,bucket_amazon,bucket_azure,bucket_file_enum,bucket_firebase,bucket_google,builtwith,c99: BAD
    • anubisdb,asn,azure_realm,azure_tenant,baddns,baddns_zone,bevigil,binaryedge: BAD
      • anubisdb,asn,azure_realm,azure_tenant: GOOD
      • baddns,baddns_zone,bevigil,binaryedge: BAD
        • baddns,baddns_zone: BAD
          • baddns: BAD
          • baddns_zone: GOOD
        • bevigil,binaryedge: GOOD
    • bucket_amazon,bucket_azure,bucket_file_enum,bucket_firebase,bucket_google,builtwith,c99: GOOD
  • censys,certspotter,chaos,columbus,crt,digitorus,dnscommonsrv,dnsdumpster,fullhunt,github_codesearch,github_org,hackertarget: GOOD
• httpx,hunterio,internetdb,leakix,myssl,oauth,otx,passivetotal,postman,rapiddns,riddler,securitytrails,shodan_dns,sitedossier,social,subdomaincenter,threatminer,urlscan,virustotal,wayback,zoomeye: GOOD

@liquidsec you're in trouble 👀

[TheTechromancer]

Fixed in https://github.com/blacklanternsecurity/bbot/pull/1417.