Limit count of active modules

[ooholo]

Hi hi! I've been running bbot on my VPS for some time now and I've noticed that when it comes to larger scans, due to a lack of RAM on my server, the process(es) will be killed. I've noticed the tool actually tells you when you're running low on memory, so it'd be nice to possibly pause all queued tasks until all active tasks are complete and the memory pool has emptied out, or define a set memory limit that will do the same thing but instead of dynamically pausing it'll just go until it reaches your set ceiling, and wait until there's more free space.

[TheTechromancer]

@ooholo thanks for opening an issue.

How much memory did your VM have available? We are aware that BBOT can be a bit memory-hungry, but usually the majority of the memory load tends to be from certain individual modules rather than the overall event queues.

Also, would you mind sharing your scan's status messages (including the debug output, so we can see the modules' memory usage)? I would not be opposed to putting a limit on some of the queue sizes. But I want to make sure this was the primary cause.

[ooholo]

Hi!

My VPS only has 2GB of usable ram, after making this post I created a 4GB swapfile on my server and it seemed to stop the process killing on small - medium scans. My larger scans are still being killed.

I don't mind sharing the status messages, but I'm unsure how to show each modules memory usages. I'll begin another large scan and grep the debug output to a new file, if I'm able to, what information is the most valuable to you from this output file? Should I be running bbot with any specific debug flags?

Thanks!

[ooholo]

I will add, before I leave everything to go, it seems nuclei is using up most of my memory. Checked using ps aux.

[TheTechromancer]

When --debug mode is enabled, the periodic status messages will include detailed stats on each module, including its memory usage.

[ooholo]

Okeeey... So! My log file is something close to 1.5GB, I'll include the last output of module memory usage before self destruction, which is:

MODULE MEMORY USAGE:

[DBUG] - crt: 515.25KB
[DBUG] - bucket_aws: 514.51KB
[DBUG] - bucket_azure: 514.51KB
[DBUG] - bucket_digitalocean: 514.51KB
[DBUG] - bucket_firebase: 514.51KB
[DBUG] - bucket_gcp: 514.51KB
[DBUG] - httpx: 130.68KB
[DBUG] - gowitness: 56.15KB
[DBUG] - sslcert: 35.15KB
[DBUG] - ipneighbor: 34.42KB
[DBUG] - iis_shortnames: 34.38KB
[DBUG] - robots: 34.07KB
[DBUG] - massdns: 21.49KB
[DBUG] - asn: 20.79KB
[DBUG] - secretsdb: 14.79KB
[DBUG] - censys: 3.55KB
[DBUG] - zoomeye: 3.55KB
[DBUG] - binaryedge: 3.49KB
[DBUG] - github: 3.46KB
[DBUG] - bevigil: 3.44KB
[DBUG] - virustotal: 3.43KB
[DBUG] - leakix: 3.42KB
[DBUG] - fullhunt: 3.40KB
[DBUG] - azure_tenant: 3.29KB
[DBUG] - securitytrails: 3.20KB
[DBUG] - emailformat: 3.15KB
[DBUG] - skymem: 3.15KB
[DBUG] - builtwith: 3.12KB
[DBUG] - shodan_dns: 3.12KB
[DBUG] - wayback: 3.10KB
[DBUG] - oauth: 3.08KB
[DBUG] - urlscan: 3.04KB
[DBUG] - speculate: 2.99KB
[DBUG] - excavate: 2.99KB
[DBUG] - anubisdb: 2.98KB
[DBUG] - certspotter: 2.98KB
[DBUG] - columbus: 2.98KB
[DBUG] - digitorus: 2.98KB
[DBUG] - dnsdumpster: 2.98KB
[DBUG] - hackertarget: 2.98KB
[DBUG] - myssl: 2.98KB
[DBUG] - otx: 2.98KB
[DBUG] - pgp: 2.98KB
[DBUG] - rapiddns: 2.98KB
[DBUG] - riddler: 2.98KB
[DBUG] - sitedossier: 2.98KB
[DBUG] - subdomaincenter: 2.98KB
[DBUG] - threatminer: 2.98KB
[DBUG] - csv: 2.97KB
[DBUG] - nuclei: 2.96KB
[DBUG] - azure_realm: 2.84KB
[DBUG] - ntlm: 2.66KB
[DBUG] - subdomains: 2.55KB
[DBUG] - human: 2.48KB
[DBUG] - json: 2.48KB
[DBUG] - nmap: 2.46KB
[DBUG] - subdomain_hijack: 2.41KB
[DBUG] - wappalyzer: 2.21KB
[DBUG] - dnszonetransfer: 2.19KB
[DBUG] - neo4j: 2.17KB
[DBUG] - dnscommonsrv: 2.10KB
[DBUG] - nsec: 2.10KB
[DBUG] - badsecrets: 2.07KB
[DBUG] - git: 2.07KB
[DBUG] - hunt: 2.07KB
[DBUG] - aggregate: 2.07KB

I'm happy to upload the entire log file or include however much of the log file you need somewhere if required.

[TheTechromancer]

nuclei is using up most of my memory

Ah, that might explain it then. The BBOT nuclei module processes URLs in batches though, so you might be able to cut down on its memory usage by decreasing the batch size.

# default batch size is 200
bbot -c modules.nuclei.batch_size=25 ...

I'm also curious to know exactly how much memory the BBOT process itself was taking when it crashed.

[ooholo]

As far as memory goes, I don't have anything set up to periodically monitor the process list and output; Though I do have the debug messages yelling at me saying I have no system memory left. The last one I received said "System memory is at 98.8% (24.57MB remaining)" (This is with 2GB of System RAM and a 2GB Swapfile) ((Unsure if this information means anything other than I ran out of memory, but there's a number at least. At a rest, my system uses around 400MB of ram. Just FYI. (: ))

[TheTechromancer]

Okay, I am willing to start an effort to decrease BBOT memory usage; however first we need to be sure it's BBOT that's hogging the memory, and not just nuclei. From the status message, it looks like the modules are using very little.

Here is a simple bash loop that will log the memory usage (in percent) of the bbot process every 10 seconds.

while true; do
    ps aux | grep bbot | awk '{print $4}' >> mem_log.txt
    sleep 10
done

After running this, the final few lines of mem_log.txt should contain the memory usage of bbot around the time it was killed.

[TheTechromancer]

Closing due to inactivity. Please feel free to reopen if needed.