search

blacklanternsecurity / bbot

A recursive internet scanner for hackers.
https://www.blacklanternsecurity.com/bbot/
GNU General Public License v3.0
4k stars 366 forks source link

Ways to optimise memory usage? #1474

Closed felipewarrener closed 7 hours ago

felipewarrener commented 1 week ago

I woke up to my instance of bbot being killed after reaching ~9GB of memory usage:

[INFO] playful_howard: Events produced so far: DNS_NAME: 268356, IP_ADDRESS: 22462, TECHNOLOGY: 6000, ORG_STUB: 5649, FINDING: 436, STORAGE_BUCKET: 6, SCAN: 1
[INFO] playful_howard: 1 events in queue (DNS_NAME: 1)
Killed

root@localhost:~/blah-again/bbot# dmesg -T | rg -i kill
[Tue Jun 18 06:04:27 2024] systemd-resolve invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[Tue Jun 18 06:04:27 2024]  oom_kill_process.cold+0xb/0x10
[Tue Jun 18 06:04:27 2024] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=systemd-resolved.service,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-3.scope,task=bbot,pid=122557,uid=0
[Tue Jun 18 06:04:27 2024] Out of memory: Killed process 122557 (bbot) total-vm:9830984kB, anon-rss:7318860kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:15716kB oom_score_adj:0

Are there any flags where I can get memory to be free'd and results written to disk more frequently or something? This was with a list of 12,000 seed domains.

TheTechromancer commented 1 week ago

According to your output, there was only 1 event in queue at the time it was killed. Without seeing your debug.log, we can't know the cause for sure. However, it's likely it's the massdns module. Massdns needs to keep track of its mutations in order to keep from trying the same ones again. This has already been fixed in BBOT 2.0 through the use of a bloom filter.

I have to ask, are these 12000 domains related in any way? I.e. do they belong to the same parent company?

TheTechromancer commented 7 hours ago

Closing due to inactivity.