Out of memory

[mgrigorov]

I'm using virtual machine with 2GB RAM from one of the cloud providers. I faced out of memory error with amass. Could you please help me with this issue?

$ uname -a
Linux bbsm 4.17.9-1-ARCH #1 SMP PREEMPT Sun Jul 22 20:23:36 UTC 2018 x86_64 GNU/Linux

$ go version
go version go1.10.3 linux/amd64

$ amass -version
version v2.3.3

$ amass -active -d criteo.com -o output.txt

amass_stacktrace.txt

[mgrigorov]

Updated amass to the latest v2.4.1 but still out of memory error. amass_stacktrace_2.4.1.txt

[mgrigorov]

Tried GCP VM instance with 8GB memory and still out of memory amass_stacktrace_gpc.txt

[caffix]

I'll take a look at reducing the number of goroutines being created for DNS resolution. Thanks!

[mgrigorov]

Thanks @caffix! JFYI, VM instance with 26 GB RAM also faced out of memory error. amass_stacktrace_GCP_26GB_RAM.txt

[caffix]

Very strange! My development system has never run out of memory during an enumeration, even when the domain contained thousands of subdomain names.

[mgrigorov]

Yeah, it's strange...

[mgrigorov]

Just for the record -

It looks like a memory leak. VM is almost ran out of free RAM. Amass meanwhile generated ~5MB file with ~174k subdomains.

---

This time before application faced out of memory error it generated 6.06 MB file with 197133 subdomains.

[CharlieEriksen]

Also having out of memory issues on v2.4.1. This is even on a machine with 8gb RAM.

[caffix]

I'd appreciate it if you could try again with 2.4.2 and share the results. Thank you!

[mgrigorov]

Sure, no worries

[mgrigorov]

@caffix,

I tried to run amass v2.4.2 on GCP VM instance with 3.75 GB RAM. And it ended up with out of memory error :( It ran out of memory pretty quickly

real    6m0.474s
user    3m29.752s
sys     0m54.460s

---

Currently running amass on VM instance with 13 GB RAM...

---

GCP VM instance with 13 GB RAM also faced out of memory

real    22m47.400s
user    16m2.684s
sys     4m20.836s

[CharlieEriksen]

Seems to happen mostly for me when I run into domains that answer all queries with a result. If I filter out those, I don't ever run out of memory.

[caffix]

Earlier, I was able to reproduce the behavior you have been experiencing. Try version 2.5.0, and let me know if your situation improves. Thanks!

[mgrigorov]

Still no luck =( VM instance with 15 GB RAM faced OOM error.

amass v2.5.0

amass -active -d criteo.com -o results.txt

[h3ku]

I'm facing the same issues with the latest version of release (amass v2.5.0)

I really don't know why.

[CharlieEriksen]

I'm seeing improved behaviour in this regard on 2.5.1. amass no longer takes up 100% CPU the first 1-2 minutes now at least.

[h3ku]

I'm facing the same memory overhead problems on 2.5.2.

[caffix]

The problem seems to be less severe, yet I have been able to recreate the issue, and will continue to dig into it.

[CharlieEriksen]

I'm not running into it anymore. But I've also started running an amass process for each domain. So I'm not passing multiple domains to each process, which probably helps.

[caffix]

I am seeing even better results for version 2.6.1! Can some of you test this and provide feedback? Thank you in advance!

[mgrigorov]

I tried v2.6.1 on VM with 15 GB RAM - fatal error: runtime: out of memory. amass -active -d criteo.com -o results.txt

[caffix]

@mgrigorov Your target has a wildcard capable of tricking the Amass detection algorithm. Thank you for bringing me this interesting test case! In the meantime, try using the following command instead:

amass -v -ip -active -bl widget.criteo.com -d criteo.com -o criteo.txt

I'll see what can be done to improve the wildcard detection.

[mgrigorov]

amass -v -ip -active -bl widget.criteo.com -d criteo.com -o criteo.txt

wow, thanks for helping! Worked like a charm!

[caffix]

@mgrigorov Happy to be able to help, and we'll continue to address the memory issue.

[caffix]

Since the Amass memory consumption has been behaving well, I'm going to close this issue unless users continue to experience problems.

[manuelbua]

I was going to give the latest version a try but i can still easily go out of memory, both on a cloud VM (1GB RAM) and locally (16GB RAM). This is the command i'm using, note that i was initially also bruteforcing, then tried to not to, but to no avail:

amass -version
version 2.8.1

amass -v -ip -active -o hosts.txt -d yahoo.com

...(~3-5 minutes passed)..

fatal error: runtime: out of memory

wc -l hosts.txt
1262 hosts.txt

[Jineeshak]

@caffix 9 GB memory usuage , while running amass intel command

[phackt]

Hello, i m experiencing the same issue (below dmesg results) with command amass enum -active -p 443 -src -norecursive -brute -w wordlist.txt -json output.json -d netflix.com:

[1433251.205365] Out of memory: Kill process 5442 (amass) score 655 or sacrifice child
[1433251.205439] Killed process 5442 (amass) total-vm:12778192kB, anon-rss:11306512kB, file-rss:0kB, shmem-rss:0kB

or

[4856371.424197] Out of memory: Kill process 31136 (amass) score 764 or sacrifice child
[4856371.425925] Killed process 31136 (amass) total-vm:6376016kB, anon-rss:6232076kB, file-rss:0kB, shmem-rss:0kB

Any help should be welcomed, thanks.

[fersingb]

Hello,

Same issue here. The command amass enum -d <domain> will trigger a OOM after some time if the domain has a lot of subdomains (VPS with 2GB of RAM, Amass Version v3.0.23)

Is it possible to reopen this issue or should I report this as a new issue?

[caffix]

@fersingb A VPS of that size is likely to cause an out-of-memory error due to the graph database requirements

[fersingb]

@caffix I understand, thanks for the answer. What's the minimum setup required to run amass?

[caffix]

People that use a 4GB VPS do not seem to have trouble, but it really just depends on how large your scope is and the size of your wordlists used for brute forcing techniques

[Rz-Rz]

Just got OOM issue on a 4gb VPS! My wordlist might be too long, I'm trying on a 16gb ram VPS

[Swiffers]

"Dirty" but working: enable swap memory if you are low on ram.