search

volatilityfoundation / dwarf2json

convert ELF/DWARF symbol and type information into vol3's intermediate JSON
Other
104 stars 28 forks source link

Linux: OOM when a kernel debug symbols is used as input #20

Closed gcmoreira closed 3 years ago

gcmoreira commented 4 years ago

Dwarf2json out of memory when parsing Linux kernel debug symbols. The QEMU VM used for this have Ubuntu 18.04.4 amd64 stock kernels 4.18.0-25-generic and I also tried with 5.4.0-42-generic. The command lines were: ./dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.4.0-42-generic or ./dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.4.0-42-generic --system-map System.map-5.4.0-42-generic

The dmesg error was:

[  148.628383] Out of memory: Killed process 1322 (dwarf2json) total-vm:5022196kB, anon-rss:3736096kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:8012kB oom_score_adj:0

I tried from inside of the virtual machine first with 2GB of RAM which later increased to 4GB and 8GB but it didn't fix the problem. Finally, I got dwarf2json working in a 16GB RAM host. I'm aware Dwarf itself requires a lot of memory but I think it would be great to see if this limitation could be addressed in the future.

ilch1 commented 4 years ago

Thanks for reporting the issue. I haven't seen an OOM error before, but my machine has 16 GB of RAM.

I would need to run a profiler to determine what is using the memory. While some optimizations may be possible, the output format is such that the entire symbols structure resides in memory before being written out as JSON. My guess is that is taking up a large chunk of total memory by dwarf2json.

At the very least, we can update documentation with recommended RAM.

NiklasBeierl commented 3 years ago

Experiencing an OOM as well in a 4gig VM, had to run it on the host.

ilch1 commented 3 years ago

Documentation added in PR #29.