search

google / binexport

Export disassemblies into Protocol Buffers
Apache License 2.0
1.04k stars 203 forks source link

Ghidra export out of memory for a large file #64

Open justanotheranonymoususer opened 3 years ago

justanotheranonymoususer commented 3 years ago

See screenshot below. Can be reproduced with electron.exe, file size: about 120 MB.

image

cblichmann commented 3 years ago

Hi there!

Can you tell me a few more details? What's the peak amount of memory that the export process uses (roughly)?

How much RAM does your system/VM have (relevant because various JVMs set different max heap size defaults)? Any special JVM settings in `support\launch.properties'?

And finally, as I have never tested with Amazon Corretto, can you please try with a recent OpenJDK build, from the official site.

justanotheranonymoususer commented 3 years ago

Hi!

What's the peak amount of memory that the export process uses (roughly)?

Is starts with "exporting instructions" at about 3 GB, then grows slowly but steadily until it fails at about 9 GB.

Here's a boring video of it (speed x25): https://imgur.com/KDLyxLX

How much RAM does your system/VM have

32 GB, I'm running on a physical PC, not a VM.

Any special JVM settings in `support\launch.properties'?

No, everything is set to default.

can you please try with a recent OpenJDK build

I'll see if I can get it running...

cblichmann commented 3 years ago

I see. I just now loaded the most recent version of electron.exe into Ghidra 9.2 on my Mac. It's still disassembling, though.

Will try this out for myself as soon as it's done.

cblichmann commented 3 years ago

I stopped Ghidra after 1h and 84%, as it was clearly not going anywhere. Situation is similar to yours, it hovers at around 9.1GiB and the garbage collector is running wild on 4 cores.

For comparison, I have exported the file with IDA Pro, which took 1.5h to finish the initial auto-analysis. The native BinExport plugin took another 14m 45s to produce a protobuf that is 661MiB in size.

What I suspect is going on is a) Ghidra limits its heap space somewhere and b) the Java Protobuf bindings are at a disadvantage by requiring tons of Builder objects to be created, leading to intense GC pressure.

justanotheranonymoususer commented 3 years ago

Ghidra limits its heap space somewhere

According to this comment: https://github.com/NationalSecurityAgency/ghidra/issues/1997#issuecomment-645627337, that's correct. I'll try to increase the limit and see whether it helps.

Not sure about b.

justanotheranonymoususer commented 3 years ago

It worked! Peak memory usage: 17 GB: https://i.imgur.com/bAOqDUl.png What's strange is that the memory usage stays at 17 GB after the export. Perhaps there's a huge memory leak, which can explain the problem.

Martmists-GH commented 1 year ago

Still an issue on latest version, setting MAXMEM to 28G did not resolve the issue.