Suspected memory leak: large amount of RAM is only freed after another file is opened

[romanholidaypancakes]

Version and Platform (required):

• Binary Ninja Version: 4.0.4911 free, 15e9149
• OS: windows
• OS Version: 10
• CPU Architecture: x86_64

Bug Description:

Steps To Reproduce: When multiple files are opened for analysis in the same process, only the tab page for file analysis is retained; it can be found that the memory does not decrease.

Expected Behavior:

Screenshots:

Additional Information:

[xusheng6]

Could you please provide repro steps? I am not exactly sure what you mean (I know what you mean in general, but what in enough detail for me to reproduce it)

[romanholidaypancakes]

1. Start a binaryninja
2. Open three files a b c and wait for analysis (waiting for memory to increase) to complete.
3. Close the tab page corresponding to b c
4. At this time, binaryninja should theoretically release the memory corresponding to b and c; but it does not

[negasora]

Does it repro with the latest free version? Should be 4.1.4958

[romanholidaypancakes]

Does it repro with the latest free version? Should be 4.1.4958

The new version I can download is 4.0.4958 (2024-03-15). In this version, part of the memory has been released, but most of the memory still does not seem to be released correctly.

[psifertex]

Pretty sure that was a typo. The last four matching are the most importabt bits.

[xusheng6]

Hi @romanholidaypancakes, I tried but cannot reproduce this. How did you measure the memory usage?

Also, by the design, type objects are always leaked even after closing the file, though I do not think that should cause significant memory usage

[romanholidaypancakes]

https://streamable.com/p6zic8

1. Memory usage at startup is 30.2mb
2. Analysis first completed: 681.4mb
3. Analysis second completed: 930.72mb
4. Analysis of the third completion: 1.14g
5. Close third: 1.02g
6. Close the second one: 762.61mb
7. Close the first one: 417.09mb

[negasora]

If after those steps you:

1. Open the file again
2. Let analysis finish
3. Close the tab

Does memory usage drop back to ~417MB?

[romanholidaypancakes]

Open the file again

Close the tab

[xusheng6]

From the given information, I am not exactly sure whether there is a memory leak. Usually we do not use the memory usage info reported by the system as the measurement for memory usage. We use the following command in the Python console:

>>> get_memory_usage_info()
{'Active BasicBlock objects': 72, 'Active BinaryView objects': 2, 'Active Component objects': 0, 'Active FileMetadata objects': 1, 'Active FlowGraph objects': 0, 'Active Function objects': 4, 'Active LowLevelILFunction objects': 12, 'Active MediumLevelILFunction objects': 16, 'Active Type objects': 2086, 'Active TypeArchive objects': 0}

Which is, unfortunately, unavailable in the free version. That said, I have performed the process on the latest dev builds of the free version (which is not available to the public), and the above command will report 0 binary view objects after the operation. Which, according to our standard, means there is no outstanding memory leak.

The memory usage you are seeing is a bit deceptive. You might think the initial RAM usage is like 30.2 MB, and in the end it becomes 417 MB and there must be a memory leak. However, this is not the case. Since, when you simply open binary ninja and do not open any file, there are many things that are not loaded, e.g., type libraries. Type libraries are only loaded when you open a file, and by design they are not freed even after the file itself has been closed. So in other words, the RAM consumption of an idle binary ninja is ~417MB.

That is also the reason why my colleague is asking you to test if the memory usage would go down to 417MB after you open another file and then close it. If it does, then there is probably no memory leak. If it goes up a lot, then there is a memory leak. However, the memory usage is about 465MB after the process, and I cannot tell for sure whether there is a memory leak or not. We will need to look further into the issue

[xusheng6]

@negasora I noticed an interesting behavior, that if I open a very simple file as the second file (rather than opening the same file), the memory usage actually goes down (smaller than 417MB), even while the second file is open. I highly suspect that certain objects associated with the first file are leaked and they are only freed when a second file is opened. I have previously seen a case where the Python console causes delayed free-up of resources. Though this particular bug has been fixed a while ago

[xusheng6]

I can observe this on Windows, but not on macOS

[romanholidaypancakes]

In fact, the larger the memory occupied by the binary you initially analyzed, the more unreleased memory will be in the end. Finding this type of binary to test will help you analyze where the leakage point is. This is the memory footprint now that I don't have any files open:

[saagarjha]

I can reproduce this quite easily on my Mac. All I need to do is open a large file (which uses massive amounts of memory–30+ GB on my machine) and then close it. It is quite obvious that it's not being freed because my memory usage barely changes after doing that. In fact if I close all my tabs I get this output which seems unexpected:

>>> get_memory_usage_info()
{'Active BasicBlock objects': 5566127, 'Active BinaryView objects': 19, 'Active Component objects': 0, 'Active FileMetadata objects': 9, 'Active FlowGraph objects': 0, 'Active Function objects': 1455938, 'Active LowLevelILFunction objects': 1261, 'Active MediumLevelILFunction objects': 1688, 'Active Type objects': 2932766, 'Active TypeArchive objects': 0}