Suspected memory leak

[cannadayr]

Hello, I am getting some behavior that makes me suspect there is a memory leak happening when removing and adding labels to a graph.

I have some code similar to the following:

[id] = MyApp.Server.get_by_label(label_val) # genserver callback w/ graph reducer
[label] = MyApp.Server.get_label(id) # wraps vertex_labels
new_label = Map.put(label,"foo",true) # update label
MyApp.Server.rm_label(id) # wraps remove_vertex_labels
MyApp.Server.add_label(id,new_label) # wraps label_vertex

This causes BEAM to run out of memory after a few hundred calls to this function. Let me know if you'd like a self-running example or if theres any tracing/debugging I can do to get more info.

[bitwalker]

If you can open a PR with a test that reproduces the issue, I can take it from there - if it will be too tricky to get it repro'd in a test, then a standalone app is fine too; the main thing is just that I have a way to do reliably see the same memory behaviour you are seeing.

One thing to keep in mind, especially when working with binaries, is that within the context of a process, binary data will be shared - the original binary which is referenced won't be released until all of the shared references (sub-binaries) are garbage collected, which can take some time if the process isn't particularly active, at least once the heap grows large enough.

You can see if this is the problem by examining the memory allocations in Observer, or dumping them with :erlang.system_info/1, e.g. :erlang.system_info(:allocated_areas). You can do the same at the process level with :erlang.process_info/1, e.g. :erlang.process_info(:binary), the :garbage_collection and :garbage_collection_info flags can be useful as well. What you are looking for is signs that objects are accumulating on the process heap, and garbage collection is not collecting enough in each cycle.

Based on what your snippet indicates, I have a feeling it is probably sub-binary references causing the memory leak - if that's the case there are a couple of strategies for dealing with that. One is using :binary.copy/1 on binary values you are inserting into the graph to force them to not reference the original large binary. If that's not doing it, then explicitly forcing a collection may be necessary - perhaps after remove_vertex_labels is called.

Once you post a repro, we can dig in further :)