Bug in "dominos" or "size" (or my understanding thereof)?

[d-maurer]

Currently, I try to analyze an apparent memory leak (under Linux) using different types of analysis, among them the following based on heappy:

class GcHpy(SnapshotData):
  """Statistics based on `gc` and `heapy` (ATTENTION: expensive and blocking)."""
  # @gc_collect_after
  def do_sample(self, view):
    gc.collect()
    h = hpy()
    objs = h.idset(gc.get_objects()).dominos
    return dict(
      count=objs.count,
      size=objs.size,
      garbage=len(gc.garbage)
      )

The idea is to let gc determine the gc managed "containers" and from them determine all dominos (for unknown reasons is this much faster then hpy.heap()) and their size.

For a concrete observation, this approach reports about 1.3 GB as size while at the same time mallinfo and malloc_info (they report the status of the "libc" memory manager) indicate that the "libc" memory manager has delivered less than 0.8 GB to the application (even requested less than that amount from the operating system).

Obviously, there is an error somewhere. Have you an idea?

[svenil]

Humm, maybe Python shares some internal data structures of for example lists until they get used and modified. And we don't detect that. Should need some rethinking how we account for objects.