How to visualize huge bin file (over 2TB CPU Memory)

[qawnaoya]

Is there an existing proposal for this?

• [X] I have searched the existing proposals

Is your feature request related to a problem?

I have a huge bin file (over 2TB), How to visualize or extract information (especially large memory consume block or functions). I have limitation physical memory 512GB memory. I execute stats 3 hours.

Describe the solution you'd like

Visualizing or extracting information memory efficient, Not all data put on memory.

Alternatives you considered

No response

[pablogsal]

@qawnaoya have you considered using https://bloomberg.github.io/memray/run.html#aggregated-capture-files ?

[godlygeek]

What have you tried? I'd expect that memray stats would be the most likely reporter to work on a capture file that large, followed by memray summary and perhaps memray tree. I think there's virtually no chance of memray flamegraph --temporal working, though a non-temporal memray flamegraph might.

Whether most of these wind up working depends less on the size of the capture file, and more on the number of distinct stacks from which allocations were performed.

[godlygeek]

Without more info on what has been tried, there's nothing more we can do here.