detailed timestamp with lineno indicator when do parse?

[leiwen83]

Is there an existing proposal for this?

• [X] I have searched the existing proposals

Is your feature request related to a problem?

Current I could see detailed top-N largest and location from stats command. But another important info I want to know is that during the specified timestamp, how many allocation has happened, which cannot get from stats result.

It seems to me that parse command could extract the info I want, but the log it extract is vague and hard to read. Is there some script to process to get what I want?

Thx

Describe the solution you'd like

I want timeline log format with detailed function location during specified timestamp range.

Alternatives you considered

No response

[pablogsal]

Hi @leiwen83.

I am afraid I don't see how you are asking is possible currently. The stats command show statistics about the whole run and there is noway to select a time range or specific timestamps.

Maybe you could check our new temporal flamegraph feature which does something similar to what you want.

[leiwen83]

Hi,

I try the temporal flamegraph feature, and feel great. But I have question for rendered result. In the bottom, there is framegraph for allocation traceback. it shows the "train" top entry consumes about 208KiB total, 78 allocations, which doesn't match the 600GiB grows in the selected range... So how could I tell where 600GiB comes from in the temporal flamegraph?

[pablogsal]

Can you share with us a small reproducer so we can investigate? Otherwise is impossible for us to answer your question

[leiwen83]

It is hard to reproduce the oom ... Whether could directly get some clue from the dumped result? Since the rendered flamegraph is generated from dumped bin, and the bin file is still there.

[godlygeek]

The flame graph, and the bin file that it was generated from, can only tell you details about heap usage, not about the resident set size (RSS). We only capture stacks for heap allocations, and we only know the total size for RSS. Whatever has caused the high RSS, the captured data won't be able to help you figure it out.

However the 600 GB was allocated, we don't have visibility into it. Perhaps it's memory-mapped files, or shared memory, or shared libraries, or caused by heap fragmentation that prevents allocated memory from being returned to the OS. It could conceivably be a Memray limitation, too: there might be some allocation function that we can't see into, and most of the memory is being allocated through that function.

We could dig into it and try to help you understand what's going on if we can get a small reproducer for the issue. It doesn't need to produce an OOM, we just need to see that RSS is growing quickly.

[leiwen83]

I see... But for now there is no way to create such small reproducer, as the memory quick grow itself is random one.

For double check, from the rendered flamegraph, we could get the conclude that for python code, it doesn't response for 600G growth? I used memray attach without additional flag like follow-fork or trace native for this dump.

[pablogsal]

Unfortunately without a reproducer we cannot be of further assistance or we cannot check if we are missing anything in memray. As this doesn't seem to be a bug, I am closing this issue for the time being. If you can give us a reproducer, please copy it here so we can further investigate. Thanks for your understanding!