pablogsal
commented
7 months ago I will look at the flamegraphs in detail tomorrow. In the meantime, to be able to better understand what's going on, do you mind sending us some reproducer or snippet that you used to create this flamegraphs? Memray tracks allocations everywhere that's not GPU (Python interpreters, native libraries, extensions...etc) so either there is something that we are missing and we need the code to understand what or there is some misconception over what's going on with the code. In any case, having something we can inspect will help us enormously.
albertz
godlygeek
drimeF0
@albertz you mentioned here https://news.ycombinator.com/reply?id=39327452&goto=item%3Fid%3D39325983%2339327452 that seems that there are some things that don’t make sense in your experience when you used memray.
Could you give us a small reproducer or explain a bit the problem so we can look into it?
Originally posted by @pablogsal in https://github.com/bloomberg/memray/issues/212#issuecomment-1937135298
I thought I do a new separate issue about this.
My current use case: I train some neural networks using PyTorch. I have already run hundreds of similar experiments before. Now I added one new aspect, and this leads to CPU out-of-memory (OOM). This was unexpected to me because I don't really see how the added code can lead to more memory consumption or a memory leak. In any case, I wanted to do some memory profiling, to better understand the issue, and maybe also find other things to optimize.
I guess Memray does not track GPU allocations, but that's ok, as I get CPU OOM here. I don't really know how it handles allocations by native Python libraries such as PyTorch though, e.g. some internal structures of PyTorch, PyTorch CPU arrays, or also Numpy arrays, etc.
So I used Memray, and then created some flamegraphs. I don't really know whether I should use the temporal option, or whether I should use the native option. For the profiling, I simplified the test case a bit, by using some dummy dataset instead of the real dataset, and also using only single GPU instead of multi GPU.
Here are two flamegraphs, one is temporal+native (which is huge), the other one just default options: https://gigamove.rwth-aachen.de/en/download/7a26a95088204b99c56cba8a5fb6ec54 (Link expires 2024-02-24.) Here is just the second flamegraph: memray-flamegraph-memray.output.html.zip
So, after some increase in memory allocation during startup (first 40 secs or so), it seems that the memory consumption is absolutely constant for the remaining runtime. This seems very unrealistic to me, and makes me doubt that this is accurate. Because this is eager execution, without using
torch.compileor so, so it allocates and deallocates a lot of Torch tensors in every step (ok, GPU tensors, but there should be some overhead on CPU memory as well), so I definitely would expect that I see some variations.Also, I don't exactly understand the order of frames. It doesn't seem to be ordered by most allocated memory, but it also doesn't seem to be ordered by time.
I don't understand what it means when a frame is marked yellow. I guess green is when it is Python, gray is when it is native. I also cannot read all the tooltips when the filename is too long.
It seems also strange to me that
torch.cuda.is_available()took 5.1GB, andtorch.cuda.mem_get_info()took 5.5GB, but maybe that's just how much CUDA takes.