Open kenvix opened 2 months ago
Hi @kenvix, thanks for raising this.
I tested it locally, but I cannot reproduce this. I use a script to create and terminate 10k processes on the GPU.
import time
import ray
import torch
@ray.remote(num_cpus=1, num_gpus=0.1)
def request_gpu():
torch.zeros(1000, device='cuda')
time.sleep(10)
ray.init()
_ = ray.get([request_gpu.remote() for _ in range(10000)])
The memory consumption of nvitop
is stable around 260M.
Hi, @XuehaiPan
The test code you provided does not seem relevant to this issue. In my case, using tmux or screen to keep nvitop running, you will find that the memory (RAM, not GPU VRAM) usage of nvitop itself will continue to slowly increase over time.
For my example below, I ran it for 12 hours:
It used about 4.5G RAM
@kenvix could you test nvitop
using pipx
? Maybe it is caused by a dependency (e.g., the unofficial pynvml
package).
Running after 2 days:
Required prerequisites
What version of nvitop are you using?
1.3.2
Operating system and version
Ubuntu 22.04.4 LTS
NVIDIA driver version
535.104.12
NVIDIA-SMI
Python environment
3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] linux nvidia-ml-py==12.535.133 nvitop==1.3.2
Problem description
Nvitop has memory leaking issue for instances inside docker container. (RAM, not VRAM). Even the operating system takes ten seconds to reclaim memory after SIGKILL a process.
Steps to Reproduce
Just keep running
nvtop
about few months. You'll seenvtop
consumed a lot of system memory. About 300GB in 77 days for my instance.Is this caused by nvitop recorded too much vRAM and GPU utilization information but not releasing it?
Traceback
No response
Logs
No response
Expected behavior
No response
Additional context
No response