search

rbonghi / jetson_stats

📊 Simple package for monitoring and control your NVIDIA Jetson [Orin, Xavier, Nano, TX] series
https://rnext.it/jetson_stats
GNU Affero General Public License v3.0
2.14k stars 261 forks source link

Mem graph doesn't update after crash of the program when used all memory #520

Open martincerven opened 4 months ago

martincerven commented 4 months ago

Describe the bug

When using models that consume all available memory and eventually crash, the jtop doesn't update, only after ctrl+c, jtop it shows available memory correctly.

To Reproduce

Steps to reproduce the behavior:

  1. use audiocraft on Orin Nano with monitor
  2. When running jupyter cells, it will eventually run out of memory and crash container
  3. Mem won't update, only after closing and reopening

Expected behavior

I would expect I won't have to restart jtop

Additional context

When running any (pytorch) models that consume more memory than available, the jtop doesn't update after they eventually crash.

Board

Happened to both Orin Nano 8GB devkit, when trying audiocraft, and AGX 32GB devkit when running larger models (in containers)