RAM and VRAM monitoring

edgenai / edgen

⚡ Edgen: Local, private GenAI server alternative to OpenAI. No GPU required. Run AI models locally: LLMs (Llama2, Mistral, Mixtral...), Speech-to-text (whisper) and many others.

https://docs.edgen.co/

Apache License 2.0

328 stars 15 forks source link

RAM and VRAM monitoring #91

Open pedro-devv opened 6 months ago

pedro-devv commented 6 months ago

Edgen should be capable of monitoring the current and RAM and VRAM usage. This will allow further functionality such as avoiding program crashes due to OOM allocations.

pedro-devv commented 4 months ago

Development for this is being put on hold, as at the moment it is not possible to properly track allocations in llama.cpp and whisper.cpp.