Open Windsoldier76 opened 1 year ago
There are a few session options might help: https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html
gpu_mem_limit: set some limt. arena_extend_strategy: kSameAsRequested (1) cudnn_conv_algo_search: HEURISTIC (1) or DEFAULT (2) cudnn_conv_use_max_workspace: 0
For example, set arena_extend_strategy to SameAsRequested could avoid allocating memory more than needed. Use a few images to warm up the service. Note that, first image might apply cudnn conv algo search, which might need a lot of workspace memory. Change cudnn_conv_algo_search and cudnn_conv_use_max_workspace could reduce memory usage (it could also impact speed since only a subset of algo is searched). I think gpu_mem_limit might also impact cudnn workspace.
Run option of memory.enable_memory_arena_shrinkage could be used to shrink arena memory, see example.
Try the following sequences to see whether it could help.
Describe the issue
Hi there,I'm a beginner so please forgive me if I say some silly words.
I'm trying to deploy multiple models on Jetson Xavier NX for real time detect.When I add to three or more models, memory is all used then my program is not responding.
Through jtop and mprof I found if using CPUProvider , each model only use about 300MiB.But using CUDAProvider, each model need nearly 1750MiB.In that case , if load more than two models, there is no memory can be used.
Is this situation normal?For real time detect,I must use multiprocess to decoupling video capture and video process, so I can't let cpu occupancy rate too high.Is there any way to reduce the memory occupancy with CUDAProvider?
btw swap space has opened through jtop but no use.Model is yolov5 to onnx and simplified.Model size about 27.1M
To reproduce
Urgency
No response
Platform
Linux
OS Version
ubuntu 18.04LTS
ONNX Runtime Installation
Built from Source
ONNX Runtime Version or Commit ID
onnxruntime_gpu-1.10.0-cp36-cp36m-linux_aarch64.whl
ONNX Runtime API
Python
Architecture
Other / Unknown
Execution Provider
CUDA
Execution Provider Library Version
CUDA10.2
Model File
test_simplied.zip
Is this a quantized model?
Unknown