[UPRISE]CUDA out of memory. Tried to allocate 3.25 GiB. GPU

[zhouchang123]

When run the code:bash inference.sh,it occurs an error.

[zhouchang123]

I use gpustat -i 1 to watch the gpu state . When it went wrong ,the gpu 2 and 3 were not on the limit ,only 0 went out of memory.(My friend is using 1) How to solve it?

[cdxeve]

1. Try reducing the inference batch_size to 4 or even lower:

   
   2. add CUDA_VISIBLE_DEVICES='0,2' if necessary.

[zhouchang123]

Yes,it works. But it still shows that the use memory on gpu 0 is double of gpu 2

[cdxeve]

This is okay, sometimes the model can load unevenly across different GPUs.