owl-project / NVISII

Apache License 2.0
319 stars 27 forks source link

Allocating NVISII to specific / multiple gpus #158

Open liswp opened 1 year ago

liswp commented 1 year ago

Hi NVISII team,

Thank you for this great repo!

Quick question: suppose you have three GPUs in a computer, but you only want NVISII to use one of the three GPUs, not the other two - is there a way to do that?

Another question: by default, NVISII will occupy all three GPUs with the same amount of memory on each GPU, but the amount of GPU memory allocated per GPU appears to be the same no matter how many GPUs you have in total for NVISII. The speed of rendering doesn't appear to be much faster either when you have multiple GPUs.

In this case, what is the benefit of using multiple GPUs for NVISII, if both the memory and the speed of rendering is the same regardless?

Finally, is it possible reduce GPU memory allocation per GPU for NVISII by providing multiple GPUs to NVISII? For example, if NVISII needs a total of 24 GB of GPU memory, is it possible to run NVISII on 3 GPUs, each with 11 GB memory?

Thank you!

natevm commented 1 year ago

Hello!

For your first question, I recommend setting the CUDA_VISIBLE_DEVICES environment variable on your system to control how many GPUs NVISII uses.

For your second question, multiple GPUs only really makes sense for very complex scenes where it’s difficult to achieve more than one or two frames per second on a card. For simpler scenes where framerate is higher, there is more overhead synchronizing the cards and then multiGPU doesn’t make as much sense. Instead, I recommend rendering the different frames on different cards using multiple processes.

And for your final question, we unfortunately do not currently support data distributed rendering. We only support data replicated rendering (ie, copying the same data to each card). Data distributed ray tracing is an open problem that we are researching though!

liswp commented 1 year ago

Hi @natevm,

I tried CUDA_VISIBLE_DEVICES and it doesn't seem to have any effect though. Could you please provide some guidance here?

Thanks.

liswp commented 1 year ago

Hi @natevm,

In particular, I tried setting CUDA_VISIBLE_DEVICES=-1, or CUDA_VISIBLE_DEVICES=0, or CUDA_VISIBLE_DEVICES=1, CUDA_VISIBLE_DEVICES=2, etc.

In all these cases, it seems like NVISII is still using all GPUs.

Thank you and please help.

natevm commented 1 year ago

How are you setting your environment variable? We use CUDA_VISIBLE_DEVICES and know that it does work, so it sounds like some issue with the environment variable not “sticking”.

Note, you must set this variable before running Python. Setting the environment variable from within Python itself doesn’t work iirc.

Maybe in your Python script for NVISII you should try reading the CUDA_VISIBLE_DEVICES and see if the interpreter has properly picked up your environment.

TontonTremblay commented 1 year ago

I have been able to run nvisii on 8 gpus on a single machine.

CUDA_VISIBLE_DEVICES=1 python render.py 

I am not saying you are doing it wrong, just that we were able to test this without any issues in the past and this was the mechanism we decided to go with for selecting GPU vs multiple GPUS.

liswp commented 1 year ago

Thank you! Problem solved as you have suggested! Thanks again.