Closed NK-CS-ZZL closed 1 year ago
Haven't noticed any issues myself, how many GPUs are you using and what is nproc_per_node
for the DDP training?
Thanks for your reply. My pc has two RTX 3090s and nproc_per_node
is 2.
BTW, we call this func softsplat(tenIn, tenFlow, None, 'avg')
as the same in softmax-splatting. I think at least I use it in the right way, because I used this code get good inference results from M2MVFI and SoftSplat. However, it doesn't work on multiple GPUs. Do we need to guarantee the tensor is contiguous before we conduct forward warping? (Our tenIn
is not contiguous, maybe this causes the problem?)
I am afraid that I am not sure why it would fail, if you are worried about the tensors being non-contiguous you can always pass them as tensor.contiguous()
to the softmax splatting function. Maybe the input tensor and the flow tensor are on different CUDA devices? As a hack you could run torchrun twice, once with CUDA_VISIBLE_DEVICES
set to 0 and once to 1 such that you effectively have two training runs on the same machine in parallel.
Thanks for your patience and I'll try these suggestions. BTW, forward warping is an interesting idea. XD
I solve this bug by setting torch.cuda.set_device
manually. If not, torch.cuda.current_stream()
would return a stream in the wrong device (always be 'cuda:0').
Thanks for sharing your finding! Good to know about that behavior of current_stream()
, I wasn't aware of it. And good luck with your work/research!
Congratulations on your awesome work. The 'softsplat' func works well under a single GPU setting. However, we encounter an error in the DDP training with more GPUs. So did you tried to train the network with more than one GPU, or the 'softsplat' func cannot support this. Here is the error info and our environment is RTX 3090/cupy11.6/pytorch11.1/cudatoolkit11.3 (Though cuda10.0/pytorc1.8.0 are recommended in README, they are incompatible with RTX 3090).