search

graphdeco-inria / gaussian-splatting

Original reference implementation of "3D Gaussian Splatting for Real-Time Radiance Field Rendering"
https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
Other
14.78k stars 1.93k forks source link

Train with multiply GPU #694

Open tfiliano opened 8 months ago

tfiliano commented 8 months ago

Is there any way to do it?

someone could point the way I need to follow to do it?

Thanks

jaco001 commented 8 months ago

As I know - they must rewrite cuda code part for it. Training part for now isn't bad, Only reason for rewrite code is very big datasets, but gaussians can be trained only from few pics.... so duno if authors do that. ;)

LoickCh commented 8 months ago

241

horford commented 8 months ago

could Dask be possibly integrated to distribute the job?

tfiliano commented 8 months ago

241

Yeah I have spotted that one, I can try to implement it, but Python, Cuda, etc is not my world

tfiliano commented 8 months ago

could Dask be possibly integrated to distribute the job?

I was looking at this https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html Once the project already uses Pytorch, I was thinking that it could be one possibility.

But yeah.. I will need some time reading the source code and trying to understand it to see how and where to apply this nn.parallel

TarzanZhao commented 4 months ago

This project enables distributed gaussian splatting training over multiple GPU: https://github.com/nyu-systems/Grendel-GS

Moreor1ess commented 4 months ago

Do you know to choose GPU? The default is to run on GPU 0, but I want it to run on GPU 1