search

ziyc / drivestudio

A 3DGS framework for omni urban scene reconstruction and simulation.
https://ziyc.github.io/omnire/
MIT License
595 stars 51 forks source link

parallel trainging #23

Closed zhangSaneC closed 1 month ago

zhangSaneC commented 2 months ago

I have a 16G GPU, and in training process, cuda out of memory error occur.

May I ask how to do parallel training like using two GPUs in this project?

Thanks for your reply.

ziyc commented 2 months ago

Hi @zhangSaneC, parallel training of Gaussian Splatting on multi-GPUs didn't exist until this new work came out. We haven't implemented support for it yet.

Typically, training for large outdoor scenes requires more GPU memory due to the millions of Gaussians needed for fitting. You can limit the growth speed of Gaussians by increasing the Gaussian densification threshold here: https://github.com/ziyc/drivestudio/blob/388642a6a998833cb388e0f1a65dfc2071cc6a61/configs/omnire.yaml#L60

Be careful though, as this will affect the final reconstruction quality. You may need to balance between computational load and final quality.