Training freezes and gets stuck

ubc-vision / 3dgs-mcmc

Implementation of the paper "3D Gaussian Splatting as Markov Chain Monte Carlo"

https://ubc-vision.github.io/3dgs-mcmc/

Other

254 stars 10 forks source link

Training freezes and gets stuck #6

Open henrypearce4D opened 1 week ago

henrypearce4D commented 1 week ago

Training freezes and gets stuck, happens at low iterations


Reading camera 176/176 [25/06 11:43:34]
Generating random point cloud (100000)... [25/06 11:43:34]
Loading Training Cameras [25/06 11:43:35]
Loading Test Cameras [25/06 11:44:43]
Number of points at initialisation :  100000 [25/06 11:44:52]
Training progress:   3%|█▎                                         | 900/30000 [00:30<05:29, 88.41it/s, Loss=0.4608608]

mbernier-arcturus commented 1 week ago

Check your GPU memory, in some cases during training it goes haywire and GPU memory bust. I got this error while trying different combinations of parameters.

shakibakh commented 6 days ago

Can you please specify the setup and the shape you are trying to run and also your environment? The code is only tested on Ubuntu, the specifics are now updated in README.