search

TimChou-ntu / GSNeRF

[CVPR 2024] GSNeRF: Generalizable Semantic Neural Radiance Fields with Enhanced 3D Scene Understanding
11 stars 0 forks source link

Training is super slow.. #2

Open GopiRajuMatta opened 2 months ago

GopiRajuMatta commented 2 months ago

Hello @TimChou-ntu !

Thank you for releasing the code.

I tried running it on same configuration, RTX 3090 with 24GB, but it is surprisingly very slow. Can you please provide a work around for this.

How much time does it to take to fully train the model on Replica dataset?

Thank you Gopi

TimChou-ntu commented 2 months ago

Hello @GopiRajuMatta, For replica dataset, GT depth supervision version generally takes 1.5 days to converge while in self supervised depth regulations it takes less than a day to converge. I also do most of my experiments on 3090ti.

Hope these information will help~ Tim

GopiRajuMatta commented 2 months ago

Thank you for the reply.

I am running it for almost 27 hours, only 70k iterations got finished. This is with GT depth supervision. Not sure why, it is so slow. According to this 250k iterations will take almost 3 days..

Please let me know if anyway it can get faster.

Thank you Gopi

TimChou-ntu commented 2 months ago

Hi @GopiRajuMatta, I'm not sure the exact reason but just found that the repo I pushed accidentally set dataloader num_worker to 0. You can set the value to the number of CPU cores of your machine. That might help speed up.

https://github.com/TimChou-ntu/GSNeRF/blob/d0bc5261772e018437412c4092aa152c1ff15da6/train.py#L165

Tim

GopiRajuMatta commented 2 months ago

Thank you for your reply. Yes, I've noticed that as well, even after changing not much difference...