Training Duration

[tigeriv]

Hi, thanks for your interesting open-source work. I am currently attempting to train the network and had some questions regarding the training process:

• Do you train the network from scratch or initialize using the pre-trained weights? And do you freeze them or leave them learnable?
• Training seems to take several hours per epoch, and the paper states 50 epochs. Do you actually train for 50 epochs? This would take my 3090 GPU at least a week.

[rmarcuzzi]

Hi, thanks for your interest in our work!

• For faster convergence we use the pretrained weights from MaskPLS. We didn't freeze them to train the final model but while developing, freezing the backbone weights (not the decoder) gave very good results in a much shorter time.
• The training took very long per epoch (around 8h) because at each step we're processing several (N_SCANS) point clouds sequentially. Therefore, training each epoch is equivalent to processing N_SCANS times the full training set.
• We trained the full 50 epochs but we obtained those final results already in an earlier epoch (I don't remember exactly but maybe at epoch 15). However, if you just want good results (but not the final ones), training for a few epochs should be enough.

In summary, if you want to try out stuff while developing you could freeze the backbone weights and the model will show very good performance after a few epochs but you'll have to unfreeze everything and train for more epochs to obtain the final results.

[tigeriv]

Thanks!