search

PRBonn / lidar-bonnetal

Semantic and Instance Segmentation of LiDAR point clouds for autonomous driving
http://semantic-kitti.org
MIT License
959 stars 206 forks source link

A problem for training SqueezeSeg model #26

Closed jsgaobiao closed 4 years ago

jsgaobiao commented 4 years ago

Hello. I'm trying to train the SqueezeSeg model from random inital weights. However, the iou seems too low after 16h training (about 16.2%).

GPU: a NVIDIA TITAN X. Training set: sequence 0~7 Validation set: sequence 8 Arch config: squeezeseg.yaml (I just set batch_size=18 due to GPU memory limitation)

There are some information about my training in this pic may help. Do I need more time for training? Could anyone give me some suggestions? @tano297 Thanks a lot ! pic

jbehley commented 4 years ago

I would say that your curves indicate that you need more training time. You also see that the losses are still going down and the IoU goes still up.

jsgaobiao commented 4 years ago

I would say that your curves indicate that you need more training time. You also see that the losses are still going down and the IoU goes still up.

Thank you for your reply ! I will keep training it and see what will happen. Do you know how long it usually takes to train a model like SqueezeSeg ?

tano297 commented 4 years ago

I have just checked, and training was done for roughly 300 epochs. Convergence will be faster than that, at 150-200 epochs, but I kept training for fairness to the original method, because I wanted to make sure there would be no improvement. I'm closing this now, if there is more questions feel free to keep commenting.