Evaluation Code for BDD100K Dataset

[eliesl]

Hi, Thanks for sharing this nice repo. Would you please provide the evaluation code for BDD100K dataset? Or give an advice where to find such code? This would be a great help. Thanks!

[InhwanBae]

Hi @eliesl

Thanks for your interest in my implementation. I am currently unable to access the server, so I will update the code as soon as possible.

In the case of BDD100K, like the semantic segmentation task, you can write a dataloader that reads the train-val-test image list and return the corresponding road image and lane segmentation image. In the implementation detail of the ENet-SAD paper, label dilation is specified during training, but I remember that the line was already thicker in the author's pre-processed label. Therefore, you can ignore this part when implementing the dataloader.

Additionally, in the case of BDD100K, since it is binary segmentation, the model and loss function should be slightly changed.

[InhwanBae]

Hi @eliesl

I just update dataloader and model for the BDD100K dataset. Check it out!

However, I found that the evaluation code for BDD100K did not work anymore, so I sadly exclude it. This will take some time to fix it. I simply copy the test_CULane.py and remove some unnecessary parts like prob2lines_CULane, and add Accuracy & IoU calculation code from semantic segmentation libraries. Hope this helps you.

[eliesl]

Hi, Thank you for updating the codes for the BDD100K dataset. That's a great help! As suggested by you, I am using a modified version of test_CULane.py for the inference mode of the BDD100K dataset. However, when I load the saved model (net.load_state_dict(save_dict['net'])), it gives me the following mismatched size error. Do you have any suggestion? Thanks for all your help.

Missing key(s) in state_dict: "ce_loss.weight". size mismatch for transposed_conv.weight: copying a param with shape torch.Size([16, 1, 3, 3]) from checkpoint, the shape in current model is torch.Size([16, 5, 3, 3]).

[InhwanBae]

Hi @eliesl

This problem is because of the channel mismatch of CULane(5: BG, LL, L, R, RR) and BDD100K(1: binary segmentation). You can fix it by simply add the dataset argument to test_CULane.py L66 like train.py L71. Hope this helps you.

[eliesl]

Thank you for your help. It solved the issue. I have another question regarding the data in the test mode. For the evaluation purpose, I need to have the ground truth labels. Using "segLabel = sample['segLabel']" returns "None" in the test mode. Do you have any suggestion? Sorry for asking so many questions as I am new to the PyTorch. Thanks for your time!

[eliesl]

I also have another question. How many epochs did you train your network on to get the reported results? I read in your paper that the total number of training episodes for BDD100K is 60K, and I can see that max_iter is set to 60000 in cfg.json. Based on what is defined for Max_Epoch in L233 of train.py, Max_Epoch will be 12. I change L233 to "exp_cfg['MAX_EPOCHES'] = exp_cfg['MAX_EPOCHES']" so that I can directly set the max_epoch in cfg.json. Again thanks for your help.

[InhwanBae]

Hi @eliesl

To get 'segLabel', you can modify the BDD100K dataloader. I update it on the main branch because I thought that other people might need this part as well.

Also, for fairness, I trained the model for all datasets by 60k iterations, the same as the paper.

[eliesl]

Hi again! Thanks for being so responsive to my comments. Also, thanks for updating the dataloader. Beside the modifications you made, a minor modification is also needed for Lines 59 and 60. Just in case that anybody else wants to use segLabel for the test mode.

When I train the network, all the predictions for the validation images (during the training) and test images (test mode) are background. I get totally black images as the segmentation mask. I also tried training the network for way more that 60k iterations, and I tried different optimizers and learning rates but no luck! Do you have any suggestions?

[DaniloXiao]

Hi, I would like to ask the author what is the calculation formula of the BDD100K evaluation index pixel accuracy. The formula is not listed in the paper. Is it category pixel accuracy (acc = (TP) / TP + FP)?

[InhwanBae]

Hi @eliesl Sorry for the late reply. I also experienced the same problem while training BDD100K. In my case, rather than applying several balancing techniques, restarting the training process was more effective when the lane disappears in the initial convergence.

[InhwanBae]

Hi, I would like to ask the author what is the calculation formula of the BDD100K evaluation index pixel accuracy. The formula is not listed in the paper. Is it category pixel accuracy (acc = (TP) / TP + FP)?

@DaniloXiao Please refer to #16

[InhwanBae]

I'm closing this issue for now. Please feel free to open another issue for other questions.