Evaluation results not accurate?

[luceeleven]

Hi, I've been trying to recreate your results from the Hybridnets paper. I've run the eval code on the 100k dataset but the results I'm getting are nowhere close to the actual results you presented in the paper. I'm sure I'm missing something here, could you please tell me what could possibly be going wrong here. I think the root locations that I'm giving are inaccurate.

data root - raw images from the 100k dataset label root - I saved separate json file for each image from the "bdd100k_labels_images_val.json" that I downloaded from the bdd100k dataset. So in total I saved 10,000 separate json (one for each image) into another folder named "val". Road labels in Seg_list - I used the drivable masks Lane labels in Seg_list - I used the lane masks

The output that I got after evaluation of 100 images:

The iou values are inconsistent with the ones mentioned in the paper, they are nowhere near them. I was also wondering why the precision is so low, is there a specific reason as to why there are so many false positives. I hope you can help me out, thanks.

[xoiga123]

This is what I got with 100 images. Are you sure you're using the same road and lane labels as us? You can uncomment some cv2.imwrite blocks in dataset.py to check if they're correctly set up.

Precision is low because we chose the confidence threshold where Recall is maximum to print out. It's reasonable that when confidence threshold is lower, you get more predictions (recall up) but they may not be correct (precision down).

[luceeleven]

Thanks a lot, I was indeed giving wrong mask location, I changed it to colormasks and now I'm getting the correct values. For the precision values, it should increase if I increase the confidence threshold right?

[xoiga123]

@luceeleven yeah, try python val.py --conf_thres 0.5 now