search

wudongming97 / TopoMLP

[ICLR2024] TopoMLP: A Simple yet Strong Pipeline for Driving Topology Reasoning
Apache License 2.0
131 stars 11 forks source link

Some problems about Evaluation Metric between the 'OpenLane-V2 Score' and 'F-Score for 3D Lane' #12

Closed Benson722 closed 5 months ago

Benson722 commented 5 months ago

Thank you for your great job again! Your work is very helpful to me! :)

When I do the evaluation on Dataset B, I'm a little confused about the results of the evaluation.

  1. Are the results of the evaluation different by one percent from the results in the paper?
  2. The result of F-Score for 3D Lane is too low. Is there something wrong with my evaluation command?
  3. Where is the Ground Truth for the road centerline of data set B given? Where can I view the road centerline results output in Dataset_B by the model?
(openlanev2) zhangyiqing@inin:~/mmdetection3d$ ./tools/dist_test.sh projects/configs/topomlp_setB_r50_wo_yolov8.py /home/zhang         yiqing/mmdetection3d/checkpoints/topomlp_setB_r50_wo_yolov8_e24.pth 1 --eval=bbox
/home/zhangyiqing/miniconda3/envs/openlanev2/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The m         odule torch.distributed.launch is deprecated
and will be removed in future. Use torch.distributed.run.
Note that --use_env is set by default in torch.distributed.run.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See
https://pytorch.org/docs/stable/distributed.html#launch-utility for
further instructions

  warnings.warn(
/home/zhangyiqing/miniconda3/envs/openlanev2/lib/python3.8/site-packages/mmdet/utils/setup_env.py:38: UserWarning: Setting OMP         _NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further t         une the variable for optimal performance in your application as needed.
  warnings.warn(
/home/zhangyiqing/miniconda3/envs/openlanev2/lib/python3.8/site-packages/mmdet/utils/setup_env.py:48: UserWarning: Setting MKL         _NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further t         une the variable for optimal performance in your application as needed.
  warnings.warn(
load checkpoint from local path: /home/zhangyiqing/mmdetection3d/checkpoints/topomlp_setB_r50_wo_yolov8_e24.pth
[                                                  ] 0/6019, elapsed: 0s, ETA:/home/zhangyiqing/miniconda3/envs/openlanev2/lib         /python3.8/site-packages/torch/_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future versio         n of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for neg         ative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, ro         unding_mode='floor'). (Triggered internally at  /opt/conda/conda-bld/pytorch_1631630815121/work/aten/src/ATen/native/BinaryOps         .cpp:467.)
  return torch.floor_divide(self, other)
calculating distances:: 100%|███████████████| 6019/6019 [26:05<00:00,  3.85it/s]sed: 3823s, ETA:     0s
/home/zhangyiqing/miniconda3/envs/openlanev2/lib/python3.8/site-packages/scipy/interpolate/_interpolate.py:641: RuntimeWarning: divide by zero encountered in divide
  slope = (y_hi - y_lo) / (x_hi - x_lo)[:, None]
/home/zhangyiqing/miniconda3/envs/openlanev2/lib/python3.8/site-packages/scipy/interpolate/_interpolate.py:641: RuntimeWarning: invalid value encountered in divide
  slope = (y_hi - y_lo) / (x_hi - x_lo)[:, None]
{'DET_l': 0.25253198, 'DET_t': 0.6305152, 'TOP_ll': 0.20700755803156554, 'TOP_lt': 0.2030884636319523, 'OpenLane-V2 Score': 0.4471703500089702, 'F-Score for 3D Lane': 0.09578137547278025}

Look forward to your reply. Best regards

wudongming97 commented 5 months ago

Hi @Benson722,

  1. It is common to have some different DET_l scores when running at different times. For example, please refer to our log file, where we re-train our model on Dataset B.
  2. It is not wrong. The low F-Score is because we use all lane predictions (referring to 300 lane queries), which introduces more FPs.
  3. The visualization script can refer to the OpenlaneV2 repo.
Benson722 commented 5 months ago

Thank you for your answer. By the way, which version of openlanev2 do you use? My version is '1.1.0'.

(openlanev2) zhangyiqing@inin:~/mmdetection3d$ pip list
...
mmdet3d                   1.0.0rc6             /home/zhangyiqing/mmdetection3d
openlanev2                1.1.0                /home/zhangyiqing/OpenLane-V2
...
wudongming97 commented 5 months ago

My openlanev2 version is 0.1.0