search

PRBonn / semantic-kitti-api

SemanticKITTI API for visualizing dataset, processing data, and evaluating results.
http://semantic-kitti.org
MIT License
783 stars 187 forks source link

How to evaluate camera-lidar fusion method on the test set? #68

Closed ICEORY closed 1 year ago

ICEORY commented 3 years ago

Hi,

I'm working on camera-lidar fusion methods for semantic segmentation. Because SemanticKITTI only contains front-view images, I can only generate the segmentation results of front-view point clouds. I want to know whether it is available to evaluate the front-view point clouds on the test set.

Most of the exiting fusion-based methods only provide results on the processed KITTI data set of SqueezeSeg (Front-view). I think it will be better if SemanticKITTI can also provide a leaderboard for fusion-based methods.

Thanks!

jbehley commented 3 years ago

thanks for the suggestion. we could provide such metric only for the frontal part of the point cloud as an additional "branch". I will think about it...I currently cannot judge if people really are interested in this particular metric.

ICEORY commented 3 years ago

Thanks for your reply!

I think this metric is important for evaluating the performance of fusion-based and single-sensor methods. I also notice that other data sets released recently (e.g. nuscenes and kitti-360) have provided multi-camera data for point clouds. Maybe I should try these data sets first.

jbehley commented 3 years ago

Good idea. I leave this issue for now open and will consider it when I have more time to checkout if we can provide such "subevaluation" on the codalab platform as separate metric.