search

ldkong1205 / LaserMix

[CVPR 2023 Highlight] LaserMix for Semi-Supervised LiDAR Semantic Segmentation
https://ldkong.com/LaserMix
Apache License 2.0
272 stars 17 forks source link

The evaluation metrics #11

Closed RyuSangwook closed 9 months ago

RyuSangwook commented 9 months ago

Dear author,

I'm wondering how you evaluate the model. In detail, do you extract the voxel/range prediction based on the voxel/range coordinate and compare them in point-wise level or you just simply compare them with voxel/range labels directly.

Regards,

ldkong1205 commented 9 months ago

Hi @RyuSangwook, thanks for asking!

For your question: The original reporting in the CVPR paper adopts voxel/range predictions directly. In our new implementation on MMDetection3D, everything goes to the point level, that is, the predictions of both voxel and range view have been re-mapped to the point-level ones. We suggest you stick with the new implementation since it is more consistent.

Let me know if you have further questions!

RyuSangwook commented 9 months ago

Hi @ldkong1205

Do you use the validation code in this previous commit version? https://github.com/ldkong1205/LaserMix/blob/ldkong1205-patch-1/script/trainer/validator.py

You torshsparse lib has already hooked the point-wise label in you model, here: https://github.com/ldkong1205/LaserMix/blob/7d57b1aa92760a94ff4bbc5c7a77a7d557cf9599/model/cylinder/cylinder3d/network.py#L529.

I'm confused for how you adopts voxel/range predictions directly, can you be more specific?

Regards

ldkong1205 commented 9 months ago

Hi @RyuSangwook, the previous implementation is stale and we do not include it here. Please adjust the configuration based on the current implementation. Thanks!

RyuSangwook commented 9 months ago

Ok, so can you demonstrate the difference between the "stale" version and the current version, as the different evaluation will definitely lead to inconsistent results.

Most importantly, I can't find anything about range in the lasermix configuration, as it is only for cylinder3d https://github.com/ldkong1205/LaserMix/blob/main/configs/lasermix/lasermix_cy3d_semi_semantickitti_1.py.

ldkong1205 commented 9 months ago

Ok, so can you demonstrate the difference between the "stale" version and the current version, as the different evaluation will definitely lead to inconsistent results.

Most importantly, I can't find anything about range in the lasermix configuration, as it is only for cylinder3d https://github.com/ldkong1205/LaserMix/blob/main/configs/lasermix/lasermix_cy3d_semi_semantickitti_1.py.

Hi @RyuSangwook, sure! The "stale" version is built based on the Cylinder3D codebase, while the current version is rebuilt based on MMDetection3D. There are notable differences between these two implementations, for example, the current one uses spconv v2 instead of the v1 in Cylinder3D. The different evaluations indeed led to inconsistent results but, luckily, our new implementation achieved even better results than the one reported in the paper. Therefore, I suggest you follow our MMDetection3D implementation and report at the point level instead of voxel or range view.

For your second question: the config file you linked is for the voxel version using Cylinder3D. The range view version is based on CENet. We will support this (and many other backbones) in our next update. Please stay tuned!

RyuSangwook commented 9 months ago

Thanks for your comments.