search

VIS4ROB-lab / voxfield

🦊 Voxfield: non-Projective Signed Distance Fields for Online Planning and 3D Reconstruction [IROS' 22]
BSD 3-Clause "New" or "Revised" License
274 stars 29 forks source link

ESDF Error Comparison #2

Closed yinloonga closed 1 year ago

yinloonga commented 2 years ago

I use cow_and_lady to run voxedt_server, voxblox_server and voxfield_server respectively, and print the Erroe-log through the VoxedtServer::evalEsdfRefOcc() function.

voxblox_server

No.147: 0.0599352 0.047562 370751
No.148: 0.0599352 0.047562 370751
No.149: 0.0599352 0.047562 370751
No.150: 0.0593335 0.0471059 374120

voxfiele_server

No.77: 0.0425458 0.0297242 313890
No.78: 0.0429067 0.0301379 317133
No.79: 0.0441347 0.0307387 336464
No.80: 0.0445938 0.0309601 339265

voxedt_server

No.37: 0.0173335 0.00275208 310175
No.38: 0.0120098 0.00155523 323364
No.39: 0.0118983 0.00152207 323668
No.40: 0.0118828 0.00151857 324090

The results are obvious. Compared with the results in the paper, the error of voxfield_server is indeed smaller than that of voxblox_server. However, the result of voxedt_server is actually smaller, and you mentioned that it is compared with the method based on the occupancy grid in FIESTA, so the code part should be fine. Because you refactored the EDT using voxblox data structure. So, can you help explain what's wrong here? Thank you.

YuePanEdward commented 1 year ago

Thanks a lot for your interest in our work. As mentioned https://github.com/VIS4ROB-lab/voxfield/issues/4, the ESDF error we reported in the paper is calculated by taking the ground truth 3D model as the reference. The evalEsdfRefOcc() function here is calculated based on the occupancy grid and does not need to load the ground truth 3D model for evaluation. The output of this function can be visualized on the fly but is not very reliable because the occupancy grid map it relied on also has some errors. In other words, It's only an indicator for debugging. To reproduce the ESDF error reported in the paper, one needs to run the evaluation code provided here after mapping in which the ground truth model is loaded as the reference.

Metric definition in the paper:

ESDF Error: For each voxel in the ESDF map, the ground-truth signed distance is calculated as the Euclidean distance from the voxel’s center to the nearest point in the ground-truth point cloud. The voxel’s ESDF error is then defined as the difference between the ground-truth value and the mapped ESDF value, and the overall ESDF error is computed as the average of all ESDF voxel errors. Compared with the ESDF error metric in FIESTA [4], which is defined with respect to the centers of the occupied voxels, the ESDF error metric used here is not affected by the grid discretization and is, therefore, more accurate.