search

Jumpat / SegmentAnythingin3D

Segment Anything in 3D with NeRFs (NeurIPS 2023)
Apache License 2.0
831 stars 52 forks source link

Evaluation scripts #78

Open Zhao-Yian opened 3 weeks ago

Zhao-Yian commented 3 weeks ago

Can evaluation scripts be provided on different datasets to validate the quantitative results provided in the paper？

Jumpat commented 3 weeks ago

Hi, the evaluation script is incompatible with this GUI version, which has gone through code refactoring and abandoned many interface for evaluation.

The evaluation mainly involves loading different datasets like SPIN and NVOS, while the metric (IoU) calculation is relatively easy. We may re-write it in the future.

Zhao-Yian commented 3 weeks ago

Thanks!

Zhao-Yian commented 3 weeks ago

Sorry to ask again, which reference views and target views are used for evaluation on the SPIn-NeRF dataset? What was the basis for the selection?

Jumpat commented 3 weeks ago

Hi, the reference view is set to the first frame of the sorted views. However, the method is robust to reference view selection since the segmentation target is relatively simple.

Zhao-Yian commented 3 weeks ago

Thank you very much for your answer. May I ask if the target views used for evaluation are all views except the first frame? And the IoU of each scene is the average IoU of these target views?

Jumpat commented 3 weeks ago

No, the target views include the reference view generally. Since though the reference view has gt mask for reference, the segmentation cannot ensure the final result align with the initial 2D mask. It is still meaningful to check whether the reference view is segmented properly.

The IoU score is calculated across all views, not (IoU_1+...+IoU_N) / N. There is a little difference since (a/b+c/d)/2 != (a+c) / (b+d). We use the latter implementation.

Zhao-Yian commented 3 weeks ago

Thank you very much for your answer！