evaluate_angle results

[deBrian07]

Hello, I trained the model the slfm_pretext and then slfm_geometric. However, when I evaluate the geometric model, the graph shows that only from 200-1000 are accurately predicted. However, from 0-200 and 1000-1200, are every inaccurately predicted. I was wondering if this is something wrong with the evaluate_angle code. Please look at the attachment

[IFICL]

Hi, Thank you for your interest in our work. As I can tell from the plot, you have successfully trained the model. Because our work are not able to handle front back confusion, we only evaluate the samples with angle range from (-90,90) which is demonstrated in the paper. Our evaluation code will map any prediction outside of (-90,90) back to the range.

[deBrian07]

Got it, but is there any way to avoid those predicted values in the mean and median calculation? Since it shows that the mean is very high.

Additionally, my vision model also seems to be a little inaccurate. Do you have any suggestions on this? Thank you so much!

[IFICL]

As I mentioned above, in our paper and released code, we evaluated samples that have angles within (-90,90). Specifically, in the test set, we filter out the samples that have angles outside of (-90,90) and create a new split for evaluation, i.e., https://github.com/IFICL/SLfM/tree/master/Dataset/AI-Habitat/data-split/hm3d-4view-rotation-filterangle.

You can create this split by

python create-csv.py --dataset='hm3d-4view-rotation' --type='hm3d-4view-rotation-filterangle' --data_split='9:1:1'  --filter_angle

[IFICL]

For the vision model, first of all, I noticed one thing is strange the vision model makes predictions greater than 90 degrees, which doesn't match with my given training config --finer_rotation which contains the rotation prediction within (-90, 90). The code is https://github.com/IFICL/SLfM/blob/master/slfm/models/slfm_geo_net.py#L326 .

One possible reason it happens is that you don't enable --finer_rotation when you run the evaluation code, leading to inaccurate results because of different outputs. The code is https://github.com/IFICL/SLfM/blob/master/slfm/evaluation/evaluate_angle.py#L79

Because we generate the two rotated views with an FOV of 60 degrees, we constrain rotation within 90 degrees to ensure there will be visual signals to pick up

[deBrian07]

Got it, I'll try --finer_rotation, thank you so much!