rohitgirdhar / CATER

CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning
https://rohitgirdhar.github.io/CATER/
Apache License 2.0
103 stars 19 forks source link

Error in gt? Label 'spl' does not refer to the snitch. #13

Closed sjtuytc closed 4 years ago

sjtuytc commented 4 years ago

Hi, when I'm visualizing the results of the tracking method (e.g. turn on DEBUG_STORE_VIDEO), I found that the gt location of spl does not place the snitch, but the ball. Btw, I can reproduce the number 33.9 exactly. You can see in the following figure and videos, the gt (in blue rectangle) refers to the yellow ball, but not the snitch. I check the gt number is the same as shown in the visualization.

cater_debug_tracking_videos.zip

sjtuytc commented 4 years ago
截屏2020-07-22 下午5 37 33

What causes this problem? I doubt that the gt label 'spl' does not correctly indicate the snitch in this case.

rohitgirdhar commented 4 years ago

Sorry, I'm a bit confused. Why do you say it's not over the snitch? It looks to me that the box is over the snitch in these images. Just for reference, this is the snitch object: 3 golden toruses intertwined (different from a yellow ball) image

sjtuytc commented 4 years ago

Sorry, I make it wrong because in Figure 2 of your paper, the snitch is contained by the tone so I took the tone as the snitch.

ysl208 commented 3 years ago

Hi, How do i get the same 33.9? I got the following result when running python main.py

Tracking baseline accuracies: {'top_1': 0.03755299818291944, 'top_5': 0.15626892792247124, 'l1_dist': 3.509993943064809}
Tracking baseline accuracy (GT end loc, this should be 1.0): {'top_1': 0.9993943064809206, 'top_5': 0.9993943064809206, 'l1_dist': 0.0036341611144760752}

Also if I understood it correctly, the blue box is the gt for the end position of the snitch and the yellow box is the tracking performance of the DaSiamRPN model?

sjtuytc commented 3 years ago

I did get the results of 33.9.