search

google-research-datasets / Objectron

Objectron is a dataset of short, object-centric video clips. In addition, the videos also contain AR session metadata including camera poses, sparse point-clouds and planes. In each video, the camera moves around and above the object and captures it from different views. Each object is annotated with a 3D bounding box. The 3D bounding box describes the object’s position, orientation, and dimensions. The dataset contains about 15K annotated video clips and 4M annotated images in the following categories: bikes, books, bottles, cameras, cereal boxes, chairs, cups, laptops, and shoes
Other
2.24k stars 263 forks source link

Question about the metric in the evaluation code #24

Closed Uio96 closed 3 years ago

Uio96 commented 3 years ago

Hi there,

I think there is a problem in your evaluation code that you do not count the case where there is no prediction in the given input. So the final number may not fully reflect the truth.

https://github.com/google-research-datasets/Objectron/blob/aa667e689848aa3619e087b493ddb3b919f9e0c8/objectron/dataset/eval.py#L124-L169

In your code snippet, the instance represents ground truth while the box represents prediction. You try to match each prediction with one ground truth. But if there is no prediction (which means no match), you just skip this case. I think you should instead record that case as missing targets (you should still add the num_instances but do not update tp & fp).

ahmadyan commented 3 years ago

As the comment mentions, we don't have negative samples (i.e. images without any instances in them) in the dataset (so num_instances is always positive). However you comment is valid. Feel free to update the code and create a PR, then I'll accept it. Otherwise I'll update the eval in the next release (which will be early Feb).

Uio96 commented 3 years ago

As the comment mentions, we don't have negative samples (i.e. images without any instances in them) in the dataset (so num_instances is always positive). However you comment is valid. Feel free to update the code and create a PR, then I'll accept it. Otherwise I'll update the eval in the next release (which will be early Feb).

Hi @ahmadyan, thanks for the reply.

I was wondering if the reported numbers in your paper were following the current evaluation code? I think it has some problems as I mentioned in my original question. Do you also plan to re-evaluate your proposed methods (MobilePose & two-stage) in the next release?

ahmadyan commented 3 years ago

definitely.

Uio96 commented 3 years ago

Thank you so much.