search

google-research-datasets / Objectron

Objectron is a dataset of short, object-centric video clips. In addition, the videos also contain AR session metadata including camera poses, sparse point-clouds and planes. In each video, the camera moves around and above the object and captures it from different views. Each object is annotated with a 3D bounding box. The 3D bounding box describes the object’s position, orientation, and dimensions. The dataset contains about 15K annotated video clips and 4M annotated images in the following categories: bikes, books, bottles, cameras, cereal boxes, chairs, cups, laptops, and shoes
Other
2.24k stars 263 forks source link

Visibility calculation #37

Closed hlaks-adl closed 3 years ago

hlaks-adl commented 3 years ago

In eval.py, each GT instance is checked against a visibility threshold. https://github.com/google-research-datasets/Objectron/blob/master/objectron/dataset/eval.py#L132 How to compute the visibility value per GT instance?

I'm using pytorch, and my system does not support xla. I'm not able to read the TFRecord that contains visibility information, so I'm starting from raw data.

ahmadyan commented 3 years ago

In this dataset, that bit is set to 1.0, you can simply ignore it for now.

hlaks-adl commented 3 years ago

Thanks for your response. I tried opening TFRecord data in Colab and observed that the visibility value is mostly 1.0, but for some instances it is 0. Therefore, to reproduce the evaluation setup of the Objectron arxiv paper, it is required to know which instances to skip accumulating errors. Is my understanding correct? Is the visibility information stored in the raw protobufs?

ahmadyan commented 3 years ago

If the object is not visible in this frame (e.g. it might be out of image) it will be set to 0. 

def check_object_visibility(keypoints: np.ndarray) -> float:
  """Check if object is visible in the image."""
  visibility = 1.0
  # Check if object center is inside image.
  cx, cy, _ = keypoints[0]
  if not (0 < cx < 1 and 0 < cy < 1):
    visibility = 0.0
  # Check if all keypoints are not too far away from image border.
  if any(not (-0.5 < x < 1.5 and -0.5 < y < 1.5) for x, y, _ in keypoints[1:]):
    visibility = 0.0

  return visibility
hlaks-adl commented 3 years ago

Thank you for the info & the code snippet!