Error in preprocessing CO3D dataset

[FantasticOven2]

Hello author:

Thank you for this amazing work! I ran the CO3D data preprocessing script and got this error It seems like the frame.crop_bbox_xywh attribute is None

[szymanowiczs]

Hi! That's unexpected, I tried reproducing this error but was not able to. On my side frame does have the crop_bbox_xywh attribute. Could you check if this is an issue with this attribute in particular or data in general by checking if your frame has some other attributes (e.g. image_rgb or camera)?

[FantasticOven2]

I printed all attributes in the frame object, here's the screen shot:

Seems like other attributes are here? I used the CO3D_v2 teddybear dataset.

[mengxuyiGit]

same problem from my side: no crop_bbox_xywh for all sequences

[szymanowiczs]

Hi @mengxuyiGit @FantasticOven2 thanks for highlighting this. I double checked my data and it looks the same except crop_bbox_xywh is not None: see attached screenshot. I'll try to dig in why this might be an issue.

[jwubz123]

I also have the same problem from my side: no crop_bbox_xywh for all sequences. The dataset I am using is CO3Dv2. I also tried CO3D you recommand but no 'set_list' folder in the dataset while pytorch3d.implicitron.dataset.json_index_dataset_map_provider_v2 requires this folder.

[apchenstu]

very great work, thanks! I have the same issue when processing the data, would it be possible to share your annotation file with us?

[szymanowiczs]

Hi @apchenstu , @mengxuyiGit @jwubz123 @FantasticOven2 - I updated the download instructions in Readme. I also tested them myself just now. As far as I can see, crop_bbox_xywh is available in all frames. Could you re-download the dataset with instructions from the Readme and check this works on your side too?

[SJTUwxz]

Hi! I am having the same issue where crop_bbox_xywh=None, I downloaded and preprocessed the CO3D dataset with the updated README.

[szymanowiczs]

Hi @SJTUwxz OK, thanks for letting me know. Since it runs on my side I will need some information to reproduce your issue. Can you tell me the exact steps you ran, and can you run the following snippet:

from pytorch3d.implicitron.dataset.json_index_dataset_map_provider_v2 import JsonIndexDatasetMapProviderV2
from pytorch3d.implicitron.tools.config import expand_args_fields
from omegaconf import DictConfig

subset_name = "fewview_dev"

expand_args_fields(JsonIndexDatasetMapProviderV2)
dataset_map = JsonIndexDatasetMapProviderV2(
      category="hydrant",
      subset_name=subset_name,
      test_on_train=False,
      only_test_set=False,
      load_eval_batches=True,
      dataset_root=DOWNLOAD_FOLDER,
      dataset_JsonIndexDataset_args=DictConfig(
          {"remove_empty_masks": False, "load_point_clouds": True}
      ),
  ).get_dataset_map()["train"]

And check that files in dataset_map, for example dataset_map[0] has crop_bbox_xywh?

[apchenstu]

Hi @SJTUwxz OK, thanks for letting me know. Since it runs on my side I will need some information to reproduce your issue. Can you tell me the exact steps you ran, and can you run the following snippet:

from pytorch3d.implicitron.dataset.json_index_dataset_map_provider_v2 import JsonIndexDatasetMapProviderV2
from pytorch3d.implicitron.tools.config import expand_args_fields
from omegaconf import DictConfig

subset_name = "fewview_dev"

expand_args_fields(JsonIndexDatasetMapProviderV2)
dataset_map = JsonIndexDatasetMapProviderV2(
      category="hydrant",
      subset_name=subset_name,
      test_on_train=False,
      only_test_set=False,
      load_eval_batches=True,
      dataset_root=DOWNLOAD_FOLDER,
      dataset_JsonIndexDataset_args=DictConfig(
          {"remove_empty_masks": False, "load_point_clouds": True}
      ),
  ).get_dataset_map()["train"]

And check that files in dataset_map, for example dataset_map[0] has crop_bbox_xywh?

Thanks for the update, I still have the same issue after using the new script. dataset_map[0]['crop_bbox_xywh'] is None on my side.

[SJTUwxz]

Hi @SJTUwxz OK, thanks for letting me know. Since it runs on my side I will need some information to reproduce your issue. Can you tell me the exact steps you ran, and can you run the following snippet:

from pytorch3d.implicitron.dataset.json_index_dataset_map_provider_v2 import JsonIndexDatasetMapProviderV2
from pytorch3d.implicitron.tools.config import expand_args_fields
from omegaconf import DictConfig

subset_name = "fewview_dev"

expand_args_fields(JsonIndexDatasetMapProviderV2)
dataset_map = JsonIndexDatasetMapProviderV2(
      category="hydrant",
      subset_name=subset_name,
      test_on_train=False,
      only_test_set=False,
      load_eval_batches=True,
      dataset_root=DOWNLOAD_FOLDER,
      dataset_JsonIndexDataset_args=DictConfig(
          {"remove_empty_masks": False, "load_point_clouds": True}
      ),
  ).get_dataset_map()["train"]

And check that files in dataset_map, for example dataset_map[0] has crop_bbox_xywh?

Thanks for the update, I still have the same issue after using the new script. dataset_map[0]['crop_bbox_xywh'] is None on my side.

Thanks for the quick response! @szymanowiczs I also have the same issue with the new script, and the output dataset_map["train"][0] has crop_bbox_xywh=None:

As instructed by Readme, I ran the following commands to download the co3d dataset: git clone https://github.com/facebookresearch/co3d.git cd co3d mkdir DOWNLOAD_FOLDER python ./co3d/download_dataset.py --download_folder DOWNLOAD_FOLDER --download_categories hydrant,teddybear and I created a proprocess folder under DOWNLOAD_FOLDER to store the preprocessed data, and set that as CO3D_DATASET_ROOT in proprocess_co3d.py.

[szymanowiczs]

What are your pytorch3d versions? I'm using pytorch3d 0.7.2

[SJTUwxz]

Mine is 0.7.5. I installed the pytorch3d library from a local clone of their repo.

[szymanowiczs]

It's a Pytorch3d version issue. The data is likely identical but pytorch3d code that builds FrameData is different. You have 2 possible solutions:

1. Install Pytorch3d 0.7.2 from a prebuilt binary that matches your PyTorch and Cuda version with pip. Here's a link to the binaries https://anaconda.org/pytorch3d/pytorch3d/files
2. (NOT VERIFIED) call these functions to retrieve crop_bbox_xywh from bbox_xywh present in your annotation https://github.com/facebookresearch/pytorch3d/blob/d0d9cae9cd76dbbb1b3e1da673ae9043277a3d9e/pytorch3d/implicitron/dataset/frame_data.py#L267C1-L276C60

I haven't tested 2., and cannot guarantee crop_bbox_xywh will match the one I used to preprocess data, but you can check they match the screenshots I posted earlier.

[SJTUwxz]

Hello! Sorry for late reply! I tried the second solution first, and add the following lines to L106 of the preprocess_co3d.py:

bbox_xyxy = get_clamp_bbox(
frame.bbox_xywh,
image_path=frame.image_path,    # pyre-ignore
box_crop_context=0.3,)
clamp_bbox_xyxy = clamp_box_to_image_bounds_and_round(bbox_xyxy, image_size_hw=tuple(frame.effective_image_size_hw), )
crop_bbox_xywh = bbox_xyxy_to_xywh(clamp_bbox_xyxy)
x0, y0, box_w, box_h = crop_bbox_xywh

The generated crop_bbox_xywh of the same frame (frame_timestamp=0.0893) as you mentioned earlier is tensor([189, 240, 353, 560]), where the height is different from the one you got (560 vs 592). Preprocessing also stopped later after processing several frames with this error:

You are right that the second one might generate crop_bbox_xywh that is inconsistent with yours. I will try the first solution instead which will save some time to debug :) I am working on replicating the results on the SRN-cars/chairs dataset now but I will get back later to update if solution 1 works!

Thank you!

[Owen-Liuyuxuan]

I get the same results by directly pulling and pulling pytorch3d. It also shows the same error mentioned by @SJTUwxz

Then I wget the version 0.7.2.0 and build locally.

wget https://github.com/facebookresearch/pytorch3d/archive/refs/tags/v0.7.2.zip
unzip v0.7.2.zip
cd pytorch3d-0.7.2
pip3 install .

then it seems to work fine this step.

[szymanowiczs]

Thanks for reporting that wget 0.7.2 and building locally works. Readme contents are now updated to indicate that this is the required pytorch3d version, so I'm closing this issue. Feel free to reopen if you have further issues.

[XunshanMan]

Hi, thanks for the discussion!

I am struggling to install pytorch3d 0.7.2 on my computer. Thus, I configure a docker image here in case someone needs it:

docker pull uerrob/pytorch3d:0.7.2