How to create /saved_detections/ycbv_posecnn.pkl for own dataset

[LucZed]

Hello, I want to train and run cosypose on my own custom dataset. Till now following scripts work fine:

• run_detector_training.py
• run_detection_eval

Now i want to train the the "run_pose_training.py" script. For now, the whole script finish without exceptions. To come to the Problem: In the /localdata/experiments directory the last run showing up, but the "errors(ds_name).txt is empty. I traced my problem back to the function "train_pose.py "-> "make_eval_bundle". There i use "load_posecnn_results" to get detections from the "yxbv_posecnn.pkl".

• First: This file has like 15k gt-data for the YCBV-dataset
• Second: There is a mismatch between this file and my own dataset (kind of obvious)
• Third: For me it looks like, that it is necessary later in (train_pose.py --- run_eval) -> (runner_utils.py --- run_pred_eval) -> (multiview_predictions.py --- get_predictions), that both, my data and the .pkl has to match, so the detection list in line 120 wont be empty.

There are already closed issues like #65 or #57 but these got closed with no answer.

How do i create my own ycbv_posecnn.pkl, which match my dataset? Is there an other, easier way to get eval results?

If you need more details about my issue, i will provide more information

Thanks :)

[LucZed]

Hello again, I still were not able to create my own .pkl file which is like the "ycbv_posecnn.pkl" but i found a workaround. (For now i don't know what impact my workaround have in the eval results)

Preparation:

• Recreate the internal data format for the detection, which got read in from the .pkl file.
• Below i copied the "detection"-format from: "cosypose -> scripts -> run_cosypose_eval.py -> load_posecnn_results()"

  data = tc.PandasTensorCollection(
      infos=pd.DataFrame(infos),
      poses=torch.as_tensor(np.stack(poses)).float(),
      bboxes=torch.as_tensor(np.stack(bboxes)).float(),
  ).cpu()

• The final format should look like this:

  
  PandasTensorCollection(
  poses: torch.Size([15435, 4, 4]) torch.float32 cpu,
  bboxes: torch.Size([15435, 4]) torch.float32 cpu,
  ----------------------------------------
  infos:
     scene_id  view_id  score       label
  0            48        1    1.0  obj_000001
  1            48        1   20.0  obj_000020
  2            48        1   14.0  obj_000014
  3            48        1   19.0  obj_000019
  4            48        1    6.0  obj_000006
  ...         ...      ...    ...         ...
  15430        59     1857   15.0  obj_000015
  15431        59     1857    6.0  obj_000006
  15432        59     1857    9.0  obj_000009
  15433        59     1857    4.0  obj_000004
  15434        59     1857   18.0  obj_000018

[15435 rows x 4 columns] )

### Create detection from BOP's scene_gt_info / scene_gt
- This was just a test to know if my workaround would work:

def own_detection_from_gt(): import os import cosypose.utils.tensor_collection as tc import pandas as pd path_data_dir = "PATH/TO/REPO/cosypose/local_data/bop_datasets/DATASET_NAME/" path_scene_dir = os.path.join(path_data_dir, "train") scene_names = os.listdir(path_scene_dir) infos, poses, bboxes = [], [], [] for scene_id, scene_name in enumerate(scene_names): path_scene_gt_info = os.path.join(path_scene_dir, scene_name, "scene_gt_info.json") path_scene_gt = os.path.join(path_scene_dir, scene_name, "scene_gt.json") with open(path_scene_gt_info, "r") as f: json_data_gt_info = json.load(f) with open(path_scene_gt, "r") as f: json_data_gt = json.load(f) img_names_rgb = os.listdir(os.path.join(path_scene_dir, scene_name, "rgb")) for img_id, img_name in enumerate(img_names_rgb): list_bbox = json_data_gt_info[f"{img_id}"][0]["bbox_obj"] xmin = list_bbox[0] ymin = list_bbox[1] xmax = list_bbox[0] + list_bbox[2] ymax = list_bbox[1] + list_bbox[3] list_bbox = [xmin, ymin, xmax, ymax] list_rot = json_data_gt[f"{img_id}"][0]["cam_R_m2c"] list_loc = json_data_gt[f"{img_id}"][0]["cam_t_m2c"] row0 = [list_rot[0], list_rot[1], list_rot[2], list_loc[0]] row1 = [list_rot[3], list_rot[4], list_rot[5], list_loc[1]] row2 = [list_rot[6], list_rot[7], list_rot[8], list_loc[2]] row3 = [0, 0, 0, 1] rot_loc_mat = [row0, row1, row2, row3] infos.append(dict( scene_id=scene_id, view_id=img_id, score=1, label="obj_000001", )) poses.append(rot_loc_mat) bboxes.append(list_bbox) data = tc.PandasTensorCollection( infos=pd.DataFrame(infos), poses=torch.as_tensor(np.stack(poses)).float(), bboxes=torch.as_tensor(np.stack(bboxes)).float(), ).cpu() return data

- Since i have only one object, the label is always "obj_000001"
- Because of the GT-Data the score is always "1"

### Other Changes
Changes in "run_cosypose_eval.py":
- Line 160: Add something like this for your own Dataset:

elif 'OWN_DS_NAME' in ds_name: compute_add = True visib_gt_min = -1 targets_filename = None n_top = 1 spheres_overlap_check = False

- Line 218:  Add something like this for your own Dataset: (check "match_threshold")

if 'OWN_DS_NAME' in ds_name: meters[f'{error_type}_ntop=1_matching=CLASS'] = PoseErrorMeter( error_type=error_type, consider_all_predictions=False, match_threshold=np.inf, report_error_stats=False, report_error_AUC=True, **base_kwargs

- Line 300: Add something like this for your own Dataset:

elif 'OWN_DS_NAME' in args.config: object_set = 'OWN_DS_NAME' refiner_run_id = 'OWN_DS_NAME--12345' n_coarse_iterations = 0 n_refiner_iterations = 2

- Line 310: Add something like this for your own Dataset:

elif args.config == 'OWN_DS_NAME': ds_name = 'OWN_DATASET_NAME'

- Line 372: Add something like this for your own Dataset:

elif "OWN_DS_NAME" in ds_name: OWN_DS_detections = own_detection_from_gt() pred_kwargs = { 'OWN_DS_NAME_GT': dict( detections=OWN_DS_detections, use_detections_TCO=OWN_DS_detections, **base_pred_kwargs ), }

- Line 405: Add something like this for your own Dataset:

elif "OWN_DS_NAME" in ds_name: det_key = 'OWN_DS_NAME_GT' all_predictions['OWN_DS_NAME_GT'] = OWN_DS_NAME_gt predictions_to_evaluate.add('OWN_DS_NAME_GT')

- Line 462: Add something like this for your own Dataset:

elif 'OWN_DS_NAME' in ds_name: metrics_to_print.update({ f'posecnn/ADD(-S)_ntop=1_matching=CLASS/AUC/objects/mean': f'PoseCNN/AUC of ADD(-S)',

        f'{det_key}/refiner/iteration={n_refiner_iterations}/ADD(-S)_ntop=1_matching=CLASS/AUC/objects/mean': f'Singleview/AUC of ADD(-S)',
        f'{det_key}/refiner/iteration={n_refiner_iterations}/ADD-S_ntop=1_matching=CLASS/AUC/objects/mean': f'Singleview/AUC of ADD-S',

        f'{det_key}/ba_output+all_cand/ADD(-S)_ntop=1_matching=CLASS/AUC/objects/mean': f'Multiview (n={args.n_views})/AUC of ADD(-S)',
        f'{det_key}/ba_output+all_cand/ADD-S_ntop=1_matching=CLASS/AUC/objects/mean': f'Multiview (n={args.n_views})/AUC of ADD-S',
    })

### Additional Information 
- There are more checks for the DS_NAME. Just copy everything that is necessary form ycbv or tless.
- I just tried to make everything work. If i made any mistakes or my code makes no sense, feel free to give me a hint :)
- If you get NaN Results in the summary.txt / full_summary.txt, maybe the threshold is to strict or your training is not good enough.
- in the full_summary.txt you will find the section "OWN_DS_NAME/refiner/iteration=2" (or something similar) which should be the score for the checkpoint you chose to evaluate. (But I am not 100% sure)

[tensarflow]

Hi there, thanks for sharing your results! How did you create your dataset in the first place? This is the first time I come across PKL files and I have no clue :)

[LucZed]

I used:

• Blenderproc: https://github.com/DLR-RM/BlenderProc
• BopToolkit: https://github.com/thodan/bop_toolkit

I wrote my own rendering pipeline based on blenderproc and used my own 3D-Model (.obj). In blenderproc there is a method called bproc.writer.write_bop(), which save rendered data in bop format.

To get the necessary scene_gt_info.json for own Dataset (BOP-Format) i used BopToolkit. I used bop_toolkit/scripts/calc_gt_info.py. I made several changes to the code. But most of the time you just have to add "if"-statements which contain dataset specific variables.

if you dont have a .ply 3D-Model than you have to convert it to that format using Meshlab.

till now i dont know how to create that .pkl file. i still use my workarround. If you want to open and create Pickel (.pkl) Files. Use Pandas. pd.read_pickle()

if you need more information i try my best to give proper answers.

There could be better ways to do it. If you find one, let me know :)

[tensarflow]

Thanks a lot for the guidance! I will have a look at those and get back to you.