search

shanice-l / gdrnpp_bop2022

PyTorch Implementation of GDRNPP, winner (most of the awards) of the BOP Challenge 2022 at ECCV'22
Apache License 2.0
242 stars 51 forks source link

problems in lmo visualization script #105

Closed monajalal closed 9 months ago

monajalal commented 10 months ago

The inference creates a csv (not a pkl) however the visualization script looks for a pkl. Also, the dataset name was wrong. I did the following small changes however, still no visualization, could you please help with drawing projected 3D cuboid/bbox around the objects for LMO dataset?

(gdrnpp) mona@ada:~/gdrnpp_bop2022$ python core/gdrn_modeling/tools/lmo/lmo_3_vis_poses_full.py
/home/mona/.local/lib/python3.10/site-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
  warnings.warn(
/home/mona/gdrnpp_bop2022/core/gdrn_modeling/tools/lmo/../../../../lib/pysixd/misc.py:586: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def get_obj_im_c(K, t):
/home/mona/gdrnpp_bop2022/core/gdrn_modeling/tools/lmo/../../../../lib/pysixd/misc.py:765: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def compute_2d_bbox_xyxy_from_pose(points, pose, K, width=640, height=480, clip=False):
/home/mona/gdrnpp_bop2022/core/gdrn_modeling/tools/lmo/../../../../lib/pysixd/misc.py:793: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def compute_2d_bbox_xyxy_from_pose_v2(points, pose, K, width=640, height=480, clip=False):
/home/mona/gdrnpp_bop2022/core/gdrn_modeling/tools/lmo/../../../../lib/pysixd/misc.py:822: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def compute_2d_bbox_xywh_from_pose(points, pose, K, width=640, height=480, clip=False):
Creating GL context for Cuda device 0
;(egl_renderer) Loaded EGL version: 1.5.
  0%|                                                                                                                              | 0/8 [00:00<?, ?it/s][0103_162829@egl_renderer_v3.py:558] datasets/BOP_DATASETS/lmo/models/obj_000001.ply
[0103_162829@meshutil.py:403] datasets/BOP_DATASETS/lmo/models/obj_000001.ply loaded cache file: .cache/5a69126f0e708423031ae00695ad654a_load_mesh_pysixd.pkl
[0103_162829@egl_renderer_v3.py:567] is_textured: False | is_cad: False | is_materialed: False
[0103_162829@egl_renderer_v3.py:570] ['vertices', 'normals', 'colors', 'faces', 'texturecoords', 'is_cad', 'uMatDiffuse', 'uMatSpecular', 'uMatAmbient', 'uMatShininess']
[0103_162829@egl_renderer_v3.py:585] colors: 0.6901960968971252
[0103_162829@egl_renderer_v3.py:558] datasets/BOP_DATASETS/lmo/models/obj_000005.ply
[0103_162829@meshutil.py:403] datasets/BOP_DATASETS/lmo/models/obj_000005.ply loaded cache file: .cache/2a63b9c7db7d4cb071cdae2803cc9785_load_mesh_pysixd.pkl
[0103_162829@egl_renderer_v3.py:567] is_textured: False | is_cad: False | is_materialed: False
[0103_162829@egl_renderer_v3.py:570] ['vertices', 'normals', 'colors', 'faces', 'texturecoords', 'is_cad', 'uMatDiffuse', 'uMatSpecular', 'uMatAmbient', 'uMatShininess']
[0103_162829@egl_renderer_v3.py:585] colors: 0.9137254953384399
[0103_162829@egl_renderer_v3.py:558] datasets/BOP_DATASETS/lmo/models/obj_000006.ply
[0103_162829@meshutil.py:403] datasets/BOP_DATASETS/lmo/models/obj_000006.ply loaded cache file: .cache/c22328d3bc0a315a6c30d71ec5721697_load_mesh_pysixd.pkl
[0103_162829@egl_renderer_v3.py:567] is_textured: False | is_cad: False | is_materialed: False
[0103_162829@egl_renderer_v3.py:570] ['vertices', 'normals', 'colors', 'faces', 'texturecoords', 'is_cad', 'uMatDiffuse', 'uMatSpecular', 'uMatAmbient', 'uMatShininess']
[0103_162829@egl_renderer_v3.py:585] colors: 0.9098039269447327
[0103_162829@egl_renderer_v3.py:558] datasets/BOP_DATASETS/lmo/models/obj_000008.ply
[0103_162829@meshutil.py:403] datasets/BOP_DATASETS/lmo/models/obj_000008.ply loaded cache file: .cache/b87ccbce6c3f76097ebc6e36e725b843_load_mesh_pysixd.pkl
[0103_162829@egl_renderer_v3.py:567] is_textured: False | is_cad: False | is_materialed: False
[0103_162829@egl_renderer_v3.py:570] ['vertices', 'normals', 'colors', 'faces', 'texturecoords', 'is_cad', 'uMatDiffuse', 'uMatSpecular', 'uMatAmbient', 'uMatShininess']
[0103_162829@egl_renderer_v3.py:585] colors: 0.7843137383460999
[0103_162829@egl_renderer_v3.py:558] datasets/BOP_DATASETS/lmo/models/obj_000009.ply
[0103_162829@meshutil.py:403] datasets/BOP_DATASETS/lmo/models/obj_000009.ply loaded cache file: .cache/60dbff1e143e852cd74727cfdd8ee3c5_load_mesh_pysixd.pkl
[0103_162829@egl_renderer_v3.py:567] is_textured: False | is_cad: False | is_materialed: False
[0103_162829@egl_renderer_v3.py:570] ['vertices', 'normals', 'colors', 'faces', 'texturecoords', 'is_cad', 'uMatDiffuse', 'uMatSpecular', 'uMatAmbient', 'uMatShininess']
[0103_162829@egl_renderer_v3.py:585] colors: 0.9529411792755127
[0103_162829@egl_renderer_v3.py:558] datasets/BOP_DATASETS/lmo/models/obj_000010.ply
[0103_162829@meshutil.py:403] datasets/BOP_DATASETS/lmo/models/obj_000010.ply loaded cache file: .cache/64a98ada262b40dba88884049926d85d_load_mesh_pysixd.pkl
[0103_162829@egl_renderer_v3.py:567] is_textured: False | is_cad: False | is_materialed: False
[0103_162829@egl_renderer_v3.py:570] ['vertices', 'normals', 'colors', 'faces', 'texturecoords', 'is_cad', 'uMatDiffuse', 'uMatSpecular', 'uMatAmbient', 'uMatShininess']
[0103_162829@egl_renderer_v3.py:585] colors: 0.9372549057006836
[0103_162829@egl_renderer_v3.py:558] datasets/BOP_DATASETS/lmo/models/obj_000011.ply
[0103_162829@meshutil.py:403] datasets/BOP_DATASETS/lmo/models/obj_000011.ply loaded cache file: .cache/0c31b6adc192f9379f49e053fcce6868_load_mesh_pysixd.pkl
[0103_162829@egl_renderer_v3.py:567] is_textured: False | is_cad: False | is_materialed: False
[0103_162829@egl_renderer_v3.py:570] ['vertices', 'normals', 'colors', 'faces', 'texturecoords', 'is_cad', 'uMatDiffuse', 'uMatSpecular', 'uMatAmbient', 'uMatShininess']
[0103_162829@egl_renderer_v3.py:585] colors: 0.800000011920929
[0103_162829@egl_renderer_v3.py:558] datasets/BOP_DATASETS/lmo/models/obj_000012.ply
[0103_162829@meshutil.py:403] datasets/BOP_DATASETS/lmo/models/obj_000012.ply loaded cache file: .cache/55aff959da94ec771030b987311bfce2_load_mesh_pysixd.pkl
[0103_162829@egl_renderer_v3.py:567] is_textured: False | is_cad: False | is_materialed: False
[0103_162829@egl_renderer_v3.py:570] ['vertices', 'normals', 'colors', 'faces', 'texturecoords', 'is_cad', 'uMatDiffuse', 'uMatSpecular', 'uMatAmbient', 'uMatShininess']
[0103_162829@egl_renderer_v3.py:585] colors: 0.4431372582912445
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 233.52it/s]
./output/gdrn/lmo_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_lmo/inference_model_final_wo_optim/lmo_bop_test/convnext-a6-AugCosyAAEGray-BG05-mlL1-DMask-amodalClipBox-classAware-lmo-test-iter0_lmo-test.csv
./output/gdrn/lmo_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_lmo/inference_model_final_wo_optim/lmo_bop_test/convnext-a6-AugCosyAAEGray-BG05-mlL1-DMask-amodalClipBox-classAware-lmo-test-iter0_lmo-test.csv
lmo_bop_test
[0104_052829@lmo_bop_test:377] DBG register dataset: lmo_bop_test
MetadataCatalog:  Metadata(name='lmo_bop_test', id='lmo', ref_key='lmo_full', objs=['ape', 'can', 'cat', 'driller', 'duck', 'eggbox', 'glue', 'holepuncher'], eval_error_types=['ad', 'rete', 'proj'], evaluator_type='bop', thing_classes=['ape', 'can', 'cat', 'driller', 'duck', 'eggbox', 'glue', 'holepuncher'], sym_infos={0: None, 1: None, 2: None, 3: None, 4: None, 5: array([[[ 1.        ,  0.        ,  0.        ],
        [ 0.        ,  1.        ,  0.        ],
        [ 0.        ,  0.        ,  1.        ]],

       [[-0.999964  , -0.00333777, -0.0077452 ],
        [ 0.00321462, -0.999869  ,  0.0158593 ],
        [-0.00779712,  0.0158338 ,  0.999844  ]]], dtype=float32), 6: array([[[ 1.00000e+00,  0.00000e+00,  0.00000e+00],
        [ 0.00000e+00,  1.00000e+00,  0.00000e+00],
        [ 0.00000e+00,  0.00000e+00,  1.00000e+00]],

       [[-9.99633e-01,  2.66790e-02,  4.79336e-03],
        [-2.66744e-02, -9.99644e-01,  1.00504e-03],
        [ 4.81847e-03,  8.76815e-04,  9.99988e-01]]], dtype=float32), 7: None})
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [00:07<00:00, 27.73it/s]

Slightly modified code for core/gdrn_modeling/tools/lmo/lmo_3_vis_poses_full.py:

import mmcv
import os.path as osp
import numpy as np
import sys
from tqdm import tqdm
from detectron2.data import DatasetCatalog, MetadataCatalog
from detectron2.structures import BoxMode
import torch
import pandas as pd

cur_dir = osp.dirname(osp.abspath(__file__))
sys.path.insert(0, osp.join(cur_dir, "../../../../"))

from lib.vis_utils.colormap import colormap
from lib.utils.mask_utils import cocosegm2mask, get_edge
from core.utils.data_utils import read_image_mmcv
from core.gdrn_modeling.datasets.dataset_factory import register_datasets
from transforms3d.quaternions import quat2mat
from lib.egl_renderer.egl_renderer_v3 import EGLRenderer
from lib.utils.mask_utils import mask2bbox_xyxy, cocosegm2mask, get_edge

def load_predicted_csv(fname):
    df = pd.read_csv(fname)
    info_list = df.to_dict("records")
    return info_list

def parse_Rt_in_csv(_item):
    return np.array([float(i) for i in _item.strip(" ").split(" ")])

score_thr = 0.3
colors = colormap(rgb=False, maximum=255)

id2obj = {
    1: "ape",
    #  2: 'benchvise',
    #  3: 'bowl',
    #  4: 'camera',
    5: "can",
    6: "cat",
    #  7: 'cup',
    8: "driller",
    9: "duck",
    10: "eggbox",
    11: "glue",
    12: "holepuncher",
    #  13: 'iron',
    #  14: 'lamp',
    #  15: 'phone'
}
objects = list(id2obj.values())

width = 640
height = 480
tensor_kwargs = {"device": torch.device("cuda"), "dtype": torch.float32}
image_tensor = torch.empty((height, width, 4), **tensor_kwargs).detach()
seg_tensor = torch.empty((height, width, 4), **tensor_kwargs).detach()
# image_tensor = torch.empty((480, 640, 4), **tensor_kwargs).detach()

model_dir = "datasets/BOP_DATASETS/lmo/models/"

model_paths = [osp.join(model_dir, f"obj_{obj_id:06d}.ply") for obj_id in id2obj]

ren = EGLRenderer(model_paths, vertex_scale=0.001, use_cache=True, width=width, height=height)

# NOTE:
#pred_path = "output/gdrn/lmo/a6_cPnP_AugAAETrunc_BG0.5_lmo_real_pbr0.1_40e/inference_model_final/lmo_test/a6-cPnP-AugAAETrunc-BG0.5-lmo-real-pbr0.1-40e-test_lmo_test_preds.pkl"
pred_path = "./output/gdrn/lmo_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_lmo/inference_model_final_wo_optim/lmo_bop_test/convnext-a6-AugCosyAAEGray-BG05-mlL1-DMask-amodalClipBox-classAware-lmo-test-iter0_lmo-test.csv"
vis_dir = (
    "output/gdrn/lmo/a6_cPnP_AugAAETrunc_BG0.5_lmo_real_pbr0.1_40e/inference_model_final/lmo_test/lmo_vis_gt_pred_full"
)
mmcv.mkdir_or_exist(vis_dir)

print(pred_path)
preds = load_predicted_csv(pred_path)

print(pred_path)
# preds = mmcv.load(pred_path)

# dataset_name = "lmo_test"
dataset_name = "lmo_bop_test"

print(dataset_name)
register_datasets([dataset_name])

meta = MetadataCatalog.get(dataset_name)
print("MetadataCatalog: ", meta)
objs = meta.objs

dset_dicts = DatasetCatalog.get(dataset_name)
for d in tqdm(dset_dicts):
    K = d["cam"]
    file_name = d["file_name"]
    img = read_image_mmcv(file_name, format="BGR")

    scene_im_id_split = d["scene_im_id"].split("/")
    scene_id = scene_im_id_split[0]
    im_id = int(scene_im_id_split[1])

    imH, imW = img.shape[:2]
    annos = d["annotations"]
    masks = [cocosegm2mask(anno["segmentation"], imH, imW) for anno in annos]
    bboxes = [anno["bbox"] for anno in annos]
    bbox_modes = [anno["bbox_mode"] for anno in annos]
    bboxes_xyxy = np.array(
        [BoxMode.convert(box, box_mode, BoxMode.XYXY_ABS) for box, box_mode in zip(bboxes, bbox_modes)]
    )
    quats = [anno["quat"] for anno in annos]
    transes = [anno["trans"] for anno in annos]
    Rs = [quat2mat(quat) for quat in quats]
    # 0-based label
    cat_ids = [anno["category_id"] for anno in annos]

    obj_names = [objs[cat_id] for cat_id in cat_ids]

    est_Rs = []
    est_ts = []

    gt_Rs = []
    gt_ts = []

    labels = []

    for anno_i, anno in enumerate(annos):
        obj_name = obj_names[anno_i]

        try:
            R_est = preds[obj_name][file_name]["R"]
            t_est = preds[obj_name][file_name]["t"]
            score = preds[obj_name][file_name]["score"]
        except:
            continue
        if score < score_thr:
            continue

        labels.append(objects.index(obj_name))  # 0-based label

        est_Rs.append(R_est)
        est_ts.append(t_est)
        gt_Rs.append(Rs[anno_i])
        gt_ts.append(transes[anno_i])

    im_gray = mmcv.bgr2gray(img, keepdim=True)
    im_gray_3 = np.concatenate([im_gray, im_gray, im_gray], axis=2)

    gt_poses = [np.hstack([_R, _t.reshape(3, 1)]) for _R, _t in zip(gt_Rs, gt_ts)]
    poses = [np.hstack([_R, _t.reshape(3, 1)]) for _R, _t in zip(est_Rs, est_ts)]

    ren.render(labels, poses, K=K, image_tensor=image_tensor, background=im_gray_3)
    ren_bgr = (image_tensor[:, :, :3].detach().cpu().numpy() + 0.5).astype("uint8")

    for label, gt_pose, est_pose in zip(labels, gt_poses, poses):
        ren.render([label], [gt_pose], K=K, seg_tensor=seg_tensor)
        gt_mask = (seg_tensor[:, :, 0].detach().cpu().numpy() > 0).astype("uint8")

        ren.render([label], [est_pose], K=K, seg_tensor=seg_tensor)
        est_mask = (seg_tensor[:, :, 0].detach().cpu().numpy() > 0).astype("uint8")

        gt_edge = get_edge(gt_mask, bw=3, out_channel=1)
        est_edge = get_edge(est_mask, bw=3, out_channel=1)

        ren_bgr[gt_edge != 0] = np.array(mmcv.color_val("blue"))
        ren_bgr[est_edge != 0] = np.array(mmcv.color_val("green"))

    vis_im = ren_bgr

    save_path_0 = osp.join(vis_dir, "{}_{:06d}_vis0.png".format(scene_id, im_id))
    mmcv.imwrite(img, save_path_0)

    save_path = osp.join(vis_dir, "{}_{:06d}_vis1.png".format(scene_id, im_id))
    mmcv.imwrite(vis_im, save_path)

    # if True:
    #     # grid_show([img[:, :, ::-1], vis_im[:, :, ::-1]], ["im", "est"], row=1, col=2)
    #     # im_show = cv2.hconcat([img, vis_im, vis_im_add])
    #     im_show = cv2.hconcat([img, vis_im])
    #     cv2.imshow("im_est", im_show)
    #     if cv2.waitKey(0) == 27:
    #         break  # esc to quit

# ffmpeg -r 5 -f image2 -s 1920x1080 -pattern_type glob -i "./lmo_vis_gt_pred_full_video/*.png" -vcodec libx264 -crf 25  -pix_fmt yuv420p lmo_vis_video.mp4

Current output in /home/mona/gdrnpp_bop2022/output/gdrn/lmo/a6_cPnP_AugAAETrunc_BG0.5_lmo_real_pbr0.1_40e/inference_model_final/lmo_test/lmo_vis_gt_pred_full/:

Screenshot from 2024-01-03 16-35-14 Screenshot from 2024-01-03 16-35-34

AmanSriv97 commented 10 months ago

@monajalal I am also getting the same output (grey image). Have you figured out a way to get the .pkl file or how to plot bounding box via csv?

monajalal commented 10 months ago

@AmanSriv97 my understanding is that core/gdrn_modeling/tools/lmo/lmo_3_vis_poses_full.py is wrong since it should look for correct file format and is not doing so. I hope that @shanice-l could provide an updated and correct version of it for lmo visualization.

monajalal commented 9 months ago

@wangg12 do you have any fix for this?

monajalal commented 9 months ago

check https://github.com/shanice-l/gdrnpp_bop2022/issues/110#issuecomment-1904240958