How can I provide "bbox" to run "inference.py" demo ?

[dedoogong]

Thank you very much for your art work.

I'm struggling to track an object like "mug" using your inference code with my webcam + depth sensor(like real sense). And I'm getting the initially estimated R/T value using DenseFusion. I got runtime error because of wrong next R/T refined value during tracker.init_estimation(...) call. I guess the cause of this error comes from the wrong inital bbox values which must have 8 x,y,z kpts(torch.size[8,3]).

Refering dateset_nocs.py and dataset/data_preprocess.py code(using model_scales/*.txt), I currently using the same pre-defined 3D model coordinates(bbox) of "mug" as one of NOCS models( model_scales/mug_brown_starbucks_norm.txt)

================================== coord=[55.0, 48.0, 38.0], bbox=[[coord[0], coord[1], coord[2]],
[coord[0], coord[1], -coord[2]],
[coord[0], -coord[1], coord[2]], [coord[0], -coord[1], -coord[2]], [-coord[0], coord[1], coord[2]], [-coord[0], coord[1], -coord[2]], [-coord[0], -coord[1], coord[2]], [-coord[0], -coord[1], -coord[2]]]

to call current_r, current_t = tracker.init_estimation(current_r, current_t, "bbox",..)

current_r,current_t comes from "DenseFusion's output"

and of course I modified cam_cx,cam_cy, cam_fx, cam_fy to fit to my sensor's intrinsic parameters.

Plus, I dont understand why there is a code like

target_tmp[:, 0] = -1.0 target_tmp[:, 1] = -1.0

it seems like the shape should be upside down and turn opposite horizontally, but why? I guess maybe the actual camera which is used to make NOCS dataset is set upside down...right?

Please give me any hint or example or guide for running your inference code. Thank you very much.

[Tchaikovic]

Hi @dedoogong , have you managed to solve this issue? If so, could you share your solution? I'm trying to do the same thing.

[dedoogong]

the solution was to connect Densefusion or similar 6-D pose estimators' output(R,T, box) to the tracker. it was not easy, though. and the real time RGBD result was not good. so I decided not to use this project and instead I developed a much better 6D pose tracker on my own.