Funky 3D box predictions

[FullMetalNicky]

We are fine-tuning the cubercnn_DLA34_FPN.pth model on our lab data. We seem to notice some unorthodox 3D boxes wheh visualizing the validation results.

The image size is (640, 480), and the calibration matrix k is

k = [[380.07025146484375, 0.0, 324.4729919433594], 
     [0.0, 379.66119384765625, 237.78517150878906], 
     [0.0, 0.0, 1.0]]

The predictions from the evaluation json are attached. The cabinet is category 9 for us. predictions.txt

Is this caused by the predicted vertices not being in the proper order for the visualization? Or do you think it's a training issue?

[gkioxari]

It's an issue with visualization!

I visualized the box coordinates for cabinet you provide in meshlab (code below)

corners3d = torch.tensor([[-2.1654415130615234, 0.5102249383926392, -0.04547452926635742], 
                          [-1.710561990737915, 0.5124485492706299, -0.11441373825073242], 
                          [-1.0022611618041992, 0.2310657501220703, 4.550054550170898], 
                          [-1.457140564918518, 0.2288421392440796, 4.618993759155273], 
                          [-2.1731417179107666, -1.3415031433105469, -0.15601015090942383], 
                          [-1.7182623147964478, -1.3392795324325562, -0.22494935989379883], 
                          [-1.0099613666534424, -1.6206623315811157, 4.439518928527832], 
                          [-1.4648408889770508, -1.6228859424591064, 4.508458137512207]])

_box_triangles = torch.tensor([[0, 1, 2], [0, 3, 2], [4, 5, 6], [4, 6, 7], [1, 5, 6], [1, 6, 2], [0, 4, 7], [0, 7, 3], [3, 2, 6], [3, 6, 7], [0, 1, 5], [0, 4, 5],])
box3d = Meshes(verts=[corners3d], faces=[_box_triangles])
IO().save_mesh(box3d, "/data/users/gkioxari/nicky.obj", include_textures=False)

The output looks great and matches the cabinet in the image!

@FullMetalNicky So what is likely happening is that the visualization code is doing something bizarre with the negative z values (if you see the box extend behind the image plane a bit). It's likely dividing with the negative z to project the coordinates and flips the (x,y) signs, which is why it appears on the other side of the image plane. Can you point us to the visualization code you are using?

[FullMetalNicky]

Thanks, great news! That's the office of our boss, so we better get it right :)

These images are produced during the evaluation phase when running the train_net.py script from this repo - we didn't change the code in any way. I suppose it's from "visualize_from_instances" function in cubercnn.vis?

[gkioxari]

Oh yeah! Gotta get the boss' office right! I will provide a fix for the visualization in the coming day :)