Any idea on debugging the coord transformations?

[Nplace-su]

Hi all, I'm trying to train with my custom dataset. When implementing the custom data parser, I found there are several coordination transformations, including the camera pose and the bbox pose, as well as opencv data convention to the openGL data convention. Is there any efficient way to debug whether I made a correct coord transform? (Through the result I think I messed up with the bbox poses... because it seems everything is treated as background. But I didn't find an easy way to visualize the bbox pose and debug the transformation directly) Thank you for any suggestions!

[wuzirui]

Some suggestions:

1. Formulate the world space, and camera coordinate systems according to the nerfstudio convention (world: +z for up; camera: OpenGL right-up-backward convention);
2. Transform the object poses (xyz and yaw angle), and the camera poses from the original data to this coordinate system.

This whole process may be very very tricky (and painful :<) because there are just too many axes to consider. I would suggest visualizing all the axes. I have implemented a toy visualizer, but I am not quite sure whether it is still working now because it is a bit dated... Maybe I will work on a newer version with the nerfstudio team's new project viser.

[Francis777]

Formulate the world space, and camera coordinate systems according to the nerfstudio convention (world: +z for up; camera: OpenGL right-up-backward convention)

Hi @wuzirui! Regarding the world space, I initially also thought MARS follows the nerfstudio convention(+z for up), but it looks like it's actually using right-down-forward? One obvious example is when moving the object centre in the code below, y_shift here is [0, -height/2, 0], suggesting -y is up here: https://github.com/OPEN-AIR-SUN/mars/blob/5d7567504220f9a8a09aa20a199954f481a26484/mars/utils/neural_scene_graph_helper.py#L533

[wuzirui]

Yes exactly. Sorry for the misleading info. We are actually planning to change it to the nerfstudio convention recently.