Questions about "vertices_z" optimation in SquareFlatGridRGBLabel

[flashcp]

Hello, I have a question. In the SquareFlatGridRGBLabel class, will the vertices_z value be optimized? I noticed it is initialized to 0, but it doesn't seem to be optimized during backpropagation. Or I misunderstanding something?

https://github.com/DRosemei/RoMe/blob/a634caa302a34d0631f4e81351a04ee612f9f654/models/voxel.py#L135C13-L135C13

If I understand correctly, what is the reason behind this design? What are the differences compared to Inverse Perspective Mapping (IPM)?

[DRosemei]

@flashcp We make poses to be flat first, refer here, so road elevations would be mostly around 0. So we do experiments on elevations=0. This is different from IPM. IPM is under assumption that near ground plane is flat, while our method is a global optimization method, a single bev elevation is updated by many images.

[flashcp]

@flashcp We make poses to be flat first, refer here, so road elevations would be mostly around 0. So we do experiments on elevations=0. This is different from IPM. IPM is under assumption that near ground plane is flat, while our method is a global optimization method, a single bev elevation is updated by many images.

Thank you for your response. In my experiments, I found that when using the local_kitti.yaml configuration file (where vertices_z is commented out), there is no change in vertices_z during the training process. Is this normal? https://github.com/DRosemei/RoMe/blob/a634caa302a34d0631f4e81351a04ee612f9f654/configs/local_kitti.yaml#L26

According to the paper and what you mentioned, RoMe should optimize the BEV elevation. However, my experimental results show that vertices_z does not change, which makes me skeptical about my experiments.

[DRosemei]

@flashcp Can you provide some data about "vertices_z does not change”. I think vertices_z will not be zero when lr vertices_z != 0.

[flashcp]

@flashcp Can you provide some data about "vertices_z does not change”. I think vertices_z will not be zero when lr vertices_z != 0.

Sure, this my 'Training Parameters' configuration, note that vertices_z is turned off.

# Training Parameters
waypoint_radius: 400  # meters
batch_size: 6
pos_enc: 4
lr: 
  vertices_rgb: 0.1
  vertices_label: 0.1
  # vertices_z: 0.001
  rotations: 0.01
  translations: 0.01

This is the heatmap of vertices_z before training. My OrthographicCameras set translation as np.array([-cx, -cy*2, 1]), so the zmin, zmax, zmean(vertices_z) is based in 1.

This is the heatmap of vertices_z at epoch=1.

This is the heatmap of vertices_z at epoch=10.

Is it because I turned off vertices_z that this is happening?

[DRosemei]

“Is it because I turned off vertices_z that this is happening?” Yes, it is. SquareFlatGridRGBLabel inherits SquareFlatGridBase, so vertices_z are not a learnable parameters. https://github.com/DRosemei/RoMe/blob/a634caa302a34d0631f4e81351a04ee612f9f654/models/voxel.py#L135

[flashcp]

@DRosemei Thanks, now I understand.