Closed SibylGao closed 1 year ago
Hi, Thanks for your interest in the work.
Voxel = F.grid_sample(out, norm_coord_imgs, align_corners=True)
This transformation maps the PSV with coordinates (u, v, d) to a 3D voxel grid with coordinates (x, y, z).
Voxel_2D = self.build_3d_geometry_volume(left_sem_feat, norm_coord_imgs, voxel_disps)
This transformation maps the camera features (HxWxC) to a 3D voxel grid with coordinates (x, y, z).
PSV_from_3dgv = F.grid_sample(Voxel, norm_coordinates_psv_to_3d)
This transformation maps the 3D voxel grid with coordinate (x, y, z) back to the frustum space (u, v, d) and compute the depth map loss. (Front Surface Depth Head)
3D voxel grid with coordinates (x, y, z).
Thanks for your reply! It really helps a lot. So the 3D voxel grid coordinate (x, y, z) actually means (depth, width, height) in pseudo lidar coordinate?
Yes, just note that The voxel grid is with shape [Channel, Height (z), Width (y), Depth (x)] where LiDAR coordinate should be [x (forward), y (left), z (up)].
By the way, have you tried train & eval on max_disp = 16, surprisingly, I got results very close to that train & eval on max_disp=288, is it might caused by overfitting?
Hi, I think it should not get similar results. Could you show the shape of your generated feature size (3DGV) with these modifications? Perhaps it is due to bugs like duplicated config definitions.
Hi! Thanks for sharing your awesome work, but i am so confused about the coordinate system in your code. Firstly, depth-wise cost volumes are build in PSV:
Then, a 3d mesh grid in pseudo lidar coordinate is generated:
and 3D world mesh grid to camera frustum space (I believe is via the
torch.cat
operation?):But I'm really confused by the following
grid_sample
operations, such as:and:
So this means that
out(cost0)
andleft_sem_feat
are both in image coordinate, and are mapping to normalized camera frustum space by filling the grid (forcost volume
, its value is sampled in the volume, while forsem_fea
, its grid is filled by replication along the depth axis )? After that, theVoxel
is 'grid_sampled' again in the last depth estimation stage:It would be so helpful if you shared more details about the coordinate system and coordinate transformation in your code :confounded: Thank you so much.