Flipped reconstruction result

[Pamphlett]

Hi Yucheng,

I find the token-reconstructed LiDAR flipped compared to the original one (see below the original in green and reconstructed in blue). Do you have any idea about this?

[myc634]

It looks like you are using different coordinate systems (ego and lidar) to do visualization

[Pamphlett]

I think I didn't. The coordinate system is the same. The green one is the input and the blue one is the reconstruction from the tokens of the input.

Is there any possibility inside the network of letting the lidar be flipped?

[myc634]

could you show me the code you are using to vis the gt point cloud and the reconstructed point cloud?

[Pamphlett]

I have the following function for the generation process:

def sparse_to_sparse_generation(self, points):
      self.iter += 1
      folder_name = "s2s_tok@50_test"
      # code_folder = "nusc_samples_lidar_code"
      os.makedirs(f"{folder_name}", exist_ok=True)
      # os.makedirs(f"{code_folder}", exist_ok=True)
      with torch.no_grad():

          points_dt = [p.detach().cpu().numpy() if isinstance(p, torch.Tensor) else np.array(p) for p in points]
          points_dt = np.vstack(points_dt)
          points_dt = points_dt[:, :3]
          # points_dt = points_dt[:, [1, 0, 2]]
          # print(points_dt.shape)

          # print(f"Saving original points, shape: {points_dt.shape}")  # 输出调试信息

          # dump_ply("{}/pc_{}_original.ply".format(folder_name, self.iter), points_dt)

          voxels = self.voxelizer([[_] for _ in points])

          lidar_feats = self.lidar_encoder(voxels)
          feats = self.pre_quant(lidar_feats)
          lidar_quant, emb_los , code_indices = self.vector_quantizer(feats, self.code_age, self.code_usage)

          # import pdb; pdb.set_trace()

          # dump lidar_quant
          # dump_code("{}/lidar_code_{}.npz".format(code_folder, self.iter), code_indices.detach().cpu().numpy())

          # import pdb; pdb.set_trace()

          estimated_tokens = self.vector_quantizer.get_codebook_entry(code_indices)

          recon_voxel = self.lidar_decoder(estimated_tokens)
          recon_voxel = gumbel_sigmoid(recon_voxel, hard=True)
          recon_pc = self.voxels2points(recon_voxel)

          recon_pc = recon_pc.detach().cpu().numpy()
          # recon_pc_fliped = recon_pc[:, [1, 0, 2]]

          # dump_ply("{}/pc_{}_generated.ply".format(folder_name, self.iter), recon_pc)

          # import pdb; pdb.set_trace()

      return recon_pc, code_indices

[myc634]

Sorry, I mean the code to draw the picture you showed.

[Pamphlett]

I saved the two point clouds into two .ply files and visualized them in cloudcompare.

[myc634]

What's the purpose of swapping the xy coordinates here: points_dt = points_dt[:, [1, 0, 2]]? Try not to swap them. As the reconstructed point cloud looks good, I'm pretty sure the training process is correct. The bug should lie somewhere when you're trying to dump your point cloud.

[Pamphlett]

Sure I will try to debug again. Thx for your help :)

[Pamphlett]

Hey Yucheng,

I think I found the reason for the flipped result. The line here take the rear two columns as xy whereas they should be yx according to the bev definition here and the return value here.

Please correct me if I'm wrong.