How to save the mesh instead of showing on line?

[rockywind]

Hi, thank you for sharing the code. I run the code on the server so I haven't the monitor to show the result.

[heiwang1997]

Here is the thing you can do! :)

vis.to_file(vis.mesh(mesh.v, mesh.f), "a.ply")

[rockywind]

Thank you very much!

[rockywind]

Hi, Can you sharing the waymo-pcd.ply?

[heiwang1997]

Oops sorry for making a mistake in the common script! I've pushed a fix. Feel free to pull and check again. The file should be automatically downloaded.

[rockywind]

Hi, I want to run the code on my own data, my data style is the same as Kitti. Can you give me some tutorial.

[heiwang1997]

We haven't tested on KITTI dataset yet. But it should be easy! Some pseudo-code look like:

xyz_list = []
sensor_list = []
for frame_idx in range(1000):
    xyz = load_kitti_lidar(frame_idx)
    pose = load_kitti_pose(frame_idx)
    xyz_list.append(pose.transform(xyz))
    sensor_list.append(pose.transform([[0,0,0] for _ in range(xyz.shape[0])]))
xyz = np.concatenate(xyz_list)
sensor = np.concatenate(sensor_list)

Hope this helps!

[rockywind]

Thank you very much!

[rockywind]

How to add the color to the mesh on waymo dataset?

[rockywind]

@heiwang1997 Hi, I confuse with the setting of sensor on kitti dataset. I think the xyz is in world coordinate and the sensor is velo_to_world's translation. The value of sensor is the same for all point clouds.

The below is the reconstruction of kitti dataset.

[heiwang1997]

Hi thanks for the questions.

1. In the paper we didn't really show the color of the mesh, instead what we showed is the normal map, where the RGB color is actually the XYZ component of the normals. We use Open3D to visualize our mesh, and if you press Ctrl+9 on your keyboard, you should be able to see the normal map we've drawn on the paper.
2. Do you mean base_pose is the same for all input frames? That means your car is static I guess. Could you maybe visualize the positions of the sensors as well as the points just for debugging purposes?

Best.

[rockywind]

@heiwang1997 您好，感谢您的解答。我用中文提问会更清晰。

1. 自车是运动的，senseor(base_pose)值是取第0帧的velo_to_world的translation(矩阵的第4列)，对所有的输入帧base_pose的值是一样的。 因为我把第一帧当做世界坐标的原点。
2. 另一个思路是，senseor(base_pose)值取每个输入帧的velo_to_world的translation(矩阵的第4列)。对所有的输入帧base_pose的值是不一样的，每一帧的sensor值是其对应的velo_to_world的translation(矩阵的第四列)。 下图是采用第一种（1）做法的结果 下图是采用第二种（2）方法的结果

[heiwang1997]

您好，我对您的第二点不是太理解，为何“senseor(base_pose)值取每个输入帧的velo_to_world”，但“对所有的输入帧base_pose的值是一样的”呢？可以理解为所有的输入帧的pose都是一样的吗？

另外，KITTI数据集噪声比较大，可以尝试提升一下Normal estimation knn的值～

[rockywind]

您好，之前描述有错误，已经修改了。 您觉得这两种做法有什么问题吗？

[heiwang1997]

好的，谢谢！ 我觉得第二种做法应该是没问题的，如果觉得噪声比较大，可以尝试两种方案：

1. 提升一下Normal estimation knn的值，就是 get_estimate_normal_preprocess_fn的第一个参数；
2. 另一个trick是吧sensor的位置全局调高一些，然后每一帧crop掉距离较远的点。

[rockywind]

十分感谢，我还有两个问题请教哈

1. 如果velo_to_world的矩阵的第一帧是单位矩阵，就是每帧的velo_to_world左乘第一帧的velo_to_world的逆矩阵，把第一帧为全局坐标的参考,这样做可以吗。[inverse(velo_to_world[0]) @ velo_to_world[i] for i in range(velo_to_world)]
2. repo里面的waymo-pcd.ply的重建效果很好，想问一下是单帧点云的效果，还是多帧点云合并的，如果是多帧合并，大概是几帧合并的？

[heiwang1997]

不客气的～有关你的问题：

1. 我不太清楚 velo_to_world 矩阵具体是pose还是pose的逆，只要能将激光雷达局部坐标系的点云变换到世界坐标系下即可，变换后尽可能Z轴朝上，因为我们的默认模型训练数据都是Z轴朝上的；
2. waymo是多帧点云合并的效果，我们总共使用了150帧左右的waymo数据（但中间有跳帧，每隔2帧选一帧）

[rockywind]

十分感谢，

1. velo_to_world是激光雷达坐标系转世界坐标系：point_in_world = velo_to_world(44) @ point_in_lidar(4N)。
2. 我们把第一帧的pose当做参考帧，[inverse(velo_to_world[0]) @ velo_to_world[i] for i in range(velo_to_world)]

[YuePanEdward]

I've got some quite good results on KITTI and Newer College datasets using NKSR and the reconstruction is also super fast (in about 10 seconds)

[rockywind]

@YuePanEdward Hi, Can you share the code on KITTI dataset?

[YuePanEdward]

You can try to add this to common.py.

And then replace xyz_np, sensor_np = load_waymo_example() with xyz_np, sensor_np = load_kitti_like_data(pc_folder, pose_file, calib_file, every_n_frame=1, begin_frame=0, end_frame=100) and provide the correct path to your dataset.

For the visualization, you may use open3d:

import open3d as o3d
mesh_v, mesh_f = mesh.v.detach().cpu().numpy(), mesh.f.detach().cpu().numpy()
mesh = o3d.geometry.TriangleMesh(o3d.utility.Vector3dVector(mesh_v), o3d.utility.Vector3iVector(mesh_f))
mesh.compute_vertex_normals()
o3d.visualization.draw_geometries([mesh])
o3d.io.write_triangle_mesh(your_mesh_path, mesh)

[qpc001]

I've got some quite good results on KITTI and Newer College datasets using NKSR and the reconstruction is also super fast (in about 10 seconds) I wonder if the nksr lib will also be made open-source? The marching cubes algorithm implemented there seems to be super fast.

How can you reconstruct large scale like kitti without CUDA out of Memory?

[fwilliams]

Hey wow @YuePanEdward these look amazing!

Any chance you want to send a PR with an example script for these datasets? This would be greatly appreciated by the community!

Also if you want to win a free GPU, feel free to upload anything cool you do here for the contest: nvda.ws/3NkPYs5

[rockywind]

You can try to add this to common.py.

And then replace xyz_np, sensor_np = load_waymo_example() with xyz_np, sensor_np = load_kitti_like_data(pc_folder, pose_file, calib_file, every_n_frame=1, begin_frame=0, end_frame=100) and provide the correct path to your dataset.

For the visualization, you may use open3d:

import open3d as o3d
mesh_v, mesh_f = mesh.v.detach().cpu().numpy(), mesh.f.detach().cpu().numpy()
mesh = o3d.geometry.TriangleMesh(o3d.utility.Vector3dVector(mesh_v), o3d.utility.Vector3iVector(mesh_f))
mesh.compute_vertex_normals()
o3d.visualization.draw_geometries([mesh])
o3d.io.write_triangle_mesh(your_mesh_path, mesh)

您好请问一下，能解释一下这段代码的含义吗

poses.append(
np.matmul(Tr_inv, np.matmul(pose, Tr))
)  # lidar pose in world frame

[YuePanEdward]

You can try to add this to common.py. And then replace xyz_np, sensor_np = load_waymo_example() with xyz_np, sensor_np = load_kitti_like_data(pc_folder, pose_file, calib_file, every_n_frame=1, begin_frame=0, end_frame=100) and provide the correct path to your dataset. For the visualization, you may use open3d:

import open3d as o3d
mesh_v, mesh_f = mesh.v.detach().cpu().numpy(), mesh.f.detach().cpu().numpy()
mesh = o3d.geometry.TriangleMesh(o3d.utility.Vector3dVector(mesh_v), o3d.utility.Vector3iVector(mesh_f))
mesh.compute_vertex_normals()
o3d.visualization.draw_geometries([mesh])
o3d.io.write_triangle_mesh(your_mesh_path, mesh)

您好请问一下，能解释一下这段代码的含义吗

        poses.append(
            np.matmul(Tr_inv, np.matmul(pose, Tr))
        )  # lidar pose in world frame

You may check the explanation here: https://github.com/PRBonn/kiss-icp/issues/156#issuecomment-1551695713, which should be similar to the issue here.

[heiwang1997]

@YuePanEdward The results look amazing! For the marching cubes implementation, please refer to the code here. Thank you! 🍉

[rockywind]

Hi, @heiwang1997 How to use the marching cubes implemention? Have a example code? The below is my own data result. How can I improve it?

[heiwang1997]

Yes, you can check out here for example code.

Have you tried out the tricks that we talked about in the previous messages? What does the input point cloud look like?

[rockywind]

Hi, This is my setting.

The below is my new result. My dataset is paking data. Can I finetune the model on my own data? The performance of reconstruction can improve when I finetune?

[heiwang1997]

Hey I'm not sure if everything is correctly set up here, and from your result it seems that parts of the ground are flipped. This could either be because your sensor positions are not accurate, or your point cloud contains too much noise, so our model fails to handle such cases. To fine-tune the model, you will need ground-truth geometry and we use the CARLA dataset for that -- it's available in the README. Feel free to check that out. Have you tried out the tricks that we talked about in the previous messages? What does the input point cloud look like?

[zgojcic]

Hi rockywind

Considering that this is parking data, the point cloud on the ground is probably very sparse because the ego-car doesn't move at all or only slightly. In such cases, you are left with individual scan lines on the ground, for which it is very hard to estimate the normals correctly. Could you maybe visualize the input point cloud?

[rockywind]

Hi,all This is the visualization of the input point cloud.

[rockywind]

Hi, @heiwang1997 当我调大sensor的Z+2值时候，重建的mesh会往上补全。

[heiwang1997]

Interesting... would you mind send me your file and let me investigate into it a bit? I think I will just need the XYZ + sensor array that you've generated

[leslieburke]

@heiwang1997 想请问一下sensor array里面给的是什么信息啊

[heiwang1997]

是每个点对应的传感器位置，即采集这个点的时候，传感器的xyz世界坐标～

[leslieburke]

@heiwang1997 再请教一下，我跑的时候报out of memory了，我有两张gpu，能不能用两张gpu来跑这个重建？另外参考readme里的内容，把 field.to_("cpu") reconstructor.network.to("cpu") 这两句话取消注释，会报错 RuntimeError: (*grid) and query are not on the same device!

[heiwang1997]

如果要用两个GPU的话，需要把input事先切成两块，然后分别输入每个GPU对应的Reconstructor 关于第二个错误，能麻烦发下完整的代码吗？我测试一下

[leslieburke]

如果要用两个GPU的话，需要把input事先切成两块，然后分别输入每个GPU对应的Reconstructor 关于第二个错误，能麻烦发下完整的代码吗？我测试一下

就是您example里的代码

device = torch.device("cuda:0")
reconstructor = nksr.Reconstructor(device)
reconstructor.chunk_tmp_device = torch.device("cpu")

input_xyz = torch.from_numpy(xyz_np).float().to(device)
input_sensor = torch.from_numpy(sensor_np).float().to(device)

field = reconstructor.reconstruct(
    input_xyz, sensor=input_sensor, detail_level=None,
    # Minor configs for better efficiency (not necessary)
    approx_kernel_grad=True, solver_tol=1e-4, fused_mode=True,
    # Chunked reconstruction (if OOM)
    chunk_size=50,
    preprocess_fn=nksr.get_estimate_normal_preprocess_fn(64, 85.0)
)

# (Optional) Convert to CPU for mesh extraction
field.to_("cpu")
reconstructor.network.to("cpu")

mesh = field.extract_dual_mesh(mise_iter=1)
mesh = vis.mesh(mesh.v, mesh.f)

[heiwang1997]

您好，我运行您的代码的时候提示找不到xyz_np和sensor_np变量，能烦请您发一下完整的代码吗？谢谢！

另外，如果直接运行 python examples/recons_waymo.py，我这里是不会报错的

[leslieburke]

您好，我运行您的代码的时候提示找不到xyz_np和sensor_np变量，能烦请您发一下完整的代码吗？谢谢！

另外，如果直接运行 python examples/recons_waymo.py，我这里是不会报错的

我这个代码上面读数据的load函数是我自己写的，读的我自己的数据，所以没放上来

我把examples/reconswaymo.py中的 field.to("cpu") reconstructor.network.to("cpu") 取消注释的话，还是会报错

[heiwang1997]

可以print一下nksr的版本吗？print(nksr.__version__)

[leslieburke]

可以print一下nksr的版本吗？print(nksr.__version__)

1.0.0+pt20cu118

[heiwang1997]

谢谢，请您安装最新版本的nksr：

pip install -U nksr -f https://nksr.huangjh.tech/whl/torch-2.0.0+cu118.html

[jeremycp3]

您好，请问可以将点云的深度信息加进去吗

[YoushaaMurhij]

You can try to add this to common.py.

And then replace xyz_np, sensor_np = load_waymo_example() with xyz_np, sensor_np = load_kitti_like_data(pc_folder, pose_file, calib_file, every_n_frame=1, begin_frame=0, end_frame=100) and provide the correct path to your dataset.

For the visualization, you may use open3d:

import open3d as o3d
mesh_v, mesh_f = mesh.v.detach().cpu().numpy(), mesh.f.detach().cpu().numpy()
mesh = o3d.geometry.TriangleMesh(o3d.utility.Vector3dVector(mesh_v), o3d.utility.Vector3iVector(mesh_f))
mesh.compute_vertex_normals()
o3d.visualization.draw_geometries([mesh])
o3d.io.write_triangle_mesh(your_mesh_path, mesh)

@YuePanEdward @rockywind I tried this visualization script and got a gray mesh. Could you please tell me how to add the original intensity colors to the mesh or any other similar feature? Thanks

[lxr319]

请问这个可以使用colmap中SFM的结果作为输入吗?

[heiwang1997]

SFM points from COLMAP don't have normal information and could be too sparse. Alternatively you could take the oriented point cloud from a typical MVG pipeline (e.g. from https://github.com/openMVG/openMVG) and replace the Poisson reconstruction with NKSR :)

[lxr319]

Alternatively you could take the oriented point cloud from a typical MVG pipeline

OK，Thank you for your reply, I'll try it.

[hungdche]

I've got some quite good results on KITTI and Newer College datasets using NKSR and the reconstruction is also super fast (in about 10 seconds)

Hi @YuePanEdward, sorry for digging this up. I'm trying to run NKSR on KITTI seq 0 and for some reason mine does not look as good as yours

I'm just asking if you use the defaults parameter as followed

field = reconstructor.reconstruct(
        input_xyz, sensor=input_sensor, detail_level=None,
        # Minor configs for better efficiency (not necessary)
        approx_kernel_grad=True, solver_tol=1e-4, fused_mode=True,
        # Chunked reconstruction (if OOM)
        chunk_size=51.2,
        preprocess_fn=nksr.get_estimate_normal_preprocess_fn(64, 85.0)
    )

Thanks in advance!