pointcloud for motivating example

[kbkartik]

Hi, I came across your work and was trying out to understand the details. For the motivating example, where do you define the number of points for the point cloud for both the source and target spheres?

When I run your code as is, obs['xyz'].shape is [1, 3, 2] and obs['rgb'].shape is [1, 3, 2]. Does this mean there are two points, 1 for source and 1 for target?

How can I increase the number of points?

Thanks!

[lz1oceani]

You can extract point clouds from RGB-D images. You can refer functions like read_pointclouds_from_camera in https://github.com/lz1oceani/pointcloud_rl/blob/main/mani_skill/mani_skill/env/camera.py

[kbkartik]

My doubt was more related to the motivating example. When I train the agent on the motivating example, I get obs dict where obs['xyz'].shape is [1, 3, 2] and obs['rgb'].shape is [1, 3, 2]. Does this mean there are two points, 1 for source and 1 for target? How can I increase the number of points of the point cloud?

Also, how do you generate the point clouds for mujoco-based envs from images?

[lz1oceani]

Yes, the point clouds are not directly computed from the images but rather from precise position data.

To increase the number of points in the point cloud, you need to compute them from the images. For better comparison, you can use the mean positions of the two point clouds derived from the images as inputs.

[kbkartik]

So for the motivating example, I saw you sample two pixel points, one for source and target each. In addition, you sample two random densities for each of these points.

Using these two sampled pixels, you use inverse projections to retrieve the point clouds.

my question is to check my following understanding: if I increase the number of sampled pixels, will it increase the number of points in the point clouds?