isl-org / Open3D

Open3D: A Modern Library for 3D Data Processing
http://www.open3d.org
Other
11.36k stars 2.29k forks source link

How to convert `open3d.cuda.pybind.geometry.trianglemesh` `open3d.visualization.rendering.TriangleMeshModel`? #6388

Open HtutLynn opened 1 year ago

HtutLynn commented 1 year ago

Checklist

My Question

Hi, I am trying to visualize a camera pose in 3D pointcloud in Open3D by using open3d.visualization.O3DVisualizer and import open3d.visualization.gui . A camera pose is created by using this function,

def draw_camera(K, R, t, w, h,
                scale=1, color=[0.8, 0.2, 0.8]):
    """Create axis, plane and pyramed geometries in Open3D format.
    :param K: calibration matrix (camera intrinsics)
    :param R: rotation matrix
    :param t: translation
    :param w: image width
    :param h: image height
    :param scale: camera model scale
    :param color: color of the image plane and pyramid lines
    :return: camera model geometries (axis, plane and pyramid)
    """

    # intrinsics
    K = K.copy() / scale
    Kinv = np.linalg.inv(K)

    # 4x4 transformation
    T = np.column_stack((R, t))
    T = np.vstack((T, (0, 0, 0, 1)))

    # axis
    axis = open3d.geometry.TriangleMesh.create_coordinate_frame(
        size=0.5 * scale)
    axis.transform(T)

    # points in pixel
    points_pixel = [
        [0, 0, 0],
        [0, 0, 1],
        [w, 0, 1],
        [0, h, 1],
        [w, h, 1],
    ]

    # pixel to camera coordinate system
    points = [Kinv @ p for p in points_pixel]

    # image plane
    width = abs(points[1][0]) + abs(points[3][0])
    height = abs(points[1][1]) + abs(points[3][1])
    plane = open3d.geometry.TriangleMesh.create_box(width, height, depth=1e-6)
    plane.paint_uniform_color(color)
    plane.translate([points[1][0], points[1][1], scale])
    plane.transform(T)

    # pyramid
    points_in_world = [(R @ p + t) for p in points]
    lines = [
        [0, 1],
        [0, 2],
        [0, 3],
        [0, 4],
    ]
    colors = [color for i in range(len(lines))]
    line_set = open3d.geometry.LineSet(
        points=open3d.utility.Vector3dVector(points_in_world),
        lines=open3d.utility.Vector2iVector(lines))
    line_set.colors = open3d.utility.Vector3dVector(colors)

    # return as list in Open3D format
    return [axis, plane, line_set]

I am trying to visualize the trianglemesh from this function in

app = gui.Application.instance
        app.initialize()

        points = self.create_point_cloud()

        vis = open3d.visualization.O3DVisualizer("Environment PointCloud", 1024, 768)
        vis.show_settings = True
        vis.add_geometry("Points", points)

        # camera Poses
        ** create frames

       # camera_poses is a list of open3d.cuda.pybind.geometry.trianglemesh objects.
        for fidx, camera_pose in enumerate(camera_poses):
            vis.add_geometry(name="camera{}".format(fidx),
                             model=camera_pose,
                             group="Camera Poses",
                             time=0.0,
                             is_visible=True)

But I am getting this error,

TypeError: add_geometry(): incompatible function arguments. The following argument types are supported:
    1. (self: open3d.cuda.pybind.visualization.O3DVisualizer, name: str, geometry: open3d.cuda.pybind.geometry.Geometry3D, material: open3d.cuda.pybind.visualization.rendering.MaterialRecord = None, group: str = '', time: float = 0.0, is_visible: bool = True) -> None
    2. (self: open3d.cuda.pybind.visualization.O3DVisualizer, name: str, geometry: open3d.cuda.pybind.t.geometry.Geometry, material: open3d.cuda.pybind.visualization.rendering.MaterialRecord = None, group: str = '', time: float = 0.0, is_visible: bool = True) -> None
    3. (self: open3d.cuda.pybind.visualization.O3DVisualizer, name: str, model: open3d.cuda.pybind.visualization.rendering.TriangleMeshModel, material: open3d.cuda.pybind.visualization.rendering.MaterialRecord = None, group: str = '', time: float = 0.0, is_visible: bool = True) -> None
    4. (self: object, arg0: dict) -> None

Invoked with: <open3d.cuda.pybind.visualization.O3DVisualizer object at 0x7f87b5d7d230>; kwargs: name='frame0', model=TriangleMesh with 1134 points and 2240 triangles., group='frames', time=0.0, is_visible=True

Previously, I was using open3d.visualization.Visualizer api to visualize this trianglemesh model by using add_geometry method and it is okay but if I switch open3d.visualization.O3DVisualizer, this error happens.

The camera pose trianglemesh that I want to visualize is something like this: pose

Thank you for the help in advance!

hernot commented 11 months ago

Cuda objects are tensor (.t) objects you have to convert them back to .geometry which are .cpu objects. To do so call the .to_legacy() of your cuda type TriangleMesh before passing to visualization.O3DVisualizer. If you need to bring legacy geometry.TriangleMesh to cuda/tensor than you have to use t.geometry.TriangleMesh.from_legacy() function which is the opposite of .to_legacy() method.