[Open3D Python] Extremely slow visualization of streamed 3D point cloud

[JeffR1992]

Checklist

• [X] I have searched for similar issues.
• [X] For Python issues, I have tested with the latest development wheel.
• [X] I have checked the release documentation and the latest documentation (for master branch).

My Question

I'm trying to visualize a 3D point cloud from a ROS bag file in a for loop with pyrealsense and open3d. What I've noticed, however, is the following:

1) Visualizing the 3D point cloud with open3d is excruciatingly slow, where the loop runs at about 0.2 Hz 2) Even though the loop runs, it doesn't look like the open3d visualizer is updating the 3D point cloud

I know it's not an issue with pyrealsense, since when I visualize depth frames with opencv's imshow it can run at up to 90 Hz in the loop. However, the moment I add in the open3d point cloud visualization the framerate drop to about 0.2 Hz.

Here is a basic working example of the code I'm using (I tested it on a bag file recording from an Intel RealSense D435i camera):

import pyrealsense2 as rs
import numpy as np
import open3d as o3d

def depth_to_pointcloud(depth_frame):
    pc = rs.pointcloud()
    points = pc.calculate(depth_frame)
    verts = np.asanyarray(points.get_vertices()).view(np.float32).reshape(-1, 3)  # xyz

    return verts

def main():
    file_path = "path/to/bag/file"

    config = rs.config()
    config.enable_device_from_file(file_path, repeat_playback=False)

    pipeline = rs.pipeline()
    pipeline.start(config)

    point_cloud = o3d.geometry.PointCloud()
    vis = o3d.visualization.Visualizer()
    vis.create_window()

    i = 0
    while True:
        print(i)
        frames = pipeline.wait_for_frames()

        depth_frame = frames.get_depth_frame()
        verts = depth_to_pointcloud(depth_frame)
        point_cloud = o3d.geometry.PointCloud(o3d.utility.Vector3dVector(verts))

        if(i == 0):
            vis.add_geometry(point_cloud)
        vis.update_geometry(point_cloud)
        vis.poll_events()
        vis.update_renderer()

        i += 1

if __name__ == "__main__":
    main()

Is there anything I'm doing wrong? I'd love to be able to visualize a 3D point cloud with open3d.

[maxfreu]

Try adding a vis.clear_geometries() before add_geometry. Otherwise you keep adding new stuff until it gets too heavy to render. Apart from that, I chose another approach for the visualization I had to do; I let the visualization run in a separate python thread and send the data to be visualized over via a Queue. The separate python process practically only does:

class VisualizationProcess(Process):
    def __init__(self, queue):
        super().__init__()
        self.q = queue

    def animation_callback(self, vis):
        view_control = vis.get_view_control()
        camera_setup = view_control.convert_to_pinhole_camera_parameters()
        try:
            config, items = self.q.get_nowait()
            self.parse_config(config)
            self.plot_system(vis, *items)

            # vis.update_geometry()
            view_control.convert_from_pinhole_camera_parameters(camera_setup)
            vis.poll_events()
            vis.update_renderer()
            time.sleep(0.01)

        except Empty:
            time.sleep(0.01)

    def run(self):
        pcd = o3d.geometry.PointCloud()
        pcd.points = o3d.utility.Vector3dVector(np.array([[0,0,600],[0,0,1600]]))
        self.vis = o3d.visualization.Visualizer()
        self.vis.create_window()
        self.vis.add_geometry(pcd)
        self.vis.register_animation_callback(self.animation_callback)
        self.vis.run()
        self.vis.destroy_window()

    def plot_system(self, stuff):
        pcd = ...
        self.vis.clear_geometries()
        self.vis.add_geometry(pcd)

[JeffR1992]

I only call add_geometry() on the very first iteration, after which it is never called again, and update_geometry() is instead called thereafter.

I'd be surprised if simply moving the Open3D visualization to another thread would produce a drastic boost in the visualization's framerate, since the non-Open3D functions only take a fraction of a second to compute.

[maxfreu]

Have you tried plotting random data to see if that slows down as well? If yes, can you provide the copy-pastable code?

[ssheorey]

Please see this example: https://github.com/isl-org/Open3D/blob/master/examples/python/visualization/online_processing.py