Non-blocking visualization on VoxelGrid fails to update visualization

[magnusbarata]

The issue I'm trying to convert realsense point cloud to voxel and visualize it in real time.

I followed this example to visualize realsense point cloud in real time. But instead of passing point cloud object to the visualizer, I passed a VoxelGrid object. The first VoxelGrid is displayed in the visualization window, but it is not getting updated at all. The same thing occurs when I passed a point cloud processed using voxel_down_sample(). Could it be that this is a bug of Visualizer class?

To reproduce Here is part of my code:

if __name__ == "__main__":
    # Realsense initialization
    pipeline = rs.pipeline()
    config = rs.config()
    config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 15)
    config.enable_stream(rs.stream.color, 1280, 720, rs.format.rgb8, 15)

    # Start streaming
    profile = pipeline.start(config)
    depth_sensor = profile.get_device().first_depth_sensor()
    depth_sensor.set_option(rs.option.visual_preset, Preset.Default)
    depth_scale = depth_sensor.get_depth_scale()

    # We will not display the background of objects more than
    #  clipping_distance_in_meters meters away
    clipping_distance_in_meters = 3  # 3 meter
    clipping_distance = clipping_distance_in_meters / depth_scale

    # Create an align object
    align_to = rs.stream.color
    align = rs.align(align_to)

    vis = o3d.visualization.Visualizer()
    vis.create_window()
    vis.get_render_option().background_color = np.array([0, 0, 0])

    pcd = o3d.geometry.PointCloud()
    flip_transform = [[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]]

    # Streaming loop
    frame_count = 0
    try:
        while True:
            dt0 = datetime.now()

            # Get frameset of color and depth
            frames = pipeline.wait_for_frames()

            # Get aligned frames
            aligned_frames = align.process(frames)
            aligned_depth_frame = aligned_frames.get_depth_frame()
            color_frame = aligned_frames.get_color_frame()
            intrinsic = o3d.camera.PinholeCameraIntrinsic(
                get_intrinsic_matrix(color_frame))

            # Validate that both frames are valid
            if not aligned_depth_frame or not color_frame:
                continue

            depth_image = o3d.geometry.Image(
                np.array(aligned_depth_frame.get_data()))
            color_temp = np.asarray(color_frame.get_data())
            color_image = o3d.geometry.Image(color_temp)

            rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(
                color_image,
                depth_image,
                depth_scale=1.0 / depth_scale,
                depth_trunc=clipping_distance_in_meters,
                convert_rgb_to_intensity=False)
            temp = o3d.geometry.PointCloud.create_from_rgbd_image(
                rgbd_image, intrinsic)
            temp.transform(flip_transform)
            pcd.points = temp.points
            pcd.colors = temp.colors
            #downpcd = pcd.voxel_down_sample(voxel_size=0.05)
            voxel_grid = o3d.geometry.VoxelGrid.create_from_point_cloud(
                            pcd, voxel_size=0.05)
            print(voxel_grid)

            if frame_count == 0:
                #vis.add_geometry(downpcd)
                #vis.add_geometry(pcd)
                vis.add_geometry(voxel_grid)

            #vis.update_geometry(downpcd)
            #vis.update_geometry(pcd)
            vis.update_geometry(voxel_grid)
            vis.poll_events()
            vis.update_renderer()

            process_time = datetime.now() - dt0
            print("FPS: " + str(1 / process_time.total_seconds()))
            frame_count += 1

    finally:
        pipeline.stop()
    vis.destroy_window()

Environment

• OS: Ubuntu 18.04
• Python: 3.6.8
• Open3D: 0.8.0, 0.10.0

Thanks for any help you can provide.

[wuyunnben]

Same issue find in c++. I'm trying to draw occupancy grid map in real time, but only a blank window. Here is my test code

int main(int argc, char *argv[]) {
    using namespace open3d;
    visualization::Visualizer vis;
    vis.CreateVisualizerWindow("Open3D", 500, 500);

    auto occ_geometry = std::make_shared<geometry::VoxelGrid>();
    Eigen::Vector3d origin(0.0,0.0,0.0);
    occ_geometry->origin_ = origin;
    occ_geometry->voxel_size_ = 0.05;
    vis.AddGeometry(occ_geometry);

    for (int widx = 0; widx < 20; widx++) {
        for (int hidx = 0; hidx < 20; hidx++) {
            Eigen::Vector3i grid_index(widx, hidx, 0);
            Eigen::Vector3d color(widx*4, hidx*4, 128);
            occ_geometry->AddVoxel(geometry::Voxel(grid_index, color));
            vis.UpdateGeometry(occ_geometry);
            vis.PollEvents();
            vis.UpdateRender();
        }
    }
    return 0;
}

[griegler]

@yxlao can you have a look at this please.

[hyungjungkim]

Hello. I have the same problem with VoxelGrid. Please check this issue.

[batoutou]

Hello, has anyone been able to fix the problem. Visualizing point cloud is ok, but for voxelgrid it still doesn't work.

[arielc-brillianetor]

Hi, i've just encountered the same problem, any chance a solution was found?

my point cloud works, but then when I run the following I get a blank screen:

voxel_grid = o3d.geometry.VoxelGrid.create_from_point_cloud(pcd, voxel_size=0.5) o3d.visualization.draw_geometries([voxel_grid])

I tried with different sizes set to the voxel_size., My grid contains 4600 voxels, but still blank screen. I'm running open3d on visual studio 2019, windows 10. Thanks a lot.

[goksanisil23]

member variables of the visualizer class:

open3d::visualization::Visualizer            gridViz_;
std::shared_ptr<open3d::geometry::VoxelGrid> voxelGrid_;

initializing the visualizer:

gridViz_.CreateVisualizerWindow("Grid", 960, 540, 480, 270);
gridViz_.GetRenderOption().background_color_      = {0.05, 0.05, 0.05};
gridViz_.GetRenderOption().point_size_            = 1;
gridViz_.GetRenderOption().show_coordinate_frame_ = true;

// Add voxel grid
    voxelGrid_ = std::make_shared<open3d::geometry::VoxelGrid>();
    Eigen::Vector3d origin(0.0, 0.0, 0.0);
    voxelGrid_->origin_     = origin;
    voxelGrid_->voxel_size_ = 0.5;

    for (int widx = 0; widx < 20; widx++)
    {
        for (int hidx = 0; hidx < 20; hidx++)
        {
            Eigen::Vector3i grid_index(widx, hidx, 0);
            Eigen::Vector3d color((double)widx / (double)20, (double)hidx / (double)20, 1.0);
            voxelGrid_->AddVoxel(open3d::geometry::Voxel(grid_index, color));
        }
    }
    gridViz_.AddGeometry(voxelGrid_);

Update the visualizer in your main loop:

gridViz_.UpdateGeometry(voxelGrid_);
gridViz_.PollEvents();
gridViz_.UpdateRender();

[JakFed]

member variables of the visualizer class:

open3d::visualization::Visualizer            gridViz_;
std::shared_ptr<open3d::geometry::VoxelGrid> voxelGrid_;

initializing the visualizer:

gridViz_.CreateVisualizerWindow("Grid", 960, 540, 480, 270);
gridViz_.GetRenderOption().background_color_      = {0.05, 0.05, 0.05};
gridViz_.GetRenderOption().point_size_            = 1;
gridViz_.GetRenderOption().show_coordinate_frame_ = true;

// Add voxel grid
    voxelGrid_ = std::make_shared<open3d::geometry::VoxelGrid>();
    Eigen::Vector3d origin(0.0, 0.0, 0.0);
    voxelGrid_->origin_     = origin;
    voxelGrid_->voxel_size_ = 0.5;

    for (int widx = 0; widx < 20; widx++)
    {
        for (int hidx = 0; hidx < 20; hidx++)
        {
            Eigen::Vector3i grid_index(widx, hidx, 0);
            Eigen::Vector3d color((double)widx / (double)20, (double)hidx / (double)20, 1.0);
            voxelGrid_->AddVoxel(open3d::geometry::Voxel(grid_index, color));
        }
    }
    gridViz_.AddGeometry(voxelGrid_);

Update the visualizer in your main loop:

gridViz_.UpdateGeometry(voxelGrid_);
gridViz_.PollEvents();
gridViz_.UpdateRender();

nice, this works. But it still not works if you create a voxelgrid from a pointcloud. Somebody has a solution to it? In Python it works, but not in C++.

[AwokeKnowing]

i can't get this to work in python. i can't see how to update from updated pointcloud

[AwokeKnowing]

i got it to work by creating a new grid each time and then swapping it vis.remove_geometry(vxgrid,False) vxgrid=vxgrid_new vis.add_geometry(vxgrid,False) vis.poll_events() vis.update_renderer()

[jianchaoci]

The issue I'm trying to convert realsense point cloud to voxel and visualize it in real time.

I followed this example to visualize realsense point cloud in real time. But instead of passing point cloud object to the visualizer, I passed a VoxelGrid object. The first VoxelGrid is displayed in the visualization window, but it is not getting updated at all. The same thing occurs when I passed a point cloud processed using voxel_down_sample(). Could it be that this is a bug of Visualizer class?

To reproduce Here is part of my code:

if __name__ == "__main__":
    # Realsense initialization
    pipeline = rs.pipeline()
    config = rs.config()
    config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 15)
    config.enable_stream(rs.stream.color, 1280, 720, rs.format.rgb8, 15)

    # Start streaming
    profile = pipeline.start(config)
    depth_sensor = profile.get_device().first_depth_sensor()
    depth_sensor.set_option(rs.option.visual_preset, Preset.Default)
    depth_scale = depth_sensor.get_depth_scale()

    # We will not display the background of objects more than
    #  clipping_distance_in_meters meters away
    clipping_distance_in_meters = 3  # 3 meter
    clipping_distance = clipping_distance_in_meters / depth_scale

    # Create an align object
    align_to = rs.stream.color
    align = rs.align(align_to)

    vis = o3d.visualization.Visualizer()
    vis.create_window()
    vis.get_render_option().background_color = np.array([0, 0, 0])

    pcd = o3d.geometry.PointCloud()
    flip_transform = [[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]]

    # Streaming loop
    frame_count = 0
    try:
        while True:
            dt0 = datetime.now()

            # Get frameset of color and depth
            frames = pipeline.wait_for_frames()

            # Get aligned frames
            aligned_frames = align.process(frames)
            aligned_depth_frame = aligned_frames.get_depth_frame()
            color_frame = aligned_frames.get_color_frame()
            intrinsic = o3d.camera.PinholeCameraIntrinsic(
                get_intrinsic_matrix(color_frame))

            # Validate that both frames are valid
            if not aligned_depth_frame or not color_frame:
                continue

            depth_image = o3d.geometry.Image(
                np.array(aligned_depth_frame.get_data()))
            color_temp = np.asarray(color_frame.get_data())
            color_image = o3d.geometry.Image(color_temp)

            rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth(
                color_image,
                depth_image,
                depth_scale=1.0 / depth_scale,
                depth_trunc=clipping_distance_in_meters,
                convert_rgb_to_intensity=False)
            temp = o3d.geometry.PointCloud.create_from_rgbd_image(
                rgbd_image, intrinsic)
            temp.transform(flip_transform)
            pcd.points = temp.points
            pcd.colors = temp.colors
            #downpcd = pcd.voxel_down_sample(voxel_size=0.05)
            voxel_grid = o3d.geometry.VoxelGrid.create_from_point_cloud(
                            pcd, voxel_size=0.05)
            print(voxel_grid)

            if frame_count == 0:
                #vis.add_geometry(downpcd)
                #vis.add_geometry(pcd)
                vis.add_geometry(voxel_grid)

            #vis.update_geometry(downpcd)
            #vis.update_geometry(pcd)
            vis.update_geometry(voxel_grid)
            vis.poll_events()
            vis.update_renderer()

            process_time = datetime.now() - dt0
            print("FPS: " + str(1 / process_time.total_seconds()))
            frame_count += 1

    finally:
        pipeline.stop()
    vis.destroy_window()

Environment

• OS: Ubuntu 18.04
• Python: 3.6.8
• Open3D: 0.8.0, 0.10.0

Thanks for any help you can provide.

I solved this problem by directly creating a Voxel_grid instead of adding a point cloud in a bland: Work: voxel_grid = o3d.geometry.VoxelGrid.create_from_point_cloud(pcd, voxel_size=self.voxel_size) axes = o3d.geometry.TriangleMesh.create_coordinate_frame(size=0.1) Failed: voxel_grid = o3d.geometry.VoxelGrid() voxel_grid.create_from_point_cloud(pcd, voxel_size=self.voxel_size) axes = o3d.geometry.TriangleMesh.create_coordinate_frame(size=0.1) o3d.visualization.draw_geometries([voxel_grid, axes]) o3d.visualization.draw_geometries([voxel_grid, axes])