For the o3d.visualization.O3DVisualizer class, how to add another panel or window with image widgets and other info?

Checklist

[X] I have searched for similar issues.
[X] For Python issues, I have tested with the latest development wheel.
[X] I have checked the release documentation and the latest documentation (for main branch).

My Question

Hi there, thanks for the great library!

Context

I am trying to add a gui to the mapping script of the concept-graphs mapping system. The code is ugly at the moment but you can have a look here.

This mapping system takes as input rgb images + depth images + poses, and iteratively constructs an object based 3D map that also encodes semantic features. I want to be able to visualize the whole map building process, so I can inspect the map and debug things as it's being built.

So I looked at the python examples and saw that the multiple_windows.py example seems to have what I want, which is to visualize the object pointclouds and the bounding boxes and so on. I can do that, it works great so far.

The Problem

The issue is that I want to also display more information. I would like to be able to see in a panel beside the map: the current rgb image, the current depth image, the current annotated image, the current frame, the current number of objects and so on.

I want it to be like the dense_slam_gui.py example in the docs.

I have tried a bunch of stuff but I cannot seem to be able to add a panel to the o3d.visualization.O3DVisualizer thing, and I was also unable to add another window or something that I can update at the same time. I don't mind if its not the same window, I just want to be able to see this information displayed somewhere alongside the map, as it would help me improve the mapping system.

My Code

Here are some relevant code snippets.

How I make the app (as you can see I have tried stuff to add another window or panel, but everything I have tried so far has resulted in some kind of error)

def run(self):
    app = o3d.visualization.gui.Application.instance
    app.initialize()

    self.main_vis = o3d.visualization.O3DVisualizer(
        "Open3D - Multi-Window Demo")
    self.main_vis.add_action("Take snapshot in new window",
                             self.on_snapshot)
    self.main_vis.add_action("Pause/Resume updates", lambda vis: self.toggle_pause())
    self.main_vis.set_on_close(self.on_main_window_closing)

    app.add_window(self.main_vis)
    # app.add_window(self.images_window)

    # # Setup the secondary window for images
    # # self.setup_image_display_window()
    # # self.image_window = app.create_window("RGB and Depth Images", 640, 480)
    # self.image_window = gui.Application.instance.create_window("RGB and Depth Images", 640, 480)
    # # self.image_window.create_window()
    # # self.image_window = o3d.visualization.Visualizer("RGB and Depth Images", 640, 480)
    # self.layout = o3d.visualization.gui.Vert(0, o3d.visualization.gui.Margins(10))
    # self.image_window.add_child(self.layout)

    # # Create image widgets
    # self.rgb_widget = o3d.visualization.gui.ImageWidget()
    # self.depth_widget = o3d.visualization.gui.ImageWidget()

    # # Add image widgets to the layout
    # self.layout.add_child(self.rgb_widget)
    # self.layout.add_child(self.depth_widget)
    # app.add_window(self.image_window)

    self.snapshot_pos = (self.main_vis.os_frame.x, self.main_vis.os_frame.y)
    threading.Thread(target=self.update_thread).start()
    app.run()

Here is how I do the update thread:

    def update_thread(self):
        # This is NOT the UI thread, need to call post_to_main_thread() to update
        # the scene or any part of the UI.

        ... lots of code, basically the whole mapping script ...

            def my_update_cloud():                

                # Remove previous objects
                for obj_name in self.prev_obj_names:
                    self.main_vis.remove_geometry(obj_name)
                self.prev_obj_names = []

                # Remove previous bounding boxes
                for bbox_name in self.prev_bbox_names:
                    self.main_vis.remove_geometry(bbox_name)
                self.prev_bbox_names = []

                # Add the new objects and bounding boxes
                for obj_num, obj in enumerate(self.objects):
                    obj_label = f"{obj['curr_obj_num']}_{obj['class_name']}"

                    obj_name = f"obj_{obj_label}"
                    bbox_name = f"bbox_{obj_label}"

                    self.prev_obj_names.append(obj_name)
                    self.main_vis.add_geometry(obj_name, obj['pcd'])

                    self.prev_bbox_names.append(bbox_name)
                    self.main_vis.add_geometry(bbox_name, obj['bbox'] )

            if self.is_done:  # might have changed while sleeping
                break
            o3d.visualization.gui.Application.instance.post_to_main_thread(
                self.main_vis, my_update_cloud)

What I want to do

For starters, I just want to be able to update a display with the current rgb and depth image and maybe the current frame number?. I'm imagining something like:

    def update_images(self):
        # Convert numpy images to Open3D images and update widgets
        o3d_rgb_image = o3d.geometry.Image(self.curr_image_rgb)
        o3d_depth_image = o3d.geometry.Image((self.curr_depth_array / self.curr_depth_array.max() * 255).astype(np.uint8))  # Example normalization

        self.rgb_widget.update_image(o3d_rgb_image)
        self.depth_widget.update_image(o3d_depth_image)

# and then in the main update thread loop I would do        
self.curr_image_rgb = image_rgb
self.curr_depth_array = depth_array
o3d.visualization.gui.Application.instance.post_to_main_thread(
    self.image_window, self.update_images())

I hope it makes sense, what I'm trying to achieve here. If anyone could point me in the right direction, this would be much appreciated. I have googled and asked chatGPT a lot to no avail.

Thank you in advance for your time and help.

isl-org / Open3D