Is it necessary to display the window from create_window()?

benjiebob commented 5 years ago

Hi there,

I am currently using Open3D's PointCloud support in a training loop for a neural network. The goal is to save 10 different point clouds as images, so that I can send to Tensorboard for visualization as fast as possible.

Currently, my process is to create the 10 point clouds and then use:

image_list = []
for pcd in pcd_list:
   vis = Visualizer()
   vis.create_window()
   vis.add_geometry(pcd)
   vis.poll_events()
   vis.update_renderer()
   image_list.append(vis.capture_screen_float_buffer())
   vis.destroy_window()

However, the annoying (and I guess slow?) part is that the create_window() command creates a GLX window which is quickly displayed and closed. This results in a 'flickering' window during training time. Even just creating it in a 'hidden' mode would be better than nothing.

In another issue report, I read that you do not recommend editing geometries added to the visualizer so at the moment, this code creates and destroys 10 windows quite a few times during training.

Is there a method more in keeping with your design philosophy?

Thanks, Ben

syncle commented 5 years ago

Hi Ben, thanks for using Open3D!

create_window is necessary to create OpenGL context. To hide the window, I can recommend two options.

Try vis.create_window(visible = False) (contributed by @adamlofts). This will not create and pop-up any windows.
Try headless rendering. This will save images without physical screens.

benjiebob commented 5 years ago

This is an excellent answer, thank you very much.

Edit: Ah! I spoke slightly too soon.

My attempt at vis.create_window(visible = False) results in:

Traceback (most recent call last): File "tests/test_open3d.py", line 26, in vis.create_window(visible = False) TypeError: create_window(): incompatible function arguments. The following argument types are supported:

(self: open3d.linux.open3d.Visualizer, window_name: unicode=u'Open3D', width: int=1920L, height: int=1080L, left: int=50L, right: int=50L) -> bool

Invoked with: Visualizer with name Open3D; kwargs: visible=False

EDIT EDIT: As you say, this has been added, although very recently :). People facing this error probably need to make sure they are on the most recent version of the library. The current `pip' version isn't quite up to date.

syncle commented 5 years ago

Oops, my bad. As you mentioned, this feature were not released in 0.3v. We are about to release new version soon. We will update our package and relevant documents accordingly. At this moment, build from source is the option to use vis.create_window(visible = False), thanks!

benjiebob commented 5 years ago

Hi there,

I am still having trouble with this workflow. I have now cloned from source so have access to the `visible' keyword as you mention.

        vis = Visualizer()
        vis.create_window(height = window_size[0], width = window_size[1], visible = False)
        vis.add_geometry(pcd)
        vis.add_geometry(self.axes)
        ctr = vis.get_view_control()
        ctr.rotate(90.0, 90.0)
        opt = vis.get_render_option()
        opt.background_color = np.asarray([0, 0, 0])
        vis.poll_events()
        vis.update_renderer()
        image = vis.capture_screen_float_buffer()
        vis.destroy_window()

Unfortunately, this code prevents my geometries being contained in the image. I just get a black screen with the `visible = False' flag set.

I've build Open3D with the headless rendering extensions and can successfully run the headless_renderering.py example code. However, I can't quite see how this applies to my case. I think the relevant code in that demo is:

    custom_draw_geometry_with_camera_trajectory.vis = Visualizer()
    # --------------------------
    vis = custom_draw_geometry_with_camera_trajectory.vis
    vis.create_window()
    vis.add_geometry(pcd)
    vis.get_render_option().load_from_json("../../TestData/renderoption.json")
    vis.register_animation_callback(move_forward)
    vis.run()
    vis.destroy_window()

This doesn't appear much different to what I'm doing at present. Still a window is created...

EDIT: Another issue I am finding is that if I accidentally remove context away from the window generated by create_window(), (say by selecting my Chrome window), a screenshot is taken of my Chrome webpage rather than the 3D geometries. A solution for this is also required to make this useable.

syncle commented 5 years ago

The code looks fine to me. I just double-checked visible = False flag works as I expected and was able to save the images.

Two suggestions:

Can you run headless_rendering.py with visible = False? It works well my testing environment (Mac and Ubuntu 16.04).
Can you try with white background instead of black background? Point cloud may be not visible if the background is black and the rendered points are black as well.

martinakos commented 5 years ago

I'm have tried to create a window with vis.create_window(visible = False) and certainly no window appears. Later in my code I'm using vis.capture_screen_float_buffer(True) to get a snapshot of what I've rendered in the now unvisible window and display it within my own GUI. However, with visible=False, the snapshot captures whatever is on the screen at that moment. Any way of capturing what is being rendered in the unvisible window?

lelouedec commented 5 years ago

I have the same issues as mention above, the Headless example code create a window that stays open until the end of the code. And when setting visible as False, the capture is either black or the context selected (Any other application selected). Headless rendering is a good and important feature, especially for scientific purpose, when it takes some times to render on screen a scene and we just want a capture of it saved for later analysis. EDIT: For scenes that take long to render, the window render the context selected first while the scene is being processed

syncle commented 5 years ago

Another helpful comment would be to check if OpenGL uses OSMesa. @benjiebob @martinakos @lelouedec: can you share your environments? Are you using Ubuntu + OSMesa?

rupeshchandgude commented 5 years ago

vis = o3d.visualization.Visualizer() vis.create_window( visible = False) vis.add_geometry(pcd) ctr = vis.get_view_control() ctr.rotate(90.0, 90.0) opt = vis.get_render_option() opt.background_color = np.asarray([0, 0, 0]) vis.poll_events() vis.update_renderer() image = vis.capture_screen_float_buffer() vis.destroy_window()

I used above code. But the output I get is the current screen shot of screen. But if I use " capture_depth_float_buffer" instead of "capture_screen_float_buffer" , I can get depth image of pcd. Is anything going wrong? Thanks in advance.

bkanchan6 commented 4 years ago

I came across the same error. With vis.create_window(), it opens up the window and the content is saved as expected with out_image = vis.capture_screen_image(image_path, True). On the other hand, with vis.create_window(visible = False), the window does not open as expected. But it captures the screen content rather than content from vis window (which is not popped up). The saved image is something like screenshot of the desktop screen. I have confirmed that I am using Ubuntu + OSMesa. Headless example works fine as it is. But even in that script, if I modify the code to vis.create_window(visible = False), the similar behavior is noticed. The depth images stored are perfectly fine but the TestData/image folder has screenshots of desktop screen instead of the content from vis window.

Armin12 commented 4 years ago

Hi there, I am using the following code to extract the pinhole camera parameters after a rotation. How could I do the same thing without need to terminate the window every time? I tried visible = False as you mentioned above but it did not work for me. vis = o3.visualization.Visualizer() vis.create_window() vis.add_geometry(pcd) vis.get_view_control().rotate(150, 0.0) vis.run() param = vis.get_view_control().convert_to_pinhole_camera_parameters() vis.destroy_window()

Thanks, Armin

g2ragrawal commented 4 years ago

i dont know why this work but it works `def move_forward(vis):

    image = vis.capture_screen_float_buffer(False)
    image=np.asarray(image)
    plt.imsave("image1.png",image)
    vis.register_animation_callback(None)
    vis.destroy_window()
    return False`

vis=open3d.visualization.Visualizer() vis.create_window(visible=False) vis.add_geometry(pcd) ctr=vis.get_view_control() ctr.scale(1000) ctr.rotate(523/90,0) vis.register_animation_callback(move_forward) vis.run()

pvarvak commented 4 years ago

This should be resolved if following updated instructions from master on Headless Rendering (after #1695 was merged): https://github.com/intel-isl/Open3D/blob/master/docs/tutorial/Advanced/headless_rendering.rst When using OSMesa there is no connection to your display, there should not be any use of $DISPLAY, any creation of windows, or any capture from your display. If there are still issues please compile master and post output of bin/GLInfo and your environment.

Sampson-Lee commented 1 year ago

Many thanks to @syncle. It works for me. Here is my code.

vis.create_window(visible=False)
vis.add_geometry(point_cloud)
image = vis.capture_screen_float_buffer(do_render=True)
depth = vis.capture_depth_float_buffer(do_render=True)

Don't forget to set do_render as True.

isl-org / Open3D

Is it necessary to display the window from create_window()? #608