how to use animation callbacks with o3d.visualization.draw()?

[cwreynolds]

Checklist

• [X] I have searched for similar issues.
• [X] For Python issues, I have tested with the latest development wheel.
• [X] I have checked the release documentation and the latest documentation (for master branch).

My Question

I am trying to structure a visualization loop for my application. All the various draw...() functions seem to support different subsets of features. I need an animation callback, and key callbacks, and to specify a camera view (eg with lookat, eye, up).

Then in this 2020 comment I learned about open3d.visualization.draw() which appears to be a generalization over all the visualization properties. Yay!

The doc for open3d.visualization.draw() is extremely terse, just a list of the allowable parameters. As mentioned in that comment, the source adds some detail, but not for example: how do I use an animation callback? While there are two seemingly relevant parameters, I could not get them to work. For example, here is a callback that recolors a mesh:

import numpy as np
import open3d as o3d

mesh = o3d.geometry.TriangleMesh.create_octahedron()

def recolor(vis):
    mesh.paint_uniform_color(np.random.rand(3))
    vis.update_geometry(mesh)
    return False

It works using draw_geometries_with_animation_callback():

o3d.visualization.draw_geometries_with_animation_callback([mesh], recolor)

but neither of those draw() parameters mentioning “animation” seem to do anything:

o3d.visualization.draw([mesh], on_animation_frame=recolor)
o3d.visualization.draw([mesh], on_animation_tick=recolor)

Good chance that my callback is incorrect, but where can I learn how to do it correctly?

[cwreynolds]

Note that a web search for [open3d "on_animation_frame"] returns just 5 results, none with any information about how to use this feature.

[cwreynolds]

I would love to hear from anyone who knows how this is supposed to work.

After reading Python and c++ sources, and various experiments, I got a little further with using this apparently undocumented feature. I still do not know how to make it work as I want: to be invoked once each render, modifying the Geometry objects registered with the Visualizer.

I discovered that the on_animation_frame and on_animation_tick keywords are related to an item at the bottom of the visualizer window’s UI. And that item is not even visible unless I provide suitable values for the animation_time_step and animation_duration keywords to open3d.visualization.draw(). Plus the user needs to interactively click on [Play] in the lower right corner:

The code below opens a window, and when I click [Play], my cb_test() function gets called (every 1/60 second?) and logs to the shell. Unfortunately, as soon as animation begins, my mesh disappears:

def test_animation_callback():
    oct = o3d.geometry.TriangleMesh.create_octahedron()
    def cb_test(vis, time):
        print('in cb_test, time =', time)
        oct.paint_uniform_color(np.random.rand(3))
    o3d.visualization.draw({'name': 'oct', 'geometry': oct},
                           on_animation_frame = cb_test,
                           animation_time_step = 1 / 60,
                           animation_duration = 1000000,
                           show_ui=True)

Prints:

in cb_test, time = 0.0
in cb_test, time = 0.016666666666666666
in cb_test, time = 0.01666666753590107
in cb_test, time = 0.03333333420256773
in cb_test, time = 0.03333333507180214
in cb_test, time = 0.0500000017384688
in cb_test, time = 0.05000000074505806
in cb_test, time = 0.06666666741172472
in cb_test, time = 0.06666667014360428
in cb_test, time = 0.08333333681027094
in cb_test, time = 0.0833333358168602
in cb_test, time = 0.10000000248352686
in cb_test, time = 0.10000000149011612
in cb_test, time = 0.10000000149011612
...

That log suggests to me that cb_test() is actually being called twice per animation frame — but who knows?

Based on some hint in some error message (maybe when I tried to update_geometry() on a tensor Mesh?) I thought I would remove, then add back my mesh to uncache the old coloring:

def test_animation_callback():
    oct = o3d.geometry.TriangleMesh.create_octahedron()
    def cb_test(vis, time):
        print('in cb_test, time =', time)
        oct.paint_uniform_color(np.random.rand(3))
        vis.remove_geometry('oct')
        vis.add_geometry('oct', oct)
    o3d.visualization.draw({'name': 'oct', 'geometry': oct},
                           on_animation_frame = cb_test,
                           animation_time_step = 1 / 60,
                           animation_duration = 1000000,
                           show_ui=True)

On clicking [Play] this did a single recoloring, then appeared to hang, or something. It logged two calls at time 0, then a warning the mesh was no longer in the scene, perhaps because it was removed twice. The bad news is the callback is either hung or no longer getting invoked by the Visualizer. The good news is that the recolored octahedron is still live in the window, I can mouse-move the view.

in cb_test, time = 0.0
in cb_test, time = 0.0
[Open3D WARNING] Geometry oct is not in the scene graph

[EwingKang]

(Not sure why it's not linked automatically by Github) There's a related discussion: Many O3DVisualizer functions cause crashes/hangs if called during OnAnimationTick/ OnAnimationFrame(https://github.com/isl-org/Open3D/discussions/6549)

[TneitaP]

I had the same problem with the geometry of the update object disappearing. The current API is indeed very illogical and hierarchically confusing. Nevertheless, according to https://github.com/isl-org/Open3D/issues/2869, the following code can be considered as a temporary solution:

import numpy as np
import open3d as o3d
import open3d.visualization as o3d_viz

if __name__ == "__main__":

    # sphere_o3d = o3d.geometry.TriangleMesh.create_sphere(radius=1.2)
    box_o3d = o3d.geometry.TriangleMesh.create_box(create_uv_map=True)
    box_o3d.compute_vertex_normals()

    frame_o3d = o3d.geometry.TriangleMesh.create_coordinate_frame(size=1.5)

    mat = o3d_viz.rendering.MaterialRecord()
    mat.shader = 'defaultLit'
    mat.albedo_img = o3d.io.read_image('data/uv1.png')

    def animate_geom(vis, time):
        # https://www.open3d.org/docs/latest/python_api/open3d.visualization.rendering.Scene.html
        print('in cb_test, time =', time)
        # R = np.eye(4)
        R = box_o3d.get_rotation_matrix_from_xyz((0, 0, np.pi / 12 * time))
        box_o3d.rotate(R) # center=(0.5, 0.5, 0.5)
        # vis.scene.set_geometry_transform('box', R)
        vis.scene.clear_geometry() # for all geoms
        # vis.scene.remove_geometry('box')
        vis.scene.add_geometry('box', box_o3d, mat)
        vis.scene.add_geometry('frame', frame_o3d, mat)

    def animate_material(vis, time):
        # https://www.open3d.org/docs/latest/python_api/open3d.visualization.rendering.Open3DScene.html
        print('in cb_test, time =', time)
        R = box_o3d.get_rotation_matrix_from_xyz((0, 0, np.pi / 12))
        box_o3d.rotate(R) # center=(0.5, 0.5, 0.5)

        # vis.scene.clear_geometry() # for all geoms
        vis.scene.update_material

    o3d_viz.draw([
        {'name': 'box', 'geometry': box_o3d, 'material': mat}, 
        {'name': 'frame', 'geometry': frame_o3d}, 
        ], 
        eye = np.array([0,0,2]),
        on_animation_frame=animate_color,
        animation_time_step = 1/30,
        animation_duration = 1000000,
        show_ui=True,
    )