New API to request userspace-callbacks from interaction events in visualizer

[jleibs]

A recurring request from users is a mechanism for user-interactions such as clicking to generate a callback in user-space code.

Some examples:

• Click on a point and receive the timepoint + entity-path + instance-id
• Show an arbitrary list in a UI element and receive an event from the selection

Alternatives:

• Until this feature is implemented, to accomplish something like this a user would need to create their own custom space-view: https://github.com/rerun-io/rerun/pull/2328 and then send data over a custom out-of-band backchannel, completely independent of Rerun.

TODO:

• More details on roadmap for this feature

[jaidevshriram]

Hey! I'd love this feature too. I went through the mentioned PR but it looks like it's meant for Rust and not Python. If there is a pythonic way to create custom space views, I'd appreciate any help. Thanks!

[Wumpf]

@jaidevshriram yes, the linked PR is sadly Rust only and this is a very heavy-handed way of extending the viewer. This ticket is indeed all about making smaller adjustments from all languages in a more data-driven way :) Unfortunately, this is a bit further down the line; we're first going to look into more basic control of the viewer layout from code (if you see us talk about blueprint in tickets and on discord that's that)

[karanchahal]

Hello,

If there is a way to expose a javascript library that will enable usage of rerun specific things with custom UI components, that would significantly help in using this tool for custom tools on top of the renderer/viewer etc.

[alik-git]

Just echoing the need for this feature on my end as well. I'm making a 3D mapping system for robotics that allows you to search for objects in the scene via natural language, and this would help a lot.

See: https://github.com/concept-graphs/concept-graphs

My specific need is: Having something like a text input box with a labelled "search" button, that when filled out and pressed would trigger a callback to our code that performs the search and opens a new view with the desired object, or recolors all the objects in the scene by how relevant they are to the search query.

This would really help people migrate more easily from other options to rerun. Thanks!! Rerun is awesome.

[Erol444]

+1 on user inputs, like buttons/text inputs/sliders (perhaps API could be similar to streamlit one?). OpenCV provides sliders, so I use opencv just for that, and then rerun for visualziation.

[bokorn-bdaii]

+1 this with python interfaces for mouse over and selection in 2D views (expanding graph nodes) as well as keyboard callbacks.

[Famok]

I would love that feature for annotating my datasets