New API to request userspace-callbacks from interaction events in visualizer

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.

https://rerun.io/

Apache License 2.0

5.77k stars 260 forks source link

New API to request userspace-callbacks from interaction events in visualizer #2691

Open jleibs opened 1 year ago

jleibs commented 1 year ago

A recurring request from users is a mechanism for user-interactions such as clicking to generate a callback in user-space code.

Some examples:

Click on a point and receive the timepoint + entity-path + instance-id
Show an arbitrary list in a UI element and receive an event from the selection

Alternatives:

Until this feature is implemented, to accomplish something like this a user would need to create their own custom space-view: https://github.com/rerun-io/rerun/pull/2328 and then send data over a custom out-of-band backchannel, completely independent of Rerun.

TODO:

More details on roadmap for this feature

jaidevshriram commented 11 months ago

Hey! I'd love this feature too. I went through the mentioned PR but it looks like it's meant for Rust and not Python. If there is a pythonic way to create custom space views, I'd appreciate any help. Thanks!

Wumpf commented 11 months ago

@jaidevshriram yes, the linked PR is sadly Rust only and this is a very heavy-handed way of extending the viewer. This ticket is indeed all about making smaller adjustments from all languages in a more data-driven way :) Unfortunately, this is a bit further down the line; we're first going to look into more basic control of the viewer layout from code (if you see us talk about blueprint in tickets and on discord that's that)

karanchahal commented 3 months ago

Hello,

If there is a way to expose a javascript library that will enable usage of rerun specific things with custom UI components, that would significantly help in using this tool for custom tools on top of the renderer/viewer etc.

alik-git commented 2 months ago

Just echoing the need for this feature on my end as well. I'm making a 3D mapping system for robotics that allows you to search for objects in the scene via natural language, and this would help a lot.

See: https://github.com/concept-graphs/concept-graphs

My specific need is: Having something like a text input box with a labelled "search" button, that when filled out and pressed would trigger a callback to our code that performs the search and opens a new view with the desired object, or recolors all the objects in the scene by how relevant they are to the search query.

This would really help people migrate more easily from other options to rerun. Thanks!! Rerun is awesome.

Erol444 commented 3 weeks ago

+1 on user inputs, like buttons/text inputs/sliders (perhaps API could be similar to streamlit one?). OpenCV provides sliders, so I use opencv just for that, and then rerun for visualziation.

bokorn-bdaii commented 2 weeks ago

+1 this with python interfaces for mouse over and selection in 2D views (expanding graph nodes) as well as keyboard callbacks.