search

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.
https://rerun.io/
Apache License 2.0
6.26k stars 292 forks source link

API / dataloader for logging a video file directly #6532

Closed jleibs closed 14 hours ago

jleibs commented 3 months ago

It is common for users to have video files they want to insert into Rerun.

Eventually, we would like to support video compression directly in the viewer (https://github.com/rerun-io/rerun/issues/5815), but doing so in a cross-platform way (including the browser) presents several challenges.

However, at present, we require users to decode video themselves and then log the individual image frames to Rerun. This is challenging to do performantly, generally requiring dependencies on external libraries, and often times multi-threaded/multiprocessing approaches to help with throughput.

It would be nice to offer a user-space API that allowed users to just directly submit a file-path or video-encoded bytestream and let Rerun take care of converting to a stream of images.

Although our data-loader APIs make it possible to do this as a plugin, it might be worth offering a dedicated API for this. In particular, a dedicated API would allow us to be explicit about how frame-ids map to time-points, which is not something we currently have the ability to do through data-loaders.

This is a situation where it would likely make sense to do the implementation in Rust once and then offer bindings through to the other languages.

emilk commented 3 weeks ago
jleibs commented 1 week ago

Needs to produce video frame references