Open emilk opened 5 months ago
Making video frame seeking really fast would be huge. My use case is hours of video and telemetry recordings from UAVs and I would like to dump all of them to rerun and then step frame by frame back and forward somewhere in the middle of the video to study frames and corresponding telemetry.
Goals
Support some forms of “bigger-than-RAM” recordings, as soon as possible
Background
Small-index vs Big-index
Table index: row ids and time points.
Does the table index fit in RAM?
Hypothesis: most “bigger-than-RAM” problems have smallish indices.
Big index
Example: 100GB of scalar plots
We need a hierarchical index file on disk, with seeking, and have store-subscribers that are aware of this, etc. Difficult!
Small index
Example: thousands of uncompressed 4k images, or big point clouds, meshes, …
We “just” need to figure out how load blobs from disk on-demand. Easier!
Promises: a solution to small-index
We replace large blobs with promises, that refer to the external data.
A promise could be a file path with optional byte offset, a URL, …
When a query results in a Promise, we (try) to resolve it.
Example: we go through a huge MCAP file and log it to Rerun, but replace big blobs by a Promise referring to a byte-offset in the MCAP.
User stories
rr.log(”image”, rr.Image(data=rr.Promise.file_path(”foo.jpg”)))
file://recording.vrs?stream=video&time=42
Design
A
Promise
is a datatype, which can be used for any component. So acomponent.Point3D
can be represented bydatatype.Promise
A promise contains a single URI string.A promise resolves to some IPC Arrow data (or an error, or pending).
The promise is resolved late, after primary caches, close to the UI/visualizer.
MVP
Steps
PromiseCache
returningComponentResult<'a, T>
entity_iterator
should eitherMaybePromise<T>
for each component (leaving it to the user to resolve)ComponentResult<'a, T>
for each componentDataCell
[file://…?bytes=…](file://)
rerun index huge.rrd > indexed.rrd
PromiseCache
Post-MVP
Latency-aware
ComponentResult
in visualizersPromise resolvers
SDK-aware
Each of these adds additional abilities:
rr.log("mypoints", rr.Promise(Position3D.name, uri))
Option<Vec<Position3D>>
withMaybePromise<Vec<Position3D>>
isinstance
Collection
type