Closed trueb2 closed 6 months ago
rr.Scalar
isn't really designed for large scalar series, rather it is designed for real-time use cases that are mostly concerned with the last few minutes of data (using the "Visible time range" feature).
The reason for this is that rr.Scalar
has access to all the fancy time-related features in Rerun, and those features require a lot of extra memory and CPU overhead to be made possible.
We're always working on improving the performance of rr.Scalar
, but it fundamentally isn't compatible with large series with millions of points.
We do have an open proposal for a ScalarChart
though, which would be a more feature-limited version of Scalar
, with basically no memory nor CPU overhead (please add a :+1: if you're interested in something like this, it helps us prioritize!):
Thanks for the insight! I saw an issue regarding audio, but many wearable sensor data streams like biopotential and accelerometer are sub-10kHz and have few columns.
I am interested in streaming data for multimodal research streams similar to the linked issue. The common factor is typically that data was sent over BLE, so data streams are typically transferred at <400kbps. In the future, I hope that data from multiple sensors could be logged (using a python or rust script) to rerun for easy visualization.
If you would like some more data please let me know. Thanks for all your work on this project!
Thanks for the report!
We've summarized this in a tracking issue now, with some concrete issues for moving forward that we'll start working on soon:
Rerun looks great! I have many hours of 1+ kHz IMU data to analyze, so I am hoping rerun can help with that. I know it is early days of support for the time series visualizations support for kHz data, but I saw https://www.rerun.io/blog/fast-plots and figured it would be good to provide some more data.
To Reproduce Steps to reproduce the behavior:
cargo install rerun
✅rerun
✅cargo run --release
✅Expected behavior I would expect memory usage 1-10x greater than the accelerometer in the data frame. Here I had 5M rows of ts, x, y, z (32 bytes total), so I expected between 150MiB and 1.5GiB.
Screenshots
Desktop (please complete the following information):
Rerun version
rerun-cli 0.15.0 [rustc 1.77.1 (7cf61ebde 2024-03-27), LLVM 17.0.6] aarch64-apple-darwin
Additional context Here is the code for the streaming. I can provide larger or smaller IMU accelerometer data at 416, 833, 1.6k, 3.3k, or 6.6kHz if desired. To bog down the UI, I used about ~25min of data or about 15MiB of parquet.
The dependencies from
Cargo.toml
The streaming code from
main.rs
The
stream-0x30.pq
(in a .zip) stream-0x30.pq.zip