search

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.
https://rerun.io/
Apache License 2.0
6.38k stars 311 forks source link

2D view of multiple DepthImages doesn't update with time scroll #7679

Open eskjorg opened 2 days ago

eskjorg commented 2 days ago

Describe the bug 2D view with multiple DepthImages doesn't show the correct image when dragging the slider.

But hovering still works, showing the correct pixel for each layer. This indicates that the correct data is there.

To Reproduce

for i, depth_image in enumerate(depth_arr):
    rr.log(
        f"{phase}/depth_imgs/{i}",
        rr.DepthImage(depth_image, meter=1000.0, colormap=Colormap.Viridis),
    )

Then press play or drag slider through time.

Expected behavior The 2D view should show each corresponding image when scrolling.

Screenshots

This is bad:

https://github.com/user-attachments/assets/cc80572e-b043-4fff-8d20-5d69c33861af

This is good:

screen_shot

Desktop macOS Sonoma 14.5 (23F79)

Rerun version rerun-cli 0.18.2 [rustc 1.76.0 (07dca489a 2024-02-04), LLVM 17.0.6] aarch64-apple-darwin release-0.18.2 59ff15b, built 2024-08-29T13:55:42Z

Wumpf commented 2 days ago

Interesting find! So actually DepthImage is not supposed to support (transparent) layering at all: this is ever so slightly hinted by the fact that only Image and SegmentationImage come with an optional Opacity component.

This means that a depth image in front will always hide everything behind it. So unless you log new data to the front-most entity, you won't see anything.
Which in turn means that the actual bug here is unfortunately the thing you called out working: Multiple previews should not show the hover interaction is meant to be disabled for fully hidden things (just like a mesh being drawn in front of a point doesn't allow hovering the point). Note also that the default DrawOrder is stable, but undefined, so you can't be sure which image is in front unless you specify it yourself.

But obviously your intention here is to get information out of several images so I'm wondering what you would like this to look like. Can you elaborate a bit on that?

So far we considered it not meaningful to have transparent layering on depth images - I have a hard time understanding how to get meaningful information out of such a visualization. Typically in those cases you'd either keep logging to the same entity path (then at every point in time you have a different image being shown) or, if all of them should show at once, you'd create more Views via a blueprint.