search

rerun-io / rerun

Visualize streams of multimodal data. Free, fast, easy to use, and simple to integrate. Built in Rust.
https://rerun.io/
Apache License 2.0
7.06k stars 353 forks source link

Add support for logging images in CHW format #5505

Open edgarriba opened 8 months ago

edgarriba commented 8 months ago

hi! i'm one of the maintainers of kornia and we are exploring to adopt more rerun as default to visualize.

Today, mainly kornia images are represented by torch tensors (some times batched) with a memory layout CxHxW. We noticed that rerun expected mainly image data in last channel order fashion HxWxC.

In kornia we have some functionality to easily handle such situation via tensor_to_image.

import kornia as K
import rerun as rr

rr.init("kornia_viz", spawn=True)

img = K.io.load_image(
     "image_path.jpeg",
    K.io.ImageLoadType.RGB32,
)

rr.log("image", rr.Image(K.utils.tensor_to_image(img)))

However, would be great if we somehow support that feature by, either adopting that kornia function, or improve the rerun Image type so that the user can customize easily the data layout. As ideas:

# 1 adopt tensor_to_image
rr.log("image", rr.Image(img))

# 2 customize layout
rr.log("image", rr.Image(img, channels_order="first"))

Besides, @johnnv1 has recently proposed to use https://einops.rocks/ which we might adopt also to handle properly visualization in tiles.

nikolausWest commented 8 months ago

Thanks for this issue @edgarriba and exciting to hear that the Kornia team is considering deeper adoption. Do you think it matters to Kornia users whether we do the CHW -> HWC conversion on the SDK side or as late as possible in the viewer. If we do it SDK side, users won't be able to see what the original channel ordering was when inspecting the data in the viewer and any transformation cost will also happen in the same process as the SDK call. We generally strive to not transform users data but doing it SDK side can be a way to ship functionality faster, which can obviously be valuable too

Wumpf commented 1 month ago

note for posterity: we have now planar YUV formats, but everything is every specialized to the exact format type. I think it would be worth exploring planarity as an orthogonal concept. Counter point: that may work with fully planar and fully interleaved formats, but semi-planar formats are very hard to express like this