search

gselzer / pymmcore-plus-sandbox

Other
0 stars 0 forks source link

Enable RGB image displays #17

Open gselzer opened 1 month ago

gselzer commented 1 month ago

The current pathway for image data getting from MMCore to an NDViewer does not consider whether the data is RGB - it would be nice to support such datasets coming from RGB cameras.

From @marktsuchida:

For historical reasons, MMCore requires RGB images to be in 32-bit or 64-bit format, with an additional (unused) "alpha" channel. And because it was designed for use with Java (AWT at the time, I think), the components are in opposite (big-endian) order: ABGR, instead of RGBA. pymmcore produces such images, but pymmcore-plus converts them back to RGB (so the NumPy shape of the image is (height, width, 3), with the last axis being r-g-b).

But if I'm not mistaken, the methods to query this (separately from actually obtaining an image) have not (so far) been overridden in pymmcore-plus, so that you will have mmc.getBytesPerPixel() which will return 4 or 8 for color images, and mmc.getNumberOfComponents() which will return 4 for color images.

On the other hand, @tlambert03 recently added a metadata schema for images produced by the MDAEngine. I suggested that we should ignore MMCore quirks here and design for the future, so the ImageInfo contains a pixel_format that will be something like RGB8, RGB16, or RGB12 (the last example meaning 16-bit format containing values 0-4095). ImageInfo also contains a NumPy-like dtype, and together these fields are quite nice for determining the image format.

We could perhaps propose that CMMCorePlus gain a method getPixelFormat() that returns the same string value that the metadata pixel_format field uses (these are compatible with the industry standard GenICam Pixel Format Naming Convention (PDF)), for convenience when acquiring images outside of the MDAEngine.

tlambert03 commented 1 month ago

I don't think it needs all this complexity... vispy and pygfx already naturally support RGB images as numpy arrays in shape (Y, X, 3/4)... and if you call CMMCorePlus.getImage() with an RGB camera, pymmcore-plus takes care of rearranging it for you and you will get an array of (Y,X,3)...
ndv currently just assumes of single-channel data (todo here: https://github.com/pyapp-kit/ndv/issues/20)), this line here might need a slight adjustment: https://github.com/tlambert03/ndv/blob/4e0c5fed7bef765c52efaac9ec4a8a72eb9f7f8d/src/ndv/viewer/_backends/_vispy.py#L138 and the slicing will need an adjustment too. But I would just start with a simple 2d image case and see where ndv breaks

gselzer commented 1 week ago

This should be fixed with pyapp-kit/ndv#41, although it would be good to test with a camera before closing!