3-channels 8-bit fluorescence images are read as RGB images

[Rdornier]

If you try to import images from this zenodo, the images are read as RGB images but they should be read as fluorescence images with 3 channels.

I also had this issue and I've tried to fix it by looking at the image histogram to decide if it is a fluorescence image or not.

[petebankhead]

What is the problem with this, and how can the images be identified as non-RGB?

QuPath often auto-converts 8-bit 3-channel images as RGB if the channel colors and names are consistent with being RGB. I'm not sure if the extension is making this conversion too aggressively, but we need to be careful not to do it too rarely - because if we guess wrong and fail to make an image RGB then it'd be impossible to set the image type to brightfield and apply color deconvolution.

[petebankhead]

Is the primary issue that the channel colors and names can't be set?

[NicoKiaru]

(sorry for the not so useful remark, but the primary issue here is that there doesn't seem to be a way to get the original pixel type from OMERO, or is there ?)

[petebankhead]

@NicoKiaru I'm not sure - it's definitely bad if we're making a non-8-bit image become 8-bit RGB.

If we're overriding any 'isRGB' flag then that's something that BioFormatsImageServer already does sometimes. I admit it's annoying when you have a fluorescence image and want to change channels/names, but it's more annoying when you've a brightfield image and can't analyze it at all... and the isRGB flag returns false for brightfield images often enough that we can't rely on it entirely.

And that's not easy to fix, because back in QuPath's early days I understood that Java is a bit more optimized for drawing (A)RGB packed int images in the viewer, and so wrote/optimized the color deconvolution code with that assumption - which helps us apply it very quickly in the viewer.

I guess my point is that, if fixing this, we'll need to be careful not to 'over-correct' and create trouble with brightfield images. I don't think we should use the histogram, since that's what the 'Auto-estimate image type' tries to do... and I've see it guess fluorescence for brightfield too.

[Rdornier]

@NicoKiaru The main issue for me is that there is no way to know if the image is RGB or not because OMERO reader doesn't provide this information (see this post)

Is the primary issue that the channel colors and names can't be set?

@petebankhead Yes, this is the main issue on QuPath side. There is also the fact that color deconvolution is applied and added many other pseudo-channels, which is confusing as the image is not RGB.

if we guess wrong and fail to make an image RGB then it'd be impossible to set the image type to brightfield and apply color deconvolution

Yes, but the contrary is also true, not as critical, but still annoying

[petebankhead]

There is also the fact that color deconvolution is applied and added many other pseudo-channels, which is confusing as the image is not RGB.

Color deconvolution isn't applied unless the image type is set to brightfield - and even them, only when requested. And the pseudo-channels should only really appear in the Brightness/Contrast dialog; they don't have to be used.

So I don't think these should be a serious issue, though I agree that the color/name issue can be very irritating.

Unfortunately we cannot use the isRGB flag, either from Bio-Formats or OMERO. It returns false too often for brightfield images where QuPath needs an RGB representation.

From what I remember, the rule in QuPath is/should be that an image where isRGB == false is automatically converted to RGB if all of these are true:

1. it has 3 channels, all uint8
2. the channel colors are red, green and blue (in that order)
3. the channel names are "Red", "Green" and "Blue"

Rules 2 and 3 aim to restrict the occurrence of converting fluorescence images wrongly - but relies upon the image metadata specifying channel colors and names.

(I'm writing this from the perspective of what we do with Bio-Formats, @Rylern knows much better than me what this extension does)

[Rylern]

Currently, this extension looks at the /webgateway/imgData// link, and defines the image to be RGB if (see here):

• The pixel type is "uint8".
• There are 3 channels.

I could add more conditions, like:

• Pixel range between 0 and 255.
• Colors of the channels to be "FF0000", "00FF00", "0000FF" (in order). However, the colors can be changed by the user, so I'm not sure if we should use this one.

I have an example of a RGB image with channel names "0", "1", and "2", so I don't think they are relevant to determine if the image is RGB.

[petebankhead]

I don't think the range test adds anything to the uint8 test.

I tentatively like the idea of checking the colors. At least, the fact that these are user-adjustable might not be a problem - because users will almost certainly not adjust them for a brightfield image, and actually it provides a workaround to the automatic version (i.e. change the colors on OMERO)... although we'd need to make sure nothing bad happens if the user changes the colors after the image has been added to a QuPath project.

[petebankhead]

@Rdornier and @NicoKiaru do you think this PR would be sufficient to workaround the issue here?

• https://github.com/qupath/qupath/pull/1659

It should:

• Avoid showing irrelevant channels for RGB image when the type is set to FLUORESCENCE within QuPath
• Enable RGB channels to be renamed

It doesn't enable RGB channels to be recolored... but that has the potential to be a lot more awkward and error-prone - so my goal with this PR was to shrink the problems with undesirable RGB conversion, while avoiding causing new ones.

[Rdornier]

@petebankhead Sorry for the delayed answer.

I've finally found the time to test it yesterday. So, the images are now read as fluorescence ones but... it also introduces a new channel 'original' Does it mean that a 3-channels 8-bit fluorescence image is now considered as a 4 channels image ?

It doesn't enable RGB channels to be recolored...

This is annoying, as it introduces a specific case in fluorescence images and raises the question 'why a 3-channels 8-bit fluorescence image should be considered differently as any other fluoresence images ?'. I understand the problems that can cause a wrong interpretation of an RGB image and why you would like to avoid them, but having an image, considered as a fluorescence one, without the possibility to change channel color is also something we want to avoid, as display colors are an important part of the visualisation of what we are interested to see.

[petebankhead]

So, the images are now read as fluorescence ones

No - at the point the image is read, there is no distinction between 'fluorescence' and 'non-fluorescence'.

Rather, there are only are two fundamentally different types of image: RGB, and non-RGB.

This distinction isn't unique to QuPath. It also exists in ImageJ: RGB uses a ColorProcessor, everything else doesn't.

And it exists in Java: RGB images usually store pixels as packed int arrays.

That is ultimately what is causing the issue here, and making it so awkward to work around: we need to decide on the representation before QuPath can set an ImageType. The ImageType is not a property of the ImageServer and it can be changed freely - but the ImageServer must be constant from the moment it is created for tile caching to work.

And if we do not have a packed-int (RGB) representation for a brightfield image, then color deconvolution is completely impossible without rewriting a whole lot of code, and potentially making error-checking and debugging vastly more complicated... and rendering less efficient.

Therefore, if there is a risk of making too many images RGB or too few, it is safer to err on the side of too many.

Changing this would be a fairly enormous change that there is no way we can do before v0.6.0, and probably not before v1.0.0.

This is annoying, as it introduces a specific case in fluorescence images and raises the question 'why a 3-channels 8-bit fluorescence image should be considered differently as any other fluoresence images ?'. I understand the problems that can cause a wrong interpretation of an RGB image and why you would like to avoid them, but having an image, considered as a fluorescence one, without the possibility to change channel color is also something we want to avoid, as display colors are an important part of the visualisation of what we are interested to see.

I understand, but note that exactly the same issue can exist in ImageJ when an image is read using a ColorProcessor.

I'm trying only to find an option that minimizes the annoyances with a feasible (limited) amount of effort on our side, as they cannot be eliminated completely any time soon. I don't think QuPath's behavior in this regard has changed substantially in the 8 years it has been available.

Does it mean that a 3-channels 8-bit fluorescence image is now considered as a 4 channels image ?

No. You can easily check this in a script, e.g. with

getCurrentServer().getMetadata().getChannels()

We might be able to remove Original as an option from the B/C dialog if it's problematic, I hadn't previously noticed it (partly because the overwhelming majority of fluorescence images I have aren't treated as RGB anyway).

[petebankhead]

Despite writing all that, there is a chance that a small change in QuPath might make changing the display colors of RGB images possible through the UI. They'd still be RGB though.

It feels a little scary since it deviates from what was originally intended, but I'll try it and make a PR if I can't find any obvious bugs...

[NicoKiaru]

Sorry, I do not catch everything that's relevant to the thread, it has lots of implications. But since you mention:

we need to decide on the representation before QuPath can set an ImageType

Can we force a non-rgb type with an extra argument like this:

Or is there absolutely no way to do that ?

[petebankhead]

Can we force a non-rgb type with an extra argument like this:

Maybe, sort of... we'd either need to support it in all ImageServer implementations, or wrap up an ImageServer when this is requested - potentially splitting the channels dynamically (it's not uncommon for a fluorescence image to be stored in a way that really is inherently RGB). Neither option feels ideal...

---

I've made the PR proposing a different approach at https://github.com/qupath/qupath/pull/1669

It does not attempt to change at the image-reading level - because I think that would lead to an unpleasant whac-a-mole scenario where there would always be some errors, and a high likelihood of incompatibility across QuPath versions.

Instead, I tried to tackle it within the ImageDisplay class and also within the Brightness/Contrast dialog.

Packed int RGB images remain RGB, including the (small number) that QuPath auto-converts to RGB when reading based on channel names & colors.

Basically, it allows the channel names and colors to be set, and these are used in the viewer specifically whenever the image type is Fluorescence. They are also used elsewhere when channel names are required, e.g. measurements, cell detection, or pixel classification.

I think it's working, but I've listed some caveats and concerns in the PR.

If you have any feedback / suggestions / complaints, I'd appreciate it if you could let me know as soon as possible. Things are super busy heading up to I2K, and this is a change that would benefit from thorough testing - so I'd like to merge it quickly, if at all.

[Rdornier]

Ok, I've tested the new PR. I works fine for me. RGB images, with type fluorescence, are considered like non-RGB images and both name and color can be changed. So, that's good 👍

Thanks for fixing this annoying issue !

[petebankhead]

Excellent, thanks for checking, glad it works. I'll merge it soon.

Please do let me me know if you find any serious weirdness - I could generate a bit with strange channel names and pixel classification, but it's very possible that similar weirdness could already have been achieved by messing with channel names anyway.

I think that means @Rylern doesn't need to worry about this with OMERO, and we can close this issue?

[Rdornier]

Yes, sure, the issue can be closed now.