imshow doesn't normalize the color range in RGB images

[michalkahle]

imshow doesn't normalize the color range in RGB images

Bug summary When single channel float image is passed to imshow it gets normalized to range [0,1] before display. This does not happen for RGB images.

Code for reproduction

img = np.arange(0, 0.6, .1).reshape(6,1)
plt.subplot(141)
plt.title('grayscale')
plt.imshow(img)
plt.colorbar()

img = np.arange(0, 0.6, .1).repeat(3).reshape(6,1,3)
plt.subplot(142)
plt.title('RGB')
plt.imshow(img)
plt.colorbar()

img = np.arange(0, 1.6, .2).reshape(8,1)
plt.subplot(143)
plt.title('grayscale')
plt.imshow(img)
plt.colorbar()

img = np.arange(0, 1.6, .2).repeat(3).reshape(8,1,3)
plt.subplot(144)
plt.title('RGB')
plt.imshow(img)
plt.colorbar()

Actual outcome Please also note how the colorbars are misleading.

Expected outcome I'd expect the RGB images to look the same as grayscale.

Real life example This is how I encountered this issue. What is the value in all three channels at [3,18]? It is actually around 1.2!

Matplotlib version: 2.0.2

Discussion The same issue: #5382 , on SO, and another SO.

In #5382 @tacaswell explains that and RGB image "goes through a different code path which do not get passed through the normalize/colormap framework". Colormapping of course makes no sense for color images but I believe that normalization should be consistently applied to RGB images.

I got used to matplotlib normalization of single channel float images and the need of using vmin and vmax to avoid this normalization and intuitively expected RGB images to be treated consistently. As I noticed subsequently the docstrigs say that "the value for each component of MxNx3 and MxNx4 float arrays should be in the range 0.0 to 1.0." But I believe that to simply overflow without raising error or warning is not correct.

[jklymak]

RGB means RGB. I don't think it should be normalized.

I think a case could be made that it shouldn't do a modulo past 1.0 or below 0.

[anntzer]

I think it's either error out ("Errors should never pass silently."), or clipping (because sometimes you may end up with values just a bit below 0 or a bit above 1 due to fp imprecision, so "practicality beats purity.").

See also #8854.

[WeatherGod]

Isn't the issue here that a colorbar is even being made in the first place for an RGB image? The colorbar code is trying to infer a norm and colormap when it just shouldn't be guessing here.

On Fri, Oct 13, 2017 at 2:06 PM, Antony Lee notifications@github.com wrote:

I think it's either error out ("Errors should never pass silently."), or clipping (because sometimes you may end up with values just a bit below 0 or a bit above 1 due to fp imprecision, so "practicality beats purity.").

See also #8854 https://github.com/matplotlib/matplotlib/issues/8854.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/matplotlib/matplotlib/issues/9391#issuecomment-336526696, or mute the thread https://github.com/notifications/unsubscribe-auth/AARy-C02T7RO_sbfgLWBFbTMsQaKUSxAks5sr6aXgaJpZM4P4h1K .

[anntzer]

that is definitely an issue too... :)

[tacaswell]

I am confused why it wraps...

Agree the colorbar should refuse to let it's self be created for an RGB(A) image.

[anntzer]

It wraps because ultimately everything is cast to uint8 (essentially (uint8_t)(val * 0xff)).

I also don't think it's release critical, the wraparound is also present in 1.5.0 and probably since ever.

Upon further thought I think I prefer erroring on such inputs, possibly with a message that points to np.clip.

[michalkahle]

It might have been unfortunate to label my data "RGB image". It would be better to call it three channel float images. uint8 RGB images should NOT be normalized. I think the viewer should do the most useful thing by default. As a user I often just want to quickly see what is in my data without much fuss. With arrays of floats my data can take the whole range. The clipping or modulo destroy my data. If I get an error about wrong range I'd be forced to normalize the data myself in which case I'd wonder why the viewer does not do this for me as it does with single channel image. If I wanted to be specific I'd use the norm, vmin and vmax parameters just as I do with single channel image. Simple. Consistent.

[jklymak]

What three-channel data do you look at that isn’t meant to be an RGB image and why do you look at it as an image?

[michalkahle]

@jklymak these are three repeated absorbance measurements in 384 well plate. I just wanted to quickly see the pattern on the plate as well as variability in the repeated measurements. I agree this is not typical use of 3 channel images.

[jklymak]

Cool. OTOH I’m sure you can appreciate that it is wrong to automatically normalize an arbitrary RGB image. If the whole image is dark, it should stay dark unless the user wants to explicitly lighten it.

[michalkahle]

I absolutely agree with that in case of integer images. But in case of say 16 bit per channel RGB images we want to show the whole range (not clip or wrap around). In case of float images we cannot show the whole range (the image would be almost always gray). Forcing the user to normalize the image to a range [0, 1] is a solution but I feel too restrictive one. Float images would be mostly used for image processing. While these are mostly single channel images, color images can be used too with tools like adapt_rgb. Please also note in the adapt_rgb examples they normalize rescale_intensity(1 - sobel_each(image)). I wondered why the 1 was needed. If you omit it the image gets normalized to [-1, 1] and the colors get wrapped around by imshow.

[WeatherGod]

I have a vague recollection that Pillow sometimes loads RGB images as floats instead of integers, so we have to accept that as input.

On Wed, Oct 18, 2017 at 7:24 AM, Michal Kahle notifications@github.com wrote:

I absolutely agree with that in case of integer images. But in case of say 16 bit per channel RGB images we want to show the whole range (not clip or wrap around). In case of float images we cannot show the whole range (the image would be almost always gray). Forcing the user to normalize the image to a range [0, 1] is a solution but I feel too restrictive one. Float images would be mostly used for image processing. While these are mostly single channel images, color images can be used too with tools like adapt_rgb http://scikit-image.org/docs/stable/auto_examples/color_exposure/plot_adapt_rgb.html#sphx-glr-auto-examples-color-exposure-plot-adapt-rgb-py. Please also note in the adapt_rgb examples they normalize rescale_intensity(1

• sobel_each(image)). I wondered why the 1 was needed. If you omit it the image gets normalized to [-1, 1] and the colors get wrapped around by imshow.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/matplotlib/matplotlib/issues/9391#issuecomment-337559828, or mute the thread https://github.com/notifications/unsubscribe-auth/AARy-O8kmEEcRlGRwJPEhc6gfLdpflhpks5std_4gaJpZM4P4h1K .

[jklymak]

imshow accepts floats between 0. and 1 or uint8. I agree that it shouldn't round floats to uint8 unless it has to for rendering. I agree it probably shouldn't wrap, and clipping between 0 and 1 makes sense to me, or returning an error. I think it should definitely not normalize arbitrary data between 0 and 1.

It'd probably be easy to add support for 16-bit integers given that a lot of images are now 16 bits. But I guess there is a bit of an issue telling between a dark 16-bit image and an 8-bit image if the user doesn't specifically cast to the right-size uint.

[Zac-HD]

tacaswell, I've recently been working on imshow in Xarray, where we concluded that the proper place for scaling and clipping fixes was upstream (ie here). My plan:

• [ ] Support norm or vmin and vmax arguments with RGB[A] images. Among other things, this makes it quite trivial to support high-bit-depth images - just pass vmin=0, vmax=2**12 for 12-bit images, for example. Rejected by consensus on pull request.
• [x] If data are outside the valid range after scaling - that is, [0 .. 255] for integer types and [0 .. 1] for floating types - log a warning and then clip to the valid range before casting, to clip instead of modulo. Without this any pixel could represent a much higher or lower value than it seems.
• [x] Open a new issue for "colorbar should refuse to let it's self be created for an RGB(A) image" - it's a logically distinct change and I don't have the time to do both right now.

What do people think? If this seems reasonable I'll write a draft tomorrow 😄

[anntzer]

Untagged from release critical as neither the wrapping and the inappropriate colorbar'ing are "new" bugs, and RGB normalization is another can of worms (see discussion followup in #10220).

[michalkahle]

@Zac-HD I like the proposal. Even without normalization the out-of-range warning and clipping instead of modulo would still be nice improvement.

[Zac-HD]

That's what I'm doing in the pull request!

(and normalisation will happen downstream in Xarray, which will cover most of my use at least 😉 )

[denis-bz]

In 3.4.2, @michalkahle's code (with subplot 143 = 142 as uint8) gives red colorbars. Is that expected ?

[jklymak]

Yes colorbars do not do anything sensible for rgba data.

[shcrela]

I realize it's an old (and unfortunately closed) issue, but I stumbled upon this discussion when searching for solution to my problem. Here are my two cents: I am trying to use plt.imshow to represent simultaneously the concentrations (values from 0 to 1) of three different components as one 2D "RGB" image (note the quotes). For completeness, the sum of the three concentrations is equal to 1 for each individual pixel. If I am to simply put each of the concentration maps in its' proper channel and plot the image as RGB, I will obtain a passable result. But there still remains the problem of the non-linearity of human eye perception. If I'm not mistaken, the common remedy would be the gamma-correction. This works well for single-channel images using the norm argument of imshow (like mpl.colors.PowerNorm(gamma=1/2.2) for example), but, alas, matplotlib doesn't seem to allow the normalization for multichannel images.

Some contributors (specifically @jklymak ) seem to consider that RGB images are not to be messed with. Why? What is your target audience? I can't think of a better way for representing multiple channels in one plot, other than this "hacking" of the RGB format. If someone does, I'd be more than happy to hear it. Or, if RGB(A) is in fact considered sacred, could we add a flag to signal that it's a multichannel array and not an rgb?

There is, of course, a possibility to re-normalize my data, as some suggest - but I don't want to change my data, the values have a precise meaning as they are, I just want to change the visual perception of my data by the human observer.

I hope this little message will make some of the decision-makers reconsider adding the possibility to normalize the data regardless of the input data type (multichannel or not).

All in all, thanks a lot for all the work you invested so far and for all your continuous efforts to make matplotlib even better!

I'll add here some of the relevant issues and comments:

18627

https://github.com/matplotlib/matplotlib/pull/10220#issuecomment-356633092 https://github.com/matplotlib/matplotlib/issues/9391#issuecomment-336833787 https://github.com/matplotlib/matplotlib/pull/10220#issuecomment-356813828

[jklymak]

Matplotlib shows the RGBA image as-is. There is nothing in Matplotlib that prevents the user from modifying a copy of their RGBA signal using gamma correction or whatever they want, and plotting that. Other packages, like scikit-image, provide many image manipulation tools.

[tacaswell]

Currently imshow can do colormapping on images of scalar data or handle RGB(A) data exactly as handed in. I completely agree with Jody here that the best course of action is to write your self a function that does what ever normalization you want and then do

ax.imshow(my_to_rgb_fuction(my_data))

We did have a GSOC project to add bi-variate color maps, which ale less than you want, but in the same direction: https://github.com/matplotlib/matplotlib/pull/8738 . I think this is still an interesting extension (mostly related to generating the 2D colorbar or wheel). If you @shcrela would like to pick this up and get it going again we would be happy to work with you.

[shcrela]

Thank you for your comments @tacaswell, I am well aware of the possiblity to modify my data (or the copy of my data) and then plot it. But in my honest opinion, if the problem is not in the data itself, but purely in the visual representation of that data - that should be matplotlib's concern. Other than that, if I modify the data, hovering over a pixel in an interactive backend would read non-sensical values (perhaps there is a workaround for this?) Other minor nuisances would be explaining and documenting what is now this modified data you're plotting - can become mind boggling, how to inverse-transform it after .get_array(), or not to forget to transform it before set_data().

Multi-variate colormap could be a solution, but it seems to me much more complicated compared to the existing norm tool.

I'm writing all this just to make you aware that there are some people out there who might need this fonctionality.

In the short run, as @mkcor mentionned, Plotly seems to provide the solution.

Cheers!

[michalkahle]

Slightly off-topic I wanted to thank the matplotlib maintainers for their hard work and especially for them to still being responsive and explaining their point of view. Even though they not always do what we want them to. :wink:

[tacaswell]

I'm writing all this just to make you aware that there are some people out there who might need this fonctionality.

We are well aware, I have personally relied on multi-variate color maps in my research in the past and had a GSOC student work on exactly this problem.

Multi-variate colormap could be a solution, but it seems to me much more complicated compared to the existing norm tool.

What you are asking for is a multi-variate color mapping where the channels are fully separable at the norm stage and then straight mapping the data channels to color channels.

Even though they not always do what we want them to.

Come join us! A vast majority of the development effort on Matplotlib is volunteer. As I said above, the multi-variate colormap work needs a champion to pick it back up, sort through the outstanding API concerns, and get it in. I do not have the bandwidth to lead that work, but would be very happy to support someone to do so.

[denis-bz]

@shcrela, "Plotly seems to provide the solution": could you post a link, or links to other examples of your use cases ?