Incorrect color conversion between IplImage and BufferedImage

[GoogleCodeExporter]

It seems that conversion of color images is buggy if the BufferedImage type is 
TYPE_INT_RGB or TYPE_INT_ARGB. The order of the components is reversed.

BufferedImage img=new BufferedImage(1, 1, BufferedImage.TYPE_INT_RGB);
img.setRGB(0, 0, new Color(10, 20, 30).getRGB());

IplImage ipl=IplImage.createFrom(img);

// these two lines are to avoid getting back the cached input BufferedImage, 
'img'
ipl=IplImage.createCompatible(ipl);
ipl.copyFrom(img, 1, new Rectangle(0, 0, 1, 1));

BufferedImage img2=ipl.getBufferedImage();

System.out.println(new Color(img2.getRGB(0, 0)));

 --> prints "java.awt.Color[r=30,g=20,b=10]"

Works correctly with TYPE_INT_BGR, TYPE_3BYTE_BGR, TYPE_4BYTE_ABGR.

Using version 20120218.

Original issue reported on code.google.com by kazocs...@gmail.com on 1 Mar 2012 at 3:31

[GoogleCodeExporter]

IplImage does not store channel order, so when you call something like 
IplImage.getBufferedImage() it assumes you have BGR data for 3-channels, since 
this is what OpenCV functions usually expect, and RGBA data for 4-channels, 
since this is what OpenCL requires. If you want a TYPE_INT_RGB or TYPE_INT_ARGB 
BufferedImage, you should *not* use IplImage.getBufferedImage(), but manually 
create a BufferedImage and use IplImage.copyTo() on it, but beware that 
TYPE_INT_RGB maps to a 4-channel IplImage. 

Did that help you solve these issues? Or is there something more to them?

Original comment by samuel.a...@gmail.com on 1 Mar 2012 at 11:42

[GoogleCodeExporter]

I believe I was a bit too concise. It's not that I want an RGB BufferedImage; 
any type I get is good. And in fact, IplImage.getBufferedImage() gives me 
TYPE_4BYTE_ABGR. The problem is that the order of the components gets reversed 
somewhere in the process.

If I ventured a guess (informed guess by looking at the source code), I'd say 
that IplImage.createFrom (more concretely IplImage.copyFrom) ignores the 
component order, and treats RGB and BGR the same way. So the IplImage I get 
from an input RGB type has the components in the wrong order.

My assumption is that the IplImage's conversion functions to/from BufferedImage 
should take care of all the details of the native IplImage's inner workings, so 
that I don't have to know, for example, the component order used by IplImage. 
But I might be misunderstanding the purpose of these functions.

Original comment by kazocs...@gmail.com on 2 Mar 2012 at 12:13

[GoogleCodeExporter]

The "inner workings" of IplImage have no "component order". So my conversion 
code maps 3 channels to BGR, and 4 channel maps to RGBA. If this is not to your 
liking, you are free to create any BufferedImage you want. I still do not 
understand what exactly the issue is ... ? Could you please explain what you 
are trying to accomplish here?

Original comment by samuel.a...@gmail.com on 2 Mar 2012 at 12:18

[GoogleCodeExporter]

I apologize, I wrote these too late in the day. Here is a more "rested" version.

I'm trying to use IplImage.createFrom to convert my input BufferedImage to a 
format suitable for OpenCV, do some processing on the IplImage, and then 
convert the result back using IplImage.getBufferedImage.

However, I've noticed that for color images these two operations mess up the 
colors in some cases. Concretely: assume that the processing I do is simple 
cloning using cvCloneImage). Then:

1. If the original BufferedImage is TYPE_INT_RGB, then the colors of the 
IplImage are correct (using opencv_highgui.cvShowImage to check), but the 
BufferedImage I get back is wrong (it has its color values in the incorrect 
channels).

2. If the original BufferedImage is TYPE_INT_BGR, then the colors of the 
IplImage appear incorrectly (the components are in the wrong color channels), 
and the BufferedImage that I get back has the correct colors (meaning that the 
same as the original input, not the IplImage), but it has an alpha value of 0 
in all pixels, so it becomes "invisible".

Am I totally misunderstanding what the functions IplImage.createFrom and 
IplImage.getBufferedImage are supposed to do?

Original comment by kazocs...@gmail.com on 2 Mar 2012 at 9:32

[GoogleCodeExporter]

Ok, so your main image format is BufferedImage and you depend on its features 
that are not available in IplImage, right? In that case, you should use only 
IplImage.createFrom() and NOT getBufferedImage(). Does this answer your 
question?

Original comment by samuel.a...@gmail.com on 4 Mar 2012 at 5:43

[GoogleCodeExporter]

It is correct that I use BufferedImage everywhere in my program, for example I 
want to display the images in my Swing GUI. Both the input and the result. So I 
want to convert to IplImage right before the OpenCV calls, and convert the 
result back to BufferedImage right after.

If getBufferedImage() is not the function to use for the latter, than my 
question is: is there a way I can get a BufferedImage which contains the same 
image as an existing Iplimage? (And I would also be curious what 
getBufferedImage() actually does.)

As long as you're saying that IplImage.createFrom() is okey to use, I have a 
bug report. For most BufferedImage types, it doesn't work correctly. I have 
tested by creating a BufferedImage, filling it with the red color (200,0,0), 
and using IplImage.createFrom(). Then I examined the data for the first pixel. 
(The first nChannels() bytes of getByteBuffer()). I also displayed the image 
using cvShowImage() to see how OpenCV interprets these color channels. Here's 
what I've found.

  BufferedImage type   |   IplImage pixel data   |   OpenCV displays it as
------------------------------------------------------------------------
1. TYPE_3BYTE_BGR             (0 0 200)                    red
2. TYPE_INT_BGR               (200 0 0 0)                  blue
3. TYPE_4BYTE_ABGR            (200 0 0 255)                blue
4. TYPE_4BYTE_ABGR_PRE        (200 0 0 255)                blue
5. TYPE_INT_RGB               (0 0 200 255)                red
6. TYPE_INT_ARGB              (0 0 200 255)                red
7. TYPE_INT_ARGB_PRE          (0 0 200 255)                red

My first observation is that OpenCV seems to interpret 3-channel images as BGR 
(like you said) and 4-channel images as BGRA (unlike what you said). Now, 
assuming that I trust cvShowImage and accept that BGRA is the correct order, 
then createFrom() produces incorrect order for cases 2, 3, and 4.

I also have to note that for TYPE_INT_BGR, the IplImage pixels will all have an 
incorrect alpha value, 0 instead of 255.

Additionally, there can be problems when I have an input color with alpha<255 
(interestingly, even if the input image has no transparency). I attach my test 
code for reference, which does the conversion first using createFrom, and then 
using getBufferedImage, displaying the color data at each step.

Original comment by kazocs...@gmail.com on 5 Mar 2012 at 10:40

Attachments:

• Test.java

[GoogleCodeExporter]

Since you say that you already have a BufferedImage, simply call 
IplImage.copyTo() on it. Or does IplImage.copyTo() not work as well?

To understand what IplImage.getBufferedImage() does, you may refer to the 
source code:
http://code.google.com/p/javacv/source/browse/trunk/javacv/src/com/googlecode/ja
vacv/cpp/opencv_core.java#980

TYPE_4BYTE_ABGR is bit special as I flip the channels in all cases, I should do 
something about that... (Changing the status to Started for this one.) The 
other types work perfectly correctly. How do you feel they should work?

On my system your Test program outputs:
    Input color (RGB): (235 1 2)
    Raw input data: (235 1 2)
    Native color: (2 1 235)
    Output color (RGB): (235 1 2)
Looks perfectly fine to me. How do you feel this is wrong?

BTW, TYPE_INT_BGR and TYPE_INT_RGB have no alpha channels, so it's perfectly 
normal that you may not be able to set values for the nonexistent alpha 
channel, and that Java may decide to set all the unused bits to either 0 or 255 
or something else, depending on the implementations and what not. If your 
applications relies on a specific values for alpha, then do not use 
TYPE_INT_BGR or TYPE_INT_RGB.

Original comment by samuel.a...@gmail.com on 7 Mar 2012 at 5:44

• Changed state: Started

[GoogleCodeExporter]

Yes, this output is in fact perfectly fine. You can select the BufferedImage 
type to use at the beginning of the main function, and I left TYPE_3BYTE_BGR 
selected when I uploaded (the only type that I think is not wrong; a bit silly 
of me). If you for example select TYPE_INT_BGR instead, you get the following:

    Input color (RGB): (235 1 2)
    Raw input data: (235 1 2)
    Native color: (235 1 2 0)
    Output color (RGBA): (235 1 2 0)

This native color not correct: it is blue instead of red. And I question why 
this IplImage has an alpha channel, since the BufferedImage had none, it was 
opaque. But even if you create a 4-channel IplImage, its alpha should be 255, 
and definitely not zero. So this native color should either be the same as in 
the TYPE_3BYTE_BGR case: (2 1 235), or have an alpha=255: (2 1 235 255).

Original comment by kazocs...@gmail.com on 8 Mar 2012 at 10:14

[GoogleCodeExporter]

The order of TYPE_INT_BGR on a little-endian machine is RGBx (which has no 
alpha channel, but each pixel still has 4 bytes), so nothing strange about that 
output, unless you are running this on some big-endian machine...

As I said, if you need a specific value for the (non)alpha channel, then you 
will have to set it yourself. Java apparently sets it to zero. If you need 255, 
then set it to 255.

Original comment by samuel.a...@gmail.com on 8 Mar 2012 at 7:19

[GoogleCodeExporter]

It is apparent that I don't quite understand you and I think that I know why. 
I'll try to describe how I believe you intend IplImage.createFrom (and 
IplImage.copyFrom) to work.

It basically ignores the meaning of the samples in a pixel completely, and just 
copies the sequence of samples in the order it finds them. So the very first 
sample in the BufferedImage will end up in the very first sample of the 
IplImage, regardless whether this sample is a red, green, blue, or alpha 
component according to the SampleModel of the BufferedImage.

If this is the case, then of course this bug report is mostly invalid. It just 
never seriously occured to me that this is what IplImage.createFrom does. I 
can't imagine how this behaviour could be useful. I know that if the 
BufferedImage is either grayscale or happens to contain the color components in 
the same order as OpenCV expects them, then the IplImage will contain the same 
image as the original BufferedImage. But otherwise, I don't see why anyone 
would use it.

What is more, this isn't documented anywhere. All I've found is a comment in 
the main demo program that says "The createFrom() factory method can construct 
an IplImage from a BufferedImage." Both this sentence and the signature of the 
function suggested to me that the IplImage it constructs will contain the same 
image.

Please confirm if my theory is correct, just so that we're on the same page.

Original comment by kazocs...@gmail.com on 9 Mar 2012 at 9:58

[GoogleCodeExporter]

Yes, it looks like you are starting to understand indeed. As I have said a 
couple of times, IplImage has no concept of "color components", while simply 
copying the data without shifting around bytes provides best performance, so 
this is what IplImage.copyTo()/copyFrom() do.

Original comment by samuel.a...@gmail.com on 9 Mar 2012 at 7:18

[GoogleCodeExporter]

Issue 169 has been merged into this issue.

Original comment by samuel.a...@gmail.com on 9 Mar 2012 at 7:21

[GoogleCodeExporter]

My issue (169) has been merged with this one.  Thank you for the response and 
pointer, but I still cannot get the simple conversion to work (code):

public static BufferedImage conversionTest(BufferedImage input)
        {
        IplImage specs = IplImage.createFrom(input);
        IplImage image = IplImage.create(specs.width(), specs.height(), specs.depth(), specs.nChannels());
        image.copyFrom(input);
        BufferedImage finalImage = new BufferedImage(image.width(), image.height(), BufferedImage.TYPE_INT_BGR);
        image.copyTo(finalImage);
        return finalImage;
        }

I changed my code to use copyTo / copyFrom, as pointed out... I've attached the 
results below.

Can someone please post a working version of the code below?  I want to 
understand what is going on, but my frustration level is high, and I'd like to 
see something like this work...  I don't want to give up on this cool library 
so soon!  ;)

Original comment by benny....@gmail.com on 9 Mar 2012 at 9:30

Attachments:

• Pacman.gif
• saved.png

[GoogleCodeExporter]

I think what Samuel meant by using copyTo is that you need to use the 
*original* BufferedImage as the target. Something like this:

  IplImage ipl=IplImage.createFrom(input);
  IplImage result=cvCloneImage(ipl); // do some operations
  result.copyTo(input);

Because createFrom keeps the order of the color samples, and so does copyTo. So 
you need your output BufferedImage to interpret the color samples in the same 
way as your input, otherwise the colors get messed up. As far as I can tell, 
the only way to ensure this is to use the same BufferedImage object. (Unless 
you want to dig deep into the API, find out how your input interpreted the raw 
buffer data, and somehow construct a BufferedImage that has the same 
representation. This seems very hard to do in general, unless your input has 
one of the predefined types.) If you want to keep your original input 
BufferedImage intact, then I really don't know how you're supposed to do this.

Well, actually, I do have a workaround. I have managed to write conversion 
functions that do what you and I expect. I don't have access to it for now, but 
in the spirit of trying to help, I'll try to recall from memory. I apologize if 
I mess up. You need something like this:

IplImage imageToCv(BufferedImage input) {
  if (input.getType()!=BufferedImage.TYPE_BYTE_GRAY && input.getType()!=BufferedImage.TYPE_3BYTE_BGR) {
    // convert it to a BufferedImage with a component order that matches the OpenCV order
    BufferedImage tmp=new BufferedImage(input.getWidth(), input.getHeight(), BufferedImage.TYPE_3BYTE_BGR);
    Graphics2D g=tmp.createGraphics();
    g.drawImage(input, 0, 0, null);
    g.dispose();
    input=tmp;
  }
  return IplImage.createFrom(input);
}

BufferedImage cvToImage(IplImage input) {
  // no worries with grayscale images
  if (input.nChannels()==1)
    return input.getBufferedImage();
  // otherwise: the order in IplImage is BGR, so create a BufferedImage accordingly
  BufferedImage result=new BufferedImage(input.width(), input.height(), BufferedImage.TYPE_3BYTE_BGR);
  input.copyTo(result);
  return result;
}

This does not work with transparent images, but otherwise, unless I made a 
mistake, this should ensure that the colors don't get messed up.

Original comment by kazocs...@gmail.com on 9 Mar 2012 at 10:36

[GoogleCodeExporter]

You say that IplImage has no channel order. I can see that most of the OpenCV 
functions don't care which sample is the red one, but still, there would be 
good reason to follow the BGR and BGRA convention.

One of these is exactly the problem which resulted in this bug report: 
interoperability with BufferedImage. Suppose that your API is given a color 
BufferedImage to convert to IplImage. Then if this channel order isn't 
enforced, then there is no further information on the channel order of the 
IplImage, which in turn means that you cannot safely convert it back, save it 
to file, or display to the user. So there's actually very little you can do 
with it. This is why I care much less about performance and always make sure 
that my IplImage is in the proper channel order.

For images with transparency, I said to follow the order BGRA. You said in your 
first reply that the proper order was RGBA, and I say that that is not true. I 
explain why I say this.

I have found only one reference in the OpenCV docs of the 4-channel order [1]: 
a function which produces an image in BGRA order. But when using OpenCV, we can 
see that 4-channel images are interpreted as BGRA all around. For example, as I 
have pointed out above, cvShowImage displays (0, 0, 255, 255) as red, not blue. 
Or if you ask OpenCV itself to load a transparent image using 
cvLoadImage(filename, CV_LOAD_IMAGE_UNCHANGED), the result will be BGRA. 
Finally, if you ask OpenCV to save a 4-channel image, then it assumes BGRA 
order (despite the docs saying that cvSaveImage doesn't support 4-channel 
images [2]; it actually does). I've tried all this out, and this is why I 
ensure that my IplImage is in BGR or BGRA order, and this is why I feel that 
the current copyFrom/copyTo implementations are not really useful and that 
JavaCV should provide IplImage<->BufferedImage conversion functions which do 
handle channel order.

[1] 
http://opencv.itseez.com/modules/gpu/doc/camera_calibration_and_3d_reconstructio
n.html#gpu-drawcolordisp
[2] 
http://opencv.itseez.com/modules/highgui/doc/reading_and_writing_images_and_vide
o.html#imwrite

Original comment by kazocs...@gmail.com on 10 Mar 2012 at 7:30

[GoogleCodeExporter]

"This is why I care much less about performance and always make sure that my 
IplImage is in the proper channel order." -> You haven't been doing computer 
vision long enough ;)

cvLoadImage()/cvSaveImage() now support images with alpha channels? I didn't 
know, good to know! I tend to use ImageIO anyway since it works better with 
i18n filenames.

I never said that OpenCV used RGBA. I said that OpenCL, with an L, not a V, 
requires RGBA. Although BGRA works better for 8-bit images on NVIDIA cards (at 
least) for some reason, all other types (such as float) are only supported in 
RGBA format, not BGRA, BGR or RGB: it's RGBA or nothing (or R or RG, but no 
RGB). So that is why I say we should use RGBA, because apart from that CUDA 
function and highgui, nothing in OpenCV uses an alpha channel. Further, IMO, we 
should not use CUDA or highgui, but more portable APIs such as OpenCL and Swing 
or whatever, so this is why it makes sense to just do everything in RGBA. 
Besides, BufferedImage has no RGBA or BGRA type, only ABGR, so debating whether 
IplImage should be this or that is kind of moot here. If you copy your data 
from an ABGR BufferedImage, your IplImage is also going to be ABGR (at least, 
it should be, will fix that).

The two methods you propose up there may be the best workaround available... 
They have to perform two copies of the data, but if performance isn't an issue, 
why not I guess. That should satisfy your requirements, no?

Original comment by samuel.a...@gmail.com on 12 Mar 2012 at 7:48

[GoogleCodeExporter]

You did say OpenCV, I'm sorry. Silly look-alike abbreviations.

Either way, now that I kind of understand what exactly these functions do, I 
still cannot quite picture how you intended them to be used. You've been 
referring to using copyTo instead of getBufferedImage, but some more concrete 
examples would be greatly appreciated. Our colleague has provided an use case, 
and I can provide another.

He needed resize:

public static BufferedImage scaleBufferedImage(BufferedImage input, int width, 
int height) {
   IplImage inputIpl = ??????
   IplImage outputIpl = IplImage.create(width, height, inputIpl.depth(), inputIpl.nChannels());
   cvResize(inputIpl, outputIpl, CV_INTER_AREA);
   BufferedImage output = ??????
   return output;
}

I needed rectification:

public BufferedImage rectify(BufferedImage input) {
   IplImage inputIpl = ??????
   IplImage outputIpl = IplImage.createCompatible(inputIpl);
   cvRemap(inputIpl, outputIpl, ...);
   BufferedImage output = ??????
   return output;
}

What exactly are we supposed to do? As I understand, in my case I could write 
at the end of my function:

   outputIpl.copyTo(input);
   return input;

For this operation, I would probably get the correct colors, whatever the 
order. But it would overwrite my input image, which I don't want. And in the 
resize case, it would copy into a BufferedImage with a different size, so I 
don't see how that would work.

Original comment by kazocs...@gmail.com on 14 Mar 2012 at 5:38

[GoogleCodeExporter]

Let's see, would creating a compatible BufferedImage this way actually work by 
any chance?

    int type = input.getType();
    if (type == BufferedImage.TYPE_CUSTOM) {
        ColorModel cm = input.getColorModel();
        boolean a = input.isAlphaPremultiplied();
        WritableRaster r = input.copyData(null);
        output = new BufferedImage(cm, r, a, null);
    } else {
        int w = input.getWidth();
        int h = input.getHeight();
        output = new BufferedImage(w, h, type);
    }

Original comment by samuel.a...@gmail.com on 14 Mar 2012 at 11:25

[GoogleCodeExporter]

I posted the Pac-man comment above, and the posted code resolved my problem!  
Thank you for your help.

Original comment by benny....@gmail.com on 15 Mar 2012 at 10:21

[GoogleCodeExporter]

Ok, I fixed this issue in the latest release by cloning the BufferedImage as 
well when calling IplImage.clone() or IplImage.createCompatible(), using the 
code in comment #17. Let me know if this works as intended or not!

Original comment by samuel.a...@gmail.com on 29 Mar 2012 at 12:38

• Changed state: Fixed

[GoogleCodeExporter]

I'm sorry but I only have a very rough understanding of how these functions are 
intended to work. And especially since my original guess was so radically 
different from your actual intention, I'm quite weary about using them now. I 
believe that having an approximate idea can be worse than having no idea at all.

For example, now we have this new flipChannels parameter. I could guess what it 
is intended to do, and I could try it and see what happens when I use it, but 
even if these two match, I cannot be sure if my guess is correct.

Still, I tried it out. I created a red TYPE_4BYTE_ABGR BufferedImage, and 
called IplImage.createFrom. The resulting IplImage was blue both with 
flipChannels=true and flipChannels=false.

Anyway, due to my lack of proper grasp of these functions, I really cannot rely 
on them too much in my code, and I cannot tell if they work right. Your 
comments here can and have shed some light on them, but for me they are still 
far too sketchy.

Original comment by kazocs...@gmail.com on 3 Apr 2012 at 9:52

[GoogleCodeExporter]

I've just tried this on the latest release:
        BufferedImage bi = new BufferedImage(512, 512, BufferedImage.TYPE_4BYTE_ABGR);
        IplImage image1 = IplImage.createFrom(bi);
        IplImage image2 = IplImage.createCompatible(image1);
        System.out.println(image1.getBufferedImage());
        System.out.println(image2.getBufferedImage());

And the output shows there are two distinct BufferedImage objects that have 
same exact type:

BufferedImage@270e3293: type = 6 ColorModel: #pixelBits = 32 numComponents = 4 
color space = java.awt.color.ICC_ColorSpace@151a64ed transparency = 3 has alpha 
= true isAlphaPre = false ByteInterleavedRaster: width = 512 height = 512 
#numDataElements 4 dataOff[0] = 3
BufferedImage@6627e353: type = 6 ColorModel: #pixelBits = 32 numComponents = 4 
color space = java.awt.color.ICC_ColorSpace@151a64ed transparency = 3 has alpha 
= true isAlphaPre = false ByteInterleavedRaster: width = 512 height = 512 
#numDataElements 4 dataOff[0] = 3

In what way does that not fulfill your needs?

Original comment by samuel.a...@gmail.com on 3 Apr 2012 at 1:59

[GoogleCodeExporter]

Sure, it is great that these are the same. (Although the output for this code 
is the same with release 20120218 too. I don't know what it's supposed to test.)

Either way, I checked that the way I used to convert images still works with 
the new version; that is all I actually really need.

Original comment by kazocs...@gmail.com on 3 Apr 2012 at 2:08

[GoogleCodeExporter]

Just FYI, Android also, like OpenCL, prefers RGBA:
https://groups.google.com/forum/?fromgroups#!topic/android-opencv/99TqTq_AjcA 

Original comment by samuel.a...@gmail.com on 15 Apr 2012 at 5:44

[GoogleCodeExporter]

I just wanted to say a big thank you to kazocs -- the code you wrote in comment 
#14 works like a charm to solve this problem.

BTW, the problem I was running into was that if you record the screen using 
Java's "Robot" command, it creates its standard BufferedImage format, and when 
converted using the default IplImage.createFrom() command the colors got all 
messed up. It took me hours to track it down to the source and only minutes to 
solve thanks to your post. Thanks again for your help, kasocs!

Original comment by aaronpowers on 24 Dec 2013 at 2:23