Java2DFrameConverter - Incorrect conversion from TYPE_INT_RGB

[light]

Hi,

when converting a BufferedImage of type TYPE_INT_RGB with Java2DFrameConverter the image seems interpreted as RGBA instead, i.e. there is an additional 4th channel (alpha) and the color channels are shifted around.

[saudet]

The channels are ARGB, and that's exactly how they are supposed to be.

[light]

In an image of type TYPE_INT_RGB the most significant byte is not used, it usually is a bunch of zeroes. So if you interpret it as the alpha channel you'll end up with an image that is entirely transparent.

Secondly the color fields are read as BGR, not RGB. For example this code :

    BufferedImage img = new BufferedImage( 1, 1, BufferedImage.TYPE_INT_RGB );
    img.setRGB( 0, 0, 0x112233 );
    Java2DFrameConverter converter = new Java2DFrameConverter();
    BufferedImage converted = converter.convert( converter.convert( img ) );
    System.out.println( Integer.toHexString( converted.getRGB( 0, 0 ) ) );

Outputs 332211. As you can see the fields are inverted.

[saudet]

There is no alpha channel, yes, but there is still a channel there, so let's just call the "abandoned" channel, and there you go: we've got ABGR, just like your output shows: 0x00332211. So, is your question about how to convert that to RGBA (whether A stands for alpha or abandoned)?

[saudet]

I think I understand the issue here. You would like to copy back to BufferedImage.TYPE_INT_RGB, is that correct? If so, I've just committed a fix for that. We can then use the copy() method to get ARGB:

    BufferedImage img = new BufferedImage( 1, 1, BufferedImage.TYPE_INT_RGB );
    img.setRGB( 0, 0, 0x112233 );
    Java2DFrameConverter converter = new Java2DFrameConverter();
    BufferedImage converted = new BufferedImage( 1, 1, BufferedImage.TYPE_INT_RGB );
    converter2.copy( converter.convert( img ), converted );
    System.out.println( Integer.toHexString( converted.getRGB( 0, 0 ) ) );

Which prints ff112233.

[light]

This code was the shortest example I could think of to illustrate the problem, but what I'm actually doing is reading a bunch of PNGs, doing some geometrical transformations (where the images get converted to TYPE_INT_RGB) and then converting them to Frames and feeding them to an FFmpegFrameRecorder. The final video has incorrect colors so the problem must be in the BufferedImage -> Frame conversion step. I think that the change you've just introduced compensates by swapping the channels again in the Frame -> BufferedImage conversion, but I don't think this can work since there are several types of image with different field orders like TYPE_INT_BGR and TYPE_INT_RGB so simply copying a buffer isn't enough.

Here is maybe a better example which uses the converter only in the BufferedImage -> Frame direction. Compare the image that is encoded and the resulting video :

    int size = 100;

    BufferedImage img = new BufferedImage( size, size, BufferedImage.TYPE_INT_RGB );
    for(int y = 0; y < size; y++) {
        for( int x = 0; x < size; x++ ) {
            img.setRGB( x, y, 0xe07030 );
        }
    }
    JOptionPane.showMessageDialog( null, new JLabel( new ImageIcon( img ) ) );

    FFmpegFrameRecorder recorder = new FFmpegFrameRecorder( "test.mp4", 0 );
    recorder.setFrameRate( 10 );
    recorder.setImageWidth( size );
    recorder.setImageHeight( size );
    recorder.setVideoCodec( AV_CODEC_ID_H264 );
    recorder.setVideoOption( "preset", "veryfast" );

    recorder.start();

    Java2DFrameConverter converter = new Java2DFrameConverter();
    for(int i = 0; i < 10; i++) {
        Frame frame = converter.convert( img );
        recorder.record( frame );
    }

    recorder.stop();
    recorder.release();

I'm filling the source image with 0xe07030, in RGB order. In the resulting video I'm reading the color 0x00E170. What I'm thinking is happening is that the (A)RGB fields of the source image are interpreted as RGBA.

Source image :

Output video :

[saudet]

I see, thanks for explaining. FFmpegFrameRecorder assumes by default that 4 channels are RGBA because that's what pretty much everything uses (except maybe Java2D and OpenCV :) given that OpenGL is pretty much limited to RGBA. But let's see, I've added a convenience method to the last commit so we can call record() this way for example:

        recorder.record( frame, AV_PIX_FMT_ARGB );

And that works just fine in the case of your example.

[light]

The confusion on my part was that I assumed that Frame was an image format to use inside of JavaCV, while it is more a data buffer. Java2DFrameConverter ferries the data between BufferedImage and Frame but does not change the way it is stored (although it makes some assumptions for gamma ?). The semantic information of the meaning of that data is still attached to the original source and must be kept around, or converted to the default RGBA beforehand.

Anyway, thanks for that last change now I don't have to swap my colors around :-)

[saudet]

It doesn't make assumptions about gamma either, but Java2D does so there's some functionality for conversion in Java2DFrameConverter as well. It's not very clear what should be done for all this. Experience in time will tell us, but for now we'll have to experiment, because all this is very much without precedents. In any case, yes, Frame should be considered as a buffer, containing only as much structure as is required, for performance reasons, but also for simplicity.

[saudet]

Fixes included in version 1.1. Thanks for reporting!