Regression in openjpeg 1.5.1...

[GoogleCodeExporter]

The attached file is regressed between openjpeg 1.5.0 and 1.5.1.
I generate a .png from the .jp2 for both versions (of course I take
care to LD_PRELOAD the correct library too, but omit that here):

j2k_to_image-1.5.0 -i test.jp2 -o test-1.5.0.png
j2k_to_image-1.5.1 -i test.jp2 -o test-1.5.1.png

Now, looking at the two files it is obvious that the file generated
by 1.5.1 is missing a large part of the image. I dug out the source
for openjpeg and located a change in libopenjpeg/j2k.c::j2k_read_sod()
that cause the issue, namely the block below:

{/* chop padding bytes: */
    unsigned char *s, *e; 

    s = cio_getbp(cio);
    e = s + len;

    if(e[-2] == 0x00 && e[-1] == 0x00) /* padding bytes */
    {
        abort();

        if(len > 8) s = e - 8;

        while(e > s)
        {       
            if(e[-2] == 0xff && e[-1] == 0xd9)  break;  
            --len; --e; truncate = 1;
        }       
    }
}

I tracked the addition of this code back to SVN r1702.
This mentions that it fixes issue 150. By reading that
issue and the google groups thread it references I gather
that the intent of the code above is to truncate 0x00 
bytes at the end of an j2k code stream when a generator
has wrongly appended padding after EOC (0xff 0xd9).

So far, so good. However, the problem is that the code
above is not only executed at the end of a j2k code 
stream, but rather it is executed for each tile-part 
bitstream found. At this point s in the code above 
will point to the first byte after SOD (0xff 0x93)
(namely the first byte of the tile part bit stream)
and len will cover the length of the data up until
but not including the next SOT (0xff 0x90). At this
point the code will be triggered if the last two 
bytes of data in the tile-part bitstream are 0x00 0x00. 
And at this point it will truncate the last 8 bytes
of the bitstream (unless it contains 0xff 0xd9 
somewhere).

Now the attached test.jp2 looks like this:
...
00007c0:                               ff 93 c7 d4 04 04            ......
00007d0: 48 c7 d4 04 03 30 c1 f2 02 01 58 c3 e7 04 00 00  H....0....X.....
00007e0: ff 90                                            ..
...

So basically we have SOD [unimportant data] 0x00 0x00 SOT,
which means that the code above get's triggered and chops
the last 8 bytes from this tile which of course causes 
problems. I don't know how to test for the end of the j2k
code stream so I can't easily produce a patch for you. But
since you presumably know the code well I believe that once 
you have the file that reproduces the issue and this analysis
you will be able to fix this in a jiffy. :) 

If further info is needed, let me know and I'll answer promptly.

Original issue reported on code.google.com by seb...@gmail.com on 20 Dec 2012 at 1:55

Attachments:

• test.jp2
• test-1.5.0.png
• test-1.5.1.png

[GoogleCodeExporter]

The reason for that patch in 'j2k.c' was the file

 'data-2012-12-11/input/nonregression/file409752.jp2'

This file has trailing zeros: these are chopped.

The following lines are from openjpeg-branch15-r2258.

In 'j2k.c' I have now made unusable lines between 1540 and 1558.

Now the output files

 ./j2k_to_image -i test.jp2 -o test.jp2[.png|.bmp|.tga|.tif]

seem to be OK. Only the BMP file does not have pale but fresh
colors.

The patch consisted of two parts: one for lines 1540 upto 1558
in 'j2k.c'. And one for below line 1965 and below line 2067:

 ./j2k_to_image -i fonzy-file409752.jp2 -o fonzy-file409752.jp2.png

[WARNING] 1965:00001f1e: expected a marker instead of 0
[INFO] tile 1 of 1
[INFO] - tiers-1 took 0.004999 s
[INFO] - dwt took 0.002000 s
[INFO] - tile decoded in 0.009998 s
Successfully generated Outfile fonzy-file409752.jp2.png

As you can see: in line 1965 of 'j2k.c' a correction is done,
namely 'j2k->state = J2K_STATE_NEOC;'. This seems to be enough
correction for the trailing zeros.

I do not know where this correction shall done in the trunk:

 ./opj_decompress -i fonzy-file409752.jp2 -o fonzy-file409752.jp2.png

[INFO] Start to read j2k main header (85).
[INFO] Main header has been correctly decoded.
[INFO] No decoded area parameters, set the decoded area to the whole image
[INFO] Psot value of the current tile-part is equal to zero, we assuming it is 
the last tile-part of the codestream.
[INFO] Header of tile 0 / 0 has been read.
[ERROR] Stream too short, expected SOT
[ERROR] Failed to decode the codestream in the JP2 file
ERROR -> opj_decompress: failed to decode image!

winfried

Original comment by szukw...@arcor.de on 20 Dec 2012 at 4:06

[GoogleCodeExporter]

Original comment by mathieu.malaterre on 20 Dec 2012 at 9:27

• Added labels: Priority-High
• Removed labels: Priority-Medium

[GoogleCodeExporter]

The poppler test suite is showing a large number of regressions between 1.5.0 
and 1.5.1. The attached file contains some of the jp2 images that are causing 
the regressions along with the pngs created with 1.5.0 and 1.5.1.

It looks like the images have been shrunk to a fraction of the original size. 
I'm not sure if this has been fixed in 2.0.0 but at present 2.0.0 is unusable 
due to the lack of pkgconfig support.

Original comment by adrian.j...@gmail.com on 1 Jan 2013 at 2:53

Attachments:

• images.zip

[GoogleCodeExporter]

1. pkgconfig support

   I install at '/usr/local/opj2'. You have to change this prefix.
   My name of the file is 'libopenjp2.pc'.

prefix=/usr/local/opj2
bindir=${prefix}/bin
mandir=${prefix}/share/man/
docdir=${prefix}/share/doc/openjpeg-2.0
libdir=${prefix}/lib
includedir=${prefix}/include/openjpeg-2.0

Name: openjp2
Description: JPEG2000 library (Part 1 and 2)
URL: http://www.openjpeg.org/
Version: 2.0.0
Libs: -L${libdir} -lopenjp2
Cflags: -I${includedir}

2. The last correct images for openjpeg-1.5.x I have created with
   openjpeg-branch15-r1701. With revision 1702 the images are broken.

3. openjpeg-trunk-r1701 creates broken images too.

4. My image viewer uses openjpeg-branch-r2258. And shows correct images
   created with openjpeg-branch15-r1701.

6. openjpeg-trunk-r2258 creates fairly broken images.

winfried

Original comment by szukw...@arcor.de on 3 Jan 2013 at 4:14

[GoogleCodeExporter]

In openjpeg-branch15-r1702 a patch of mine had been applied.

The patch consisted of two parts.

Part 1 in 'j2k.c' has the comment "chop padding bytes". 

If you deactivate the code in 'j2k.c'

  in r1702 between lines 1490 and 1506

or

  in r2258 between lines 1540 and 1558

then the output files should be OK.

winfried

#ifdef UNUSED_CODE
   {/* chop padding bytes: */
    unsigned char *s, *e;

    s = cio_getbp(cio);
    e = s + len;

  if(len > 8) s = e - 8;

  if(e[-2] == 0x00 && e[-1] == 0x00) /* padding bytes */
  {
    while(e > s)
 {
    if(e[-2] == 0xff && e[-1] == 0xd9)  break;
  --len; --e; truncate = 1;
 }
  }
   }
#endif /* UNUSED_CODE */

Original comment by szukw...@arcor.de on 4 Jan 2013 at 5:39

[GoogleCodeExporter]

winfried, if I understand correctly, are you advocating that the portion of 
j2k.c after /* chop padding bytes */ be omitted (for now)?

Original comment by rdieter@gmail.com on 7 Jan 2014 at 9:30

[GoogleCodeExporter]

I'll remove the offending code, since I was the one adding it.

Original comment by mathieu.malaterre on 24 Feb 2014 at 3:03

[GoogleCodeExporter]

Original comment by mathieu.malaterre on 25 Feb 2014 at 2:29

• Added labels: Milestone-Release1.5.2

[GoogleCodeExporter]

This issue was closed by revision r2423.

Original comment by mathieu.malaterre on 25 Feb 2014 at 2:45

• Changed state: Fixed