search

akarjal2 / openjpeg

Automatically exported from code.google.com/p/openjpeg
0 stars 0 forks source link

conformance file p0_07.j2k not decoding correctly #80

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
Attached file is not decoding correctly. 

With r856 of trunk and openjpeg-1.5 branch : 

[16:58:48 
~/Documents/TELE/JPG2K/OPENJPEG/dvlp/all/branches/openjpeg-1.5/cmbuild/bin] 
./j2k_to_image -i ~/Desktop/p0_07.j2k -o test.bmp

[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (0) than 
number of tile-parts (0)
[INFO] tile 1 of 256
[ERROR] Expected EPH marker
[ERROR] tcd_decode: incomplete bistream
[INFO] - tiers-1 took 0.011788 s
[INFO] - dwt took 0.000748 s
[INFO] - tile decoded in 0.013870 s
ERROR -> j2k_to_image: failed to decode image!
[16:58:52 
~/Documents/TELE/JPG2K/OPENJPEG/dvlp/all/branches/openjpeg-1.5/cmbuild/bin]

With tag version.1.4 : 

[16:58:16 
~/Documents/TELE/JPG2K/OPENJPEG/dvlp/all/tags/version.1.4/cmbuild/bin] 
./j2k_to_image -i ~/Desktop/p0_07.j2k -o test.bmp

[INFO] tile 1 of 256
[ERROR] Expected EPH marker
[ERROR] tcd_decode: incomplete bistream
[INFO] - tiers-1 took 0.011339 s
[INFO] - dwt took 0.001735 s
[INFO] - tile decoded in 0.014901 s
ERROR -> j2k_to_image: failed to decode image!
[16:58:20 ~/Documents/TELE/JPG2K/OPENJPEG/dvlp/all/tags/version.1.4/cmbuild/bin]

Any idea on how to fix this is welcome.

Original issue reported on code.google.com by antonin on 10 Aug 2011 at 3:39

Attachments:

GoogleCodeExporter commented 8 years ago 
This issue can be followed on the openjpeg dashboard :
http://my.cdash.org/testSummary.php?project=237&name=ETS-C1P0-p0_07.j2k-decode&d
ate=2011-08-11

The name of this test is ETS-C1P0-p0_07.j2k-decode and it could be found for 
trunk and openjpeg 1.5 branch.

Mickaël

Original comment by savmick...@gmail.com on 11 Aug 2011 at 4:03

GoogleCodeExporter commented 8 years ago 
I have used OpenJPEG-1.5.0-revision-863.

Problem 1:
==========
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (0) than 
number of tile-parts (0)

This warning tells that '0' is greater than '0'. But the
test condition is:

if (partno >= numparts)

Problem 2:
==========
[ERROR] Expected EPH marker

I have written a trace line into 't2.c':

t2.c:494:hd(0xff,0x92) <== EPH nr 73
t2.c:494:hd(0x06,0x05) <== expected nr 74
[ERROR] Expected EPH marker

I have a simple J2K reader that shows:

Observation 1:
==============
NAME(p0_07.j2k)
LENG(7885684)

ENTER read_jp2c
[0]marker(0xff4f)
    soc len(0)
[2]marker(0xff51)
    siz len(47)
    capabilities(1)
    x(0 : 2048) y(0 : 2048)
    xt(0 : 128) yt(0 : 128)
    nr_components(3)
      component[0]signed(1) prec(12) hsep(1) vsep(1)
      component[1]signed(1) prec(12) hsep(1) vsep(1)
      component[2]signed(1) prec(12) hsep(1) vsep(1)
[51]marker(0xff52)
    cod  len(12)
[65]marker(0xff5c)
    qcd  len(13)
[80]marker(0xff64)
    com  len(16)
    R[1](General use (ISO 8859-1 (latin-1) values))
       T(Kakadu-3.0.7)
[98]marker(0xff90)
    sot  tile_nr(0) Psot(9951) TPsot(0) TNsot(0)
    len(10)

(...)

[7836303]marker(0xff90)
    sot  tile_nr(255) Psot(34820) TPsot(0) TNsot(1)
    len(10)

and after this tile nr 255:
---------------------------

[7871123]marker(0xff90)
    sot  tile_nr(0) Psot(14559) TPsot(1) TNsot(2)
    len(10)

Observation 2:
==============
[10049]marker(0xff90)
    sot  tile_nr(1) Psot(23681) TPsot(0) TNsot(1)
    len(10)
[10061]marker(0xff58)
    plt  len(134)
[10197]marker(0xff93)
    sod  len(0)
[10199]marker(0xff91)
    sop  data(4)
[10209]marker(0xff92)
    eph  skip_len(137) <== EPH nr 73
[10348]marker(0xff91)
    sop  data(3)
[10357]marker(0xff92)
    eph  skip_len(172) <== EPH nr 74
(...)

winfried

Original comment by szukw...@arcor.de on 12 Aug 2011 at 4:42

GoogleCodeExporter commented 8 years ago 
Many thanks Winfried. 

Problem 1 is actually a separate problem I should fix aside from this issue. 
Thanks for pointing this.

Problem 2 seems to come from the fact that the first tile is divided into 2 
tile-parts, one located at the very beginning of the code-stream and the second 
one at the very end (with all the 255 remaining tiles in between).

Attached is a useful file showing the structure of all the code-stream.

Still investigating ...

Antonin

Original comment by antonin on 12 Aug 2011 at 10:01

Attachments:

GoogleCodeExporter commented 8 years ago 
[98]marker(0xff90)
    sot  tile_nr(0) Psot(9951) TPsot(0) TNsot(0)
(...)
[10049]marker(0xff90)
    sot  tile_nr(1) Psot(23681) TPsot(0) TNsot(1)

98 + 9951 => 10049

Is there no possibility to test against the end of the tile?

winfried

Original comment by szukw...@arcor.de on 12 Aug 2011 at 5:56

GoogleCodeExporter commented 8 years ago 
tileno(0) is split into two parts.

The first  part has a totlen of  9951.
The second part has a totlen of 14559.

j2k.c:1529:j2k_read_eoc
    tileno(0) tile_len(24319)
(...)

t2.c:724:ENTER t2_decode_packets
    tileno(0) tilelen(24319)

(...)

t2.c:340:ENTER t2_decode_packet len(14942)
t2.c:365:SOP(0xff,0x91)
    t2.c:445:bandno(0) cblkno(0)
    t2.c:445:bandno(1) cblkno(0)
    t2.c:445:bandno(2) cblkno(0)
    t2.c:506:bio_numbytes(7)
t2.c:520:EPH(0xff,0x92)
    t2.c:743:t2_decode_packet returns(448) len(24319) curlen(9825)
    t2.c:733:tileno(0) pino(0) poc.prg(0)

t2.c:340:ENTER t2_decode_packet len(14494)
t2.c:365:SOP(0xff,0x91)
    t2.c:445:bandno(0) cblkno(0)
    t2.c:506:bio_numbytes(1)
t2.c:520:EPH(0xff,0x92)
    t2.c:743:t2_decode_packet returns(9) len(24319) curlen(9834)
    t2.c:733:tileno(0) pino(0) poc.prg(0)

t2.c:340:ENTER t2_decode_packet len(14485)
t2.c:365:SOP(0xff,0x91)
    t2.c:445:bandno(0) cblkno(0)
    t2.c:506:bio_numbytes(2)
t2.c:515:ERR(0x06,0x05)
[ERROR] Expected EPH marker
    t2.c:743:t2_decode_packet returns(-999) tilelen(24319) curlen(9834)
-------------------
tileno(0) seems to use the second len value for the first part.
So the pointer runs into tileno(1).

Apart from the different length values: is splitting is allowed?

If yes: could there be a collector?

add to tileno(0)
add to tileno(1)
(...)
add to tileno(255)
add to tileno(0)

winfried

Original comment by szukw...@arcor.de on 13 Aug 2011 at 3:52

GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
No, I am wrong. The contents of both tile parts are in
one buffer. The pointer does not run into tileno(1).
The len of the first part is 9825 byte.

t2.c:520:[13]EPH(0xff,0x92)
    t2.c:743:t2_decode_packet returns(448) tilelen(24319) curlen(9825)
    t2.c:733:tileno(0) pino(0) poc.prg(0)

------------ start of second part of tileno(0) ------------
t2.c:340:ENTER t2_decode_packet len(14494)

t2.c:365:[0]SOP(0xff,0x91)
    t2.c:445:bandno(0) cblkno(0)
    t2.c:506:bio_numbytes(1)

t2.c:520:[7]EPH(0xff,0x92)
    t2.c:743:t2_decode_packet returns(9) tilelen(24319) curlen(9834)
    t2.c:733:tileno(0) pino(0) poc.prg(0)

t2.c:340:ENTER t2_decode_packet len(14485)

t2.c:365:[0]SOP(0xff,0x91)
    t2.c:445:bandno(0) cblkno(0)
    t2.c:506:bio_numbytes(2)

t2.c:515:ERR(0x06,0x05)
    +2(0xd5,0x80) +4(0xff,0x92) +6(0xe2,0x63)
[ERROR] Expected EPH marker
    t2.c:743:t2_decode_packet returns(-999) tilelen(24319) curlen(9834)
[ERROR] tcd_decode: incomplete bistream
ERROR -> j2k_to_image: failed to decode image!
-------------------------------
winfried

Original comment by szukw...@arcor.de on 17 Aug 2011 at 6:38

GoogleCodeExporter commented 8 years ago 
In 'j2k_read_sod()' I have sent some values of part2 of
tileno(0) to stderr:

 tileno(0)
 OLD data(0x812a060) len(9825)
 NEW len(14494)

The new data:

        [0](0xff,0x91)  sop(0xff,0x91)
        [1](0x00,0x04)  len(4) + data(1)
        [2](0x00,0x48)  seq_nr(72)(0x00,0x48)

        [3](0x80,0xff)  eph(0xff,0x92), data(0)

        [4](0x92,0xff)  sop(0xff,0x91)
        [5](0x91,0x00)  len(4) + data(6)
        [6](0x04,0x00)  seq_nr(73)(0x00,0x49)
        [7](0x49,0xc1)
        [8](0xdd,0x06)  ERR((0x06,0x05)) : bio_numbytes(2)
        [9](0x05,0xd5)

        [10](0x80,0xff) eph((0xff,0x92), data(108)
(...)

winfried

Original comment by szukw...@arcor.de on 17 Aug 2011 at 10:44

GoogleCodeExporter commented 8 years ago 
seq 70 and 71, the last two sequences in part 1, have the
'included' flag set; their sum of all 'segs[segno].newlen'
is correct (the same holds for all other before).

seq 72, the first sequence in part 2, does not have the 
'included' flag set.

seq 73 has the 'included' flag set. And fails.
'segs[segno].newlen' is 7, 'bio_numbytes()' is 2.
The restart begins at the third byte within the sop:
'0x06'; instead of at the first byte after the sop:
'0xff'.

winfried

Original comment by szukw...@arcor.de on 19 Aug 2011 at 2:05

GoogleCodeExporter commented 8 years ago 
p0_07.j2k has two different resno1(i.e. REpoc) values:

j2k.c:1005:TRACE ENTER j2k_read_poc old_nr(0) new_nr(1)
    [0]resno0(0) compno0(0) layno1(9) resno1(3) compno1(3) prg(0)

j2k.c:1005:TRACE ENTER j2k_read_poc old_nr(1) new_nr(2)
    [1]resno0(0) compno0(0) layno1(9) resno1(8) compno1(3) prg(0)

The first part ends with seqno(71).
The second part starts with seqno(72).

t2.c:720:TRACE ENTER t2_decode_packets
    tileno(0) max_pino(1)

t2.c:339:TRACE ENTER t2_decode_packet
    resno(0) compno(0) layno(0) precno(0)
t2.c:375:SOP(0xff,0x91) len(4)  seqno(0)

(...)

t2.c:339:TRACE ENTER t2_decode_packet
    resno(2) compno(2) layno(7) precno(0)
t2.c:375:SOP(0xff,0x91) len(4)  seqno(71)

t2.c:339:TRACE ENTER t2_decode_packet
    resno(0) compno(0) layno(8) precno(0)
t2.c:375:SOP(0xff,0x91) len(4)  seqno(72)

t2.c:339:TRACE ENTER t2_decode_packet
    resno(0) compno(1) layno(8) precno(0)
t2.c:375:SOP(0xff,0x91) len(4)  seqno(73)
[ERROR] Expected EPH marker
[ERROR] tcd_decode: incomplete bistream
ERROR -> j2k_to_image: failed to decode image!

winfried

Original comment by szukw...@arcor.de on 3 Sep 2011 at 11:42

GoogleCodeExporter commented 8 years ago 
I have the same problem with trunk and V2 style (rev961):
http://my.cdash.org/testDetails.php?test=7655496&build=242949

Mickaël

Original comment by savmick...@gmail.com on 5 Oct 2011 at 4:48

GoogleCodeExporter commented 8 years ago 
Hi,

Dashboard (rev998 + winfried heap corruption fix) provide some observations 
about this bug: 
all the platform and v1.5 failed to decode this file: 
http://my.cdash.org/testSummary.php?project=237&name=ETS-C1P0-p0_07.j2k-decode&d
ate=2011-10-12 because EPH marker is not detected as attended.

with trunk, windows platform decode the file (without eph marker error) and the 
output image is equal to the conformance image: 
http://my.cdash.org/testDetails.php?test=7684271&build=246254 and 
http://my.cdash.org/testDetails.php?test=7684272&build=246254

On the other hand, the macOS and ubuntu decode the file with the EPH error and 
the comparison between output image and reference image failed: 
http://my.cdash.org/testDetails.php?test=7664016&build=246236

Any idea, why these differences ?

Mickaël

Original comment by savmick...@gmail.com on 12 Oct 2011 at 3:01

GoogleCodeExporter commented 8 years ago 
Looks like a struct is not being initialized:

[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (0) than 
number of tile-parts (0)

[INFO] tile 1 of 256
==7949== Invalid read of size 2
==7949==    at 0x4E3DA57: pi_next_lrcp (pi.c:109)
==7949==    by 0x4E413A2: pi_next (pi.c:713)
==7949==    by 0x4E49070: t2_decode_packets (t2.c:706)
==7949==    by 0x4E4FBA9: tcd_decode_tile (tcd.c:1367)
==7949==    by 0x4E36662: j2k_read_eoc (j2k.c:1566)
==7949==    by 0x4E36CF5: j2k_decode (j2k.c:1877)
==7949==    by 0x4E3D32B: opj_decode_with_info (openjpeg.c:163)
==7949==    by 0x4E3D2D7: opj_decode (openjpeg.c:156)
==7949==    by 0x404458: main (j2k_to_image.c:631)
==7949==  Address 0x72b64b0 is 0 bytes after a block of size 192 alloc'd
==7949==    at 0x4C2380C: calloc (vg_replace_malloc.c:467)
==7949==    by 0x4E3FB65: pi_create_decode (pi.c:505)
==7949==    by 0x4E48B68: t2_decode_packets (t2.c:697)
==7949==    by 0x4E4FBA9: tcd_decode_tile (tcd.c:1367)
==7949==    by 0x4E36662: j2k_read_eoc (j2k.c:1566)
==7949==    by 0x4E36CF5: j2k_decode (j2k.c:1877)
==7949==    by 0x4E3D32B: opj_decode_with_info (openjpeg.c:163)
==7949==    by 0x4E3D2D7: opj_decode (openjpeg.c:156)
==7949==    by 0x404458: main (j2k_to_image.c:631)
==7949== 
==7949== Invalid write of size 2
==7949==    at 0x4E3DA71: pi_next_lrcp (pi.c:110)
==7949==    by 0x4E413A2: pi_next (pi.c:713)
==7949==    by 0x4E49070: t2_decode_packets (t2.c:706)
==7949==    by 0x4E4FBA9: tcd_decode_tile (tcd.c:1367)
==7949==    by 0x4E36662: j2k_read_eoc (j2k.c:1566)
==7949==    by 0x4E36CF5: j2k_decode (j2k.c:1877)
==7949==    by 0x4E3D32B: opj_decode_with_info (openjpeg.c:163)
==7949==    by 0x4E3D2D7: opj_decode (openjpeg.c:156)
==7949==    by 0x404458: main (j2k_to_image.c:631)
==7949==  Address 0x72b64b0 is 0 bytes after a block of size 192 alloc'd
==7949==    at 0x4C2380C: calloc (vg_replace_malloc.c:467)
==7949==    by 0x4E3FB65: pi_create_decode (pi.c:505)
==7949==    by 0x4E48B68: t2_decode_packets (t2.c:697)
==7949==    by 0x4E4FBA9: tcd_decode_tile (tcd.c:1367)
==7949==    by 0x4E36662: j2k_read_eoc (j2k.c:1566)
==7949==    by 0x4E36CF5: j2k_decode (j2k.c:1877)
==7949==    by 0x4E3D32B: opj_decode_with_info (openjpeg.c:163)
==7949==    by 0x4E3D2D7: opj_decode (openjpeg.c:156)
==7949==    by 0x404458: main (j2k_to_image.c:631)
==7949==

Original comment by mathieu.malaterre on 11 Jan 2012 at 9:37

GoogleCodeExporter commented 8 years ago 
Have you tried to read this same image using V2? I get expected EPH marker 
errors there too and the first tile is messed up.

Original comment by Bob...@gmail.com on 22 May 2013 at 4:46

GoogleCodeExporter commented 8 years ago

Original comment by mathieu.malaterre on 20 Feb 2014 at 11:24

GoogleCodeExporter commented 8 years ago 
Here is what I see in the svn log:

------------------------------------------------------------------------
r1007 | savmickael | 2011-10-12 16:44:46 +0200 (mer. 12 oct. 2011) | 1 ligne

[trunk] WIP: resolve heap corruption with p0_07.j2k (credit to Winfried)
------------------------------------------------------------------------

Original comment by mathieu.malaterre on 20 Feb 2014 at 2:25

GoogleCodeExporter commented 8 years ago 
openjpeg trunk was able to cope (= parse) with this file during changes 
introduce in r961

Original comment by mathieu.malaterre on 20 Feb 2014 at 4:29

GoogleCodeExporter commented 8 years ago 
the following code in j2k.c is bogus:

    partno = cio_read(cio, 1);
    numparts = cio_read(cio, 1);

  if (partno >= numparts) {
    opj_event_msg(j2k->cinfo, EVT_WARNING, "SOT marker inconsistency in tile %d: tile-part index greater (%d) than number of tile-parts (%d)\n", tileno, partno, numparts);
    numparts = partno+1;
  }

The case where numparts is 0 (unknown) is not handled at all.

Original comment by mathieu.malaterre on 20 Feb 2014 at 4:50

GoogleCodeExporter commented 8 years ago

Original comment by mathieu.malaterre on 25 Feb 2014 at 3:26

GoogleCodeExporter commented 8 years ago 
Seems like this is working today, I can get an output very close to the 
expected PGX (conformance). Need to add a PSNR test.

Original comment by mathieu.malaterre on 27 Feb 2014 at 10:10

GoogleCodeExporter commented 8 years ago 
PSNR test is already included in the test suite. This is precisely because PSNR 
is out of tolerated range that compare2ref test is failing

Original comment by antonin on 27 Feb 2014 at 10:30

GoogleCodeExporter commented 8 years ago

Original comment by antonin on 4 Mar 2014 at 9:08

GoogleCodeExporter commented 8 years ago 
yes this file is not correctly decoded according to the conformance file. I 
have spend some time to understand why but it is not tricky.

Original comment by savmick...@gmail.com on 6 Mar 2014 at 11:42

GoogleCodeExporter commented 8 years ago 
I suspect the issue comes from the weird tile ordering of the file:

[...]
99      : New marker: SOT (Start of tile-part)

    Tile       : 0
    Length     : 9850
    Index      : 0
    Tile-Parts : unknown
[...]
7833153 : New marker: SOT (Start of tile-part)

    Tile       : 0
    Length     : 14519
    Index      : 1
    Tile-Parts : 2

Original comment by mathieu.malaterre on 11 Mar 2014 at 11:43

GoogleCodeExporter commented 8 years ago 
Confirmed. If I re-organize the file:

$ kdu_transcode -i data/input/conformance/p0_07.j2k -o clean.j2k 
Porder:T0={0,0,9,8,3,LRCP}

openjpeg is able to decompress the first tile properly (= the whole image).

Note the original image was compressed using something like:

kdu_compress [...] Porder:T0={0,0,9,3,3,LRCP} Porder:T0={0,0,9,8,3,LRCP}

Original comment by mathieu.malaterre on 11 Mar 2014 at 11:55

GoogleCodeExporter commented 8 years ago 
Let's divide the issue into sub issues:

$ kdu_expand -i input/conformance/p0_07.j2k -o 1.tif,2.tif,3.tif -record ref.txt
$ kdu_compress -no_info -i 1.tif,2.tif,3.tif -o kdu_ref.tif.j2k Creversible=yes 
-record test.txt Stiles="{128,128}" Corder=RLCP Clayers=8 Cuse_sop=yes 
Cuse_eph=yes Clevels=3 Sprofile=PROFILE0  Porder:T0="{0,0,9,3,3,LRCP}" 
Porder:T0="{0,0,9,8,3,LRCP}" Cycc=no ORGgen_plt=yes

-> this is approximately the same image as p0_07.j2k

Let's start by removing the tile handling:

$ kdu_compress -no_info -i 1.tif,2.tif,3.tif -o kdu_ref.tif.j2k Creversible=yes 
-record test.txt  Corder=RLCP Clayers=8 Cuse_sop=yes Cuse_eph=yes Clevels=3 
Sprofile=PROFILE2  Porder:T0="{0,0,9,3,3,LRCP}" Porder:T0="{0,0,9,8,3,LRCP}" 
Cycc=no ORGgen_plt=yes

Original comment by mathieu.malaterre on 11 Mar 2014 at 2:33

GoogleCodeExporter commented 8 years ago 
Using a smaller image might be easier: 

$ kdu_compress -no_info -i lena512.pgm -o lena512.i80.j2k Creversible=yes 
-record test.txt  Corder=RLCP Clayers=8 Cuse_sop=yes Cuse_eph=yes Clevels=3 
Sprofile=PROFILE0  Porder:T0="{0,0,9,3,3,LRCP}" Porder:T0="{0,0,9,8,3,LRCP}" 
Cycc=no ORGgen_plt=yes 

Steps:
$ ./bin/opj_decompress -i lena512.i80.j2k -o lena512.i80.j2k.pgm

[INFO] Start to read j2k main header (0).
[INFO] Main header has been correctly decoded.
[INFO] No decoded area parameters, set the decoded area to the whole image
[INFO] Header of tile 0 / 0 has been read.
Error : expected EPH marker
Error : expected EPH marker
Error : expected EPH marker
Error : expected EPH marker
Error : expected EPH marker
Error : expected EPH marker
[INFO] Tile 1/1 has been decoded.
[INFO] Image data has been updated with tile 1.

Generated Outfile lena512.i80.j2k.pgm

Original comment by mathieu.malaterre on 11 Mar 2014 at 2:35

Attachments:

GoogleCodeExporter commented 8 years ago 
Just for reference EPH handling seems broken according to previous notes. 
Removing the EPH leads to the same error.

Original comment by mathieu.malaterre on 11 Mar 2014 at 2:52

GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
This issue was updated by revision r2690.

Original comment by mathieu.malaterre on 11 Mar 2014 at 3:17

GoogleCodeExporter commented 8 years ago 
This issue was closed by revision r2692.

Original comment by mathieu.malaterre on 12 Mar 2014 at 10:10

GoogleCodeExporter commented 8 years ago 
This issue was updated by revision r2693.

Original comment by mathieu.malaterre on 12 Mar 2014 at 10:35

GoogleCodeExporter commented 8 years ago 
Both images are OK. Thank you :-)
winfried

Original comment by szukw...@arcor.de on 12 Mar 2014 at 12:26

GoogleCodeExporter commented 8 years ago 
This issue was updated by revision r2710.

Original comment by mathieu.malaterre on 13 Mar 2014 at 10:56

GoogleCodeExporter commented 8 years ago

Original comment by antonin on 24 Mar 2014 at 3:57