conformance file p0_07.j2k not decoding correctly

gcode-importer commented 13 years ago

Originally reported on Google Code with ID 80

Attached file is not decoding correctly. 

With r856 of trunk and openjpeg-1.5 branch : 

[16:58:48 ~/Documents/TELE/JPG2K/OPENJPEG/dvlp/all/branches/openjpeg-1.5/cmbuild/bin]
./j2k_to_image -i ~/Desktop/p0_07.j2k -o test.bmp

[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (0) than number
of tile-parts (0)
[INFO] tile 1 of 256
[ERROR] Expected EPH marker
[ERROR] tcd_decode: incomplete bistream
[INFO] - tiers-1 took 0.011788 s
[INFO] - dwt took 0.000748 s
[INFO] - tile decoded in 0.013870 s
ERROR -> j2k_to_image: failed to decode image!
[16:58:52 ~/Documents/TELE/JPG2K/OPENJPEG/dvlp/all/branches/openjpeg-1.5/cmbuild/bin]

With tag version.1.4 : 

[16:58:16 ~/Documents/TELE/JPG2K/OPENJPEG/dvlp/all/tags/version.1.4/cmbuild/bin] ./j2k_to_image
-i ~/Desktop/p0_07.j2k -o test.bmp

[INFO] tile 1 of 256
[ERROR] Expected EPH marker
[ERROR] tcd_decode: incomplete bistream
[INFO] - tiers-1 took 0.011339 s
[INFO] - dwt took 0.001735 s
[INFO] - tile decoded in 0.014901 s
ERROR -> j2k_to_image: failed to decode image!
[16:58:20 ~/Documents/TELE/JPG2K/OPENJPEG/dvlp/all/tags/version.1.4/cmbuild/bin]

Any idea on how to fix this is welcome.

Reported by detonin on 2011-08-10 15:39:35

_Attachment: p0_07.j2k_

gcode-importer commented 13 years ago

This issue can be followed on the openjpeg dashboard :
http://my.cdash.org/testSummary.php?project=237&name=ETS-C1P0-p0_07.j2k-decode&date=2011-08-11

The name of this test is ETS-C1P0-p0_07.j2k-decode and it could be found for trunk
and openjpeg 1.5 branch.

Mickaël

Reported by savmickael on 2011-08-11 16:03:51

gcode-importer commented 13 years ago

I have used OpenJPEG-1.5.0-revision-863.

Problem 1:
==========
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (0) than number
of tile-parts (0)

This warning tells that '0' is greater than '0'. But the
test condition is:

if (partno >= numparts)

Problem 2:
==========
[ERROR] Expected EPH marker

I have written a trace line into 't2.c':

t2.c:494:hd(0xff,0x92) <== EPH nr 73
t2.c:494:hd(0x06,0x05) <== expected nr 74
[ERROR] Expected EPH marker

I have a simple J2K reader that shows:

Observation 1:
==============
NAME(p0_07.j2k)
LENG(7885684)

ENTER read_jp2c
[0]marker(0xff4f)
    soc len(0)
[2]marker(0xff51)
    siz len(47)
    capabilities(1)
    x(0 : 2048) y(0 : 2048)
    xt(0 : 128) yt(0 : 128)
    nr_components(3)
      component[0]signed(1) prec(12) hsep(1) vsep(1)
      component[1]signed(1) prec(12) hsep(1) vsep(1)
      component[2]signed(1) prec(12) hsep(1) vsep(1)
[51]marker(0xff52)
    cod  len(12)
[65]marker(0xff5c)
    qcd  len(13)
[80]marker(0xff64)
    com  len(16)
    R[1](General use (ISO 8859-1 (latin-1) values))
       T(Kakadu-3.0.7)
[98]marker(0xff90)
    sot  tile_nr(0) Psot(9951) TPsot(0) TNsot(0)
    len(10)

(...)

[7836303]marker(0xff90)
    sot  tile_nr(255) Psot(34820) TPsot(0) TNsot(1)
    len(10)

and after this tile nr 255:
---------------------------

[7871123]marker(0xff90)
    sot  tile_nr(0) Psot(14559) TPsot(1) TNsot(2)
    len(10)

Observation 2:
==============
[10049]marker(0xff90)
    sot  tile_nr(1) Psot(23681) TPsot(0) TNsot(1)
    len(10)
[10061]marker(0xff58)
    plt  len(134)
[10197]marker(0xff93)
    sod  len(0)
[10199]marker(0xff91)
    sop  data(4)
[10209]marker(0xff92)
    eph  skip_len(137) <== EPH nr 73
[10348]marker(0xff91)
    sop  data(3)
[10357]marker(0xff92)
    eph  skip_len(172) <== EPH nr 74
(...)

winfried

Reported by szukw000 on 2011-08-12 04:42:37

gcode-importer commented 13 years ago

Many thanks Winfried. 

Problem 1 is actually a separate problem I should fix aside from this issue. Thanks
for pointing this.

Problem 2 seems to come from the fact that the first tile is divided into 2 tile-parts,
one located at the very beginning of the code-stream and the second one at the very
end (with all the 255 remaining tiles in between).

Attached is a useful file showing the structure of all the code-stream.

Still investigating ...

Antonin

Reported by detonin on 2011-08-12 10:01:07

_Attachment: p0_07syntax.txt_

gcode-importer commented 13 years ago

[98]marker(0xff90)
    sot  tile_nr(0) Psot(9951) TPsot(0) TNsot(0)
(...)
[10049]marker(0xff90)
    sot  tile_nr(1) Psot(23681) TPsot(0) TNsot(1)

98 + 9951 => 10049

Is there no possibility to test against the end of the tile?

winfried

Reported by szukw000 on 2011-08-12 17:56:56

gcode-importer commented 13 years ago

tileno(0) is split into two parts.

The first  part has a totlen of  9951.
The second part has a totlen of 14559.

j2k.c:1529:j2k_read_eoc
    tileno(0) tile_len(24319)
(...)

t2.c:724:ENTER t2_decode_packets
    tileno(0) tilelen(24319)

(...)

t2.c:340:ENTER t2_decode_packet len(14942)
t2.c:365:SOP(0xff,0x91)
    t2.c:445:bandno(0) cblkno(0)
    t2.c:445:bandno(1) cblkno(0)
    t2.c:445:bandno(2) cblkno(0)
    t2.c:506:bio_numbytes(7)
t2.c:520:EPH(0xff,0x92)
    t2.c:743:t2_decode_packet returns(448) len(24319) curlen(9825)
    t2.c:733:tileno(0) pino(0) poc.prg(0)

t2.c:340:ENTER t2_decode_packet len(14494)
t2.c:365:SOP(0xff,0x91)
    t2.c:445:bandno(0) cblkno(0)
    t2.c:506:bio_numbytes(1)
t2.c:520:EPH(0xff,0x92)
    t2.c:743:t2_decode_packet returns(9) len(24319) curlen(9834)
    t2.c:733:tileno(0) pino(0) poc.prg(0)

t2.c:340:ENTER t2_decode_packet len(14485)
t2.c:365:SOP(0xff,0x91)
    t2.c:445:bandno(0) cblkno(0)
    t2.c:506:bio_numbytes(2)
t2.c:515:ERR(0x06,0x05)
[ERROR] Expected EPH marker
    t2.c:743:t2_decode_packet returns(-999) tilelen(24319) curlen(9834)
-------------------
tileno(0) seems to use the second len value for the first part.
So the pointer runs into tileno(1).

Apart from the different length values: is splitting is allowed?

If yes: could there be a collector?

add to tileno(0)
add to tileno(1)
(...)
add to tileno(255)
add to tileno(0)

winfried

Reported by szukw000 on 2011-08-13 03:52:30

gcode-importer commented 13 years ago

No, I am wrong. The contents of both tile parts are in
one buffer. The pointer does not run into tileno(1).
The len of the first part is 9825 byte.

t2.c:520:[13]EPH(0xff,0x92)
    t2.c:743:t2_decode_packet returns(448) tilelen(24319) curlen(9825)
    t2.c:733:tileno(0) pino(0) poc.prg(0)

------------ start of second part of tileno(0) ------------
t2.c:340:ENTER t2_decode_packet len(14494)

t2.c:365:[0]SOP(0xff,0x91)
    t2.c:445:bandno(0) cblkno(0)
    t2.c:506:bio_numbytes(1)

t2.c:520:[7]EPH(0xff,0x92)
    t2.c:743:t2_decode_packet returns(9) tilelen(24319) curlen(9834)
    t2.c:733:tileno(0) pino(0) poc.prg(0)

t2.c:340:ENTER t2_decode_packet len(14485)

t2.c:365:[0]SOP(0xff,0x91)
    t2.c:445:bandno(0) cblkno(0)
    t2.c:506:bio_numbytes(2)

t2.c:515:ERR(0x06,0x05)
    +2(0xd5,0x80) +4(0xff,0x92) +6(0xe2,0x63)
[ERROR] Expected EPH marker
    t2.c:743:t2_decode_packet returns(-999) tilelen(24319) curlen(9834)
[ERROR] tcd_decode: incomplete bistream
ERROR -> j2k_to_image: failed to decode image!
-------------------------------
winfried

Reported by szukw000 on 2011-08-17 18:38:57

gcode-importer commented 13 years ago

In 'j2k_read_sod()' I have sent some values of part2 of
tileno(0) to stderr:

 tileno(0)
 OLD data(0x812a060) len(9825)
 NEW len(14494)

The new data:

        [0](0xff,0x91)  sop(0xff,0x91)
        [1](0x00,0x04)  len(4) + data(1)
        [2](0x00,0x48)  seq_nr(72)(0x00,0x48)

        [3](0x80,0xff)  eph(0xff,0x92), data(0)

        [4](0x92,0xff)  sop(0xff,0x91)
        [5](0x91,0x00)  len(4) + data(6)
        [6](0x04,0x00)  seq_nr(73)(0x00,0x49)
        [7](0x49,0xc1)
        [8](0xdd,0x06)  ERR((0x06,0x05)) : bio_numbytes(2)
        [9](0x05,0xd5)

        [10](0x80,0xff) eph((0xff,0x92), data(108)
(...)

winfried

Reported by szukw000 on 2011-08-17 22:44:50

gcode-importer commented 13 years ago

seq 70 and 71, the last two sequences in part 1, have the
'included' flag set; their sum of all 'segs[segno].newlen'
is correct (the same holds for all other before).

seq 72, the first sequence in part 2, does not have the 
'included' flag set.

seq 73 has the 'included' flag set. And fails.
'segs[segno].newlen' is 7, 'bio_numbytes()' is 2.
The restart begins at the third byte within the sop:
'0x06'; instead of at the first byte after the sop:
'0xff'.

winfried

Reported by szukw000 on 2011-08-19 02:05:20

gcode-importer commented 13 years ago

p0_07.j2k has two different resno1(i.e. REpoc) values:

j2k.c:1005:TRACE ENTER j2k_read_poc old_nr(0) new_nr(1)
    [0]resno0(0) compno0(0) layno1(9) resno1(3) compno1(3) prg(0)

j2k.c:1005:TRACE ENTER j2k_read_poc old_nr(1) new_nr(2)
    [1]resno0(0) compno0(0) layno1(9) resno1(8) compno1(3) prg(0)

The first part ends with seqno(71).
The second part starts with seqno(72).

t2.c:720:TRACE ENTER t2_decode_packets
    tileno(0) max_pino(1)

t2.c:339:TRACE ENTER t2_decode_packet
    resno(0) compno(0) layno(0) precno(0)
t2.c:375:SOP(0xff,0x91) len(4)  seqno(0)

(...)

t2.c:339:TRACE ENTER t2_decode_packet
    resno(2) compno(2) layno(7) precno(0)
t2.c:375:SOP(0xff,0x91) len(4)  seqno(71)

t2.c:339:TRACE ENTER t2_decode_packet
    resno(0) compno(0) layno(8) precno(0)
t2.c:375:SOP(0xff,0x91) len(4)  seqno(72)

t2.c:339:TRACE ENTER t2_decode_packet
    resno(0) compno(1) layno(8) precno(0)
t2.c:375:SOP(0xff,0x91) len(4)  seqno(73)
[ERROR] Expected EPH marker
[ERROR] tcd_decode: incomplete bistream
ERROR -> j2k_to_image: failed to decode image!

winfried

Reported by szukw000 on 2011-09-03 11:42:07

gcode-importer commented 13 years ago

I have the same problem with trunk and V2 style (rev961):
http://my.cdash.org/testDetails.php?test=7655496&build=242949

Mickaël

Reported by savmickael on 2011-10-05 16:48:53

gcode-importer commented 13 years ago

Hi,

Dashboard (rev998 + winfried heap corruption fix) provide some observations about this
bug: 
all the platform and v1.5 failed to decode this file: http://my.cdash.org/testSummary.php?project=237&name=ETS-C1P0-p0_07.j2k-decode&date=2011-10-12
because EPH marker is not detected as attended.

with trunk, windows platform decode the file (without eph marker error) and the output
image is equal to the conformance image: http://my.cdash.org/testDetails.php?test=7684271&build=246254
and http://my.cdash.org/testDetails.php?test=7684272&build=246254

On the other hand, the macOS and ubuntu decode the file with the EPH error and the
comparison between output image and reference image failed: http://my.cdash.org/testDetails.php?test=7664016&build=246236

Any idea, why these differences ?

Mickaël

Reported by savmickael on 2011-10-12 15:01:58

gcode-importer commented 12 years ago

Looks like a struct is not being initialized:

[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (0) than number
of tile-parts (0)

[INFO] tile 1 of 256
==7949== Invalid read of size 2
==7949==    at 0x4E3DA57: pi_next_lrcp (pi.c:109)
==7949==    by 0x4E413A2: pi_next (pi.c:713)
==7949==    by 0x4E49070: t2_decode_packets (t2.c:706)
==7949==    by 0x4E4FBA9: tcd_decode_tile (tcd.c:1367)
==7949==    by 0x4E36662: j2k_read_eoc (j2k.c:1566)
==7949==    by 0x4E36CF5: j2k_decode (j2k.c:1877)
==7949==    by 0x4E3D32B: opj_decode_with_info (openjpeg.c:163)
==7949==    by 0x4E3D2D7: opj_decode (openjpeg.c:156)
==7949==    by 0x404458: main (j2k_to_image.c:631)
==7949==  Address 0x72b64b0 is 0 bytes after a block of size 192 alloc'd
==7949==    at 0x4C2380C: calloc (vg_replace_malloc.c:467)
==7949==    by 0x4E3FB65: pi_create_decode (pi.c:505)
==7949==    by 0x4E48B68: t2_decode_packets (t2.c:697)
==7949==    by 0x4E4FBA9: tcd_decode_tile (tcd.c:1367)
==7949==    by 0x4E36662: j2k_read_eoc (j2k.c:1566)
==7949==    by 0x4E36CF5: j2k_decode (j2k.c:1877)
==7949==    by 0x4E3D32B: opj_decode_with_info (openjpeg.c:163)
==7949==    by 0x4E3D2D7: opj_decode (openjpeg.c:156)
==7949==    by 0x404458: main (j2k_to_image.c:631)
==7949== 
==7949== Invalid write of size 2
==7949==    at 0x4E3DA71: pi_next_lrcp (pi.c:110)
==7949==    by 0x4E413A2: pi_next (pi.c:713)
==7949==    by 0x4E49070: t2_decode_packets (t2.c:706)
==7949==    by 0x4E4FBA9: tcd_decode_tile (tcd.c:1367)
==7949==    by 0x4E36662: j2k_read_eoc (j2k.c:1566)
==7949==    by 0x4E36CF5: j2k_decode (j2k.c:1877)
==7949==    by 0x4E3D32B: opj_decode_with_info (openjpeg.c:163)
==7949==    by 0x4E3D2D7: opj_decode (openjpeg.c:156)
==7949==    by 0x404458: main (j2k_to_image.c:631)
==7949==  Address 0x72b64b0 is 0 bytes after a block of size 192 alloc'd
==7949==    at 0x4C2380C: calloc (vg_replace_malloc.c:467)
==7949==    by 0x4E3FB65: pi_create_decode (pi.c:505)
==7949==    by 0x4E48B68: t2_decode_packets (t2.c:697)
==7949==    by 0x4E4FBA9: tcd_decode_tile (tcd.c:1367)
==7949==    by 0x4E36662: j2k_read_eoc (j2k.c:1566)
==7949==    by 0x4E36CF5: j2k_decode (j2k.c:1877)
==7949==    by 0x4E3D32B: opj_decode_with_info (openjpeg.c:163)
==7949==    by 0x4E3D2D7: opj_decode (openjpeg.c:156)
==7949==    by 0x404458: main (j2k_to_image.c:631)
==7949==

Reported by malaterre on 2012-01-11 09:37:58

gcode-importer commented 11 years ago

Have you tried to read this same image using V2? I get expected EPH marker errors there
too and the first tile is messed up.

Reported by Bobndc on 2013-05-22 16:46:32