Error 'segment too long (2563) with max (687) for codeblock'

[JaviBecerra]

Hi,

We submitted an issue at rasterio's repository regarding an error message opening some jp2 files (https://github.com/mapbox/rasterio/issues/1790) you can check there for full details; we are opening one here too because it seems to be an openjpeg issue.

The error message when opening some jp2 files with rasterio (offending files available at https://www.dropbox.com/sh/gomtjpuivaxz2sd/AABb9cMojBQxWqpUkLAftwyTa?dl=0):

ERROR 1: skip: segment too long (2563) with max (687) for codeblock 0 (p=1301, b=0, r=7, c=0)

ERROR 1: Failed to decode.

ERROR 1: Failed to decode tile 1/1

ERROR 1: opj_decode() failed
ERROR 1: ov185090_201700_c025u89x10c24n.jp2, band 1: IReadBlock failed at X offset 0, Y offset 0: opj_decode() failed
---------------------------------------------------------------------------
CPLE_AppDefinedError                      Traceback (most recent call last)
rasterio/_io.pyx in rasterio._io.DatasetReaderBase._read()

rasterio/shim_rasterioex.pxi in rasterio._shim.io_multi_band()

rasterio/_err.pyx in rasterio._err.exc_wrap_int()

CPLE_AppDefinedError: ov185090_201700_c025u89x10c24n.jp2, band 1: IReadBlock failed at X offset 0, Y offset 0: opj_decode() failed

During handling of the above exception, another exception occurred:

RasterioIOError                           Traceback (most recent call last)
<ipython-input-1-cb5d7807c346> in <module>
      5 
      6 ortho = rasterio.open(ortho_fn)
----> 7 ortho_array = ortho.read()
      8 print(ortho_array.shape)

rasterio/_io.pyx in rasterio._io.DatasetReaderBase.read()

rasterio/_io.pyx in rasterio._io.DatasetReaderBase._read()

RasterioIOError: Read or write failed. jp2/ov185090_201700_c025u89x10c24n.jp2, band 1: IReadBlock failed at X offset 0, Y offset 0: opj_decode() failed

Another gdal installation (inside QGIS) which includes several jpeg 2000 drivers in addition to JP2OpenJPEG opens the files properly, although I am not sure which is the driver that is being used.

Regards, Javier

[szukw000]

@JaviBecerra , the OK-file ist correctly constructed, both ERR-files are not. //----------------------------------------------- ERR1-ov171076_201700_c025u89x10c24n.jp2:

[107]marker(0xff64) com len(32) R[1](General use (ISO 8859-1 (latin-1) values)) T(ECW JPEG 2000 SDK v3.3.0.161) [141]marker(0xff90) ------- Psot (0) -------- read_jp2.c:1090: box_end(0x7f45b3e6fa10) - s(0x7f45b0824240) ==> Psot(56932304) sot tile_nr(0) Psot(56932304) TPsot(0) TNsot(0) sot len(10) [153]marker(0xff58) plt len(65534) [65689]marker(0xff58) plt len(6722) [72413]marker(0xff93) SOD len(56860040) [56932455]marker(0x711) 1:MARKER 0x711 is unknown. read_jp2.c:1343:END OF JP2C BOX EXIT read_jp2c WARNING: s(0x7f45b3e72b43) > box_end(0x7f45b3e6fa10) ==> 12595 last marker(0x711)

//----------------------------------------------- ERR2-ov185090_201700_c025u89x10c24n.jp2:

[107]marker(0xff64) com len(32) R[1](General use (ISO 8859-1 (latin-1) values)) T(ECW JPEG 2000 SDK v3.3.0.161) [141]marker(0xff90) ------- Psot (0) -------- read_jp2.c:1090: box_end(0x7f429cf5f610) - s(0x7f429a61b240) ==> Psot(43271120) sot tile_nr(0) Psot(43271120) TPsot(0) TNsot(0) sot len(10) [153]marker(0xff58) plt len(65534) [65689]marker(0xff58) plt len(11070) [76761]marker(0xff93) SOD len(43194508) [43271271]marker(0x2fe3) 1:MARKER 0x2fe3 is unknown. read_jp2.c:1343:END OF JP2C BOX EXIT read_jp2c WARNING: s(0x7f429cf62743) > box_end(0x7f429cf5f610) ==> 12595 last marker(0x2fe3)

//-----------------------------------------------

Both ERR-files pretend to have 2 PLT-boxes.

winfried

[rouault]

the OK-file ist correctly constructed, both ERR-files are not.

I don't agree with that analysis. Having several PLT boxes is authorized by the standard as far as I remember, as each of them is limited to 65536 bytes. And anyway PLT boxes are just skipped by openjpeg. The error is unrelated to that. And those files can be read by other JPEG2000 codecs.

[szukw000]

@rouault , part01_fcd15444-1-2016-10-15.pdf shows a strict sequence of boxes:

A.3 Construction of the codestream

-Figure A.2 ­ Construction of the codestream: SOC Required as the first marker main Main header marker segments

SOT Required at the beginning of each tile-part header T0, TP0 Tile 0, tile-part 0 header marker segments SOD Required at the end of each tile-part header bit stream Tile-part bit stream. Might include SOP and EPH

SOT Required at the beginning of each tile-part header T1, TP0 Tile 1, tile-part 0 header marker segments SOD Required at the end of each tile-part header bit stream

EOC

-Figure A.3 ­ Construction of the main header: SOC Required as the first marker SIZ Required as the second marker segment COD Required

-Figure A.4 ­ Construction of the first tile-part header of a given tile: SOT Required as the first marker segment of every tile-part header PLT Optional SOD Required as the last marker segment of every tile-part header

-Figure A.5 ­ Construction of a non-first tile-part header: SOT Required as the first marker segment of every tile-part header PLT Optional SOD Required as the last marker of every tile-part header

ERR1 and ERR2 both have the sequence:

SOT tile_nr(0) PLT PLT SOD

Whereas OK has the sequence:

SOT PLT SOD EOC

winfried

[rouault]

@szukw000 "A.7.3 Packet length, tile-part header (PLT)" mentions "There may be multiple PLT marker segments per tile". And openjpeg just parses them without doing any further processing with what they contain. Look at opj_j2k_read_plt() in j2k.c. The error here is emitted by t2.c and is completely unrelated

[szukw000]

@rouault , I observed the following.

1. kdu_expand-7.10.2 accepts all three files.
2. IrfanView accepts all three files.
3. kdu_show.exe accepts all three files.

winfried

[adriantre]

I experience the same behaviour reading jp2000 from a Google Bucket using /vsigs/. When downloading the file and reading from it locally, no problem occurs.

[sgillies]

That last comment suggests to me that the problem was in GDAL's VSI module and has been resolved in https://github.com/OSGeo/gdal/pull/2012. Yes @rouault ?

[rouault]

That last comment suggests to me that the problem was in GDAL's VSI module

No, the problem in this ticket is openjpeg specific. Can be reproduced with opj_decompress only & local file

[adriantre]

I think my problem may be caused by something else than the original issue. Although the error message is the same, I think it is caused by the fact that I try to stream random windows of a big image from a Google Bucket. To my understanding, random reading (/vsigs/) of jp2 files will result in one HTTP request per block accessed.

As I read many windows each overlapping with multiple blocks, Google will complain about not using exponential backoff. That error is not caught by OpenJPEG, which results in the RasterioIOError. Seems like the error does not occur when sequential reading the whole image (/vsigs_streaming/), which (I think) is only one HTTP request.

[michaelaye]

I am having the same problem with downloaded large images from the planetary data system:

Label file:

https://pds-geosciences.wustl.edu/lro/lro-l-dlre-4-rdr-v1/lrodlr_1001/data/gdr_l3/2010/cylindrical/jp2/dgdr_st_clc_cyl_20100523n_128_jp2.lbl

and the actual image: https://pds-geosciences.wustl.edu/lro/lro-l-dlre-4-rdr-v1/lrodlr_1001/data/gdr_l3/2010/cylindrical/jp2/dgdr_st_clc_cyl_20100523n_128_jp2.jp2

Interestingly, I only get below error when trying to read the image via the label (which apparently applies the conversion from lon/lat to map coordinates in meters.

For example:

$ gdal_translate dgdr_st_clc_cyl_20100523n_128_jp2.lbl dgdr_st_clc_cyl_20100523n_128_jp2.tif                                                         (py37) 
Input file size is 46080, 20480
0..ERROR 3: Failed to read scanline 1178.
ERROR 1: dgdr_st_clc_cyl_20100523n_128_jp2.lbl, band 1: IReadBlock failed at X offset 0, Y offset 1178: Failed to read scanline 1178.

When I directly read the JP2, the bounding box is in lon/lat and the read is working. I was wondering if the large coordinate numbers for the label reading case maybe cause a problem? This is the bounding box when reading via the label, using rasterio:

BoundingBox(left=-5458203.072, bottom=-2425868.032, right=5458203.072, top=2425868.032)