Color issue in Whitehorse GPKG

jyutzler commented 9 years ago

In the Whitehorse GPKG, the image tiles have strange colors.

Steps to reproduce:

Download https://portal.opengeospatial.org/files/63156
Open it in SQLite Manager
Open the Whitehorse table
Pick a tile row and double-click
Select the Save icon for the tile_data blob and save it to disk

Expected behavior: normal looking image Actual behavior: image with strange colors

pepijnve commented 9 years ago

As stated in #104, the ERDC file does not contain valid JFIF files. These are what's called 'Adobe JPEG' files. Quoting from the wikipedia article on JFIF:

JFIF is compatible with Adobe Photoshop's JPEG "Information Resource Block" extensions, and IPTC Information Interchange Model metadata, since JFIF does not preclude other application segments, and the Photoshop extensions are not required to be the first in the file. However, Photoshop generally saves CMYK buffers as four-component "Adobe JPEGs" that are not conformant with JFIF. Since these files are not in a YCbCr color space, they are typically not decodable by Web browsers and other Internet software.

The technical detailed answer is that:

JFIF requires 1 or 3 color components in an image. Luminance only (Y) or luminance and two chromatic channels (YCbCr).
JFIF specifies the formulas that should be used for conversions between RGB and YCbCr.
'Adobe JPEGs' have four components (YCCK or CMYK; not 100% sure which it is) and contain the additional colour profile information.
The images in the ERDC file are 4 component 'Adobe JPEG' images; not JFIF images.

JFIF and the Adobe JPEG format are both based on the JIF format. A somewhat tolerant JFIF reader will happily open an Adobe JPEG file and interpret the first 3 components as YCbCr and convert those to RGB. Since the wrong conversion formulas are being applied you see strange colours.

jyutzler commented 9 years ago

I'm still trying to work out why this happened and what to do about it. In the meantime, does this file look correct? https://drive.google.com/file/d/0B6m-m7RTR-ZIVHlYZXVyeDJKUTQ/view?usp=sharing If so, I will update it on geopackage.org.

pepijnve commented 9 years ago

@jyutzler I dumped a tile from each with jpegsnoop again. I only checked one tile. Looks correct now.

The output below is for the new files. Here the file correctly starts with a JFIF APP0 segment right after the SOI. The SOF0 segment also correctly shows 3 components.

JPEGsnoop 1.7.3 by Calvin Hass
  http://www.impulseadventure.com/photo/
  -------------------------------------

  Filename: [C:\Users\Administrator\Desktop\temp.jpg]
  Filesize: [13568] Bytes

Start Offset: 0x00000000
*** Marker: SOI (xFFD8) ***
  OFFSET: 0x00000000

*** Marker: APP0 (xFFE0) ***
  OFFSET: 0x00000002
  Length     = 16
  Identifier = [JFIF]
  version    = [1.1]
  density    = 1 x 1 (aspect ratio)
  thumbnail  = 0 x 0

*** Marker: DQT (xFFDB) ***
  Define a Quantization Table.
  OFFSET: 0x00000014
  Table length = 67
  ----
  Precision=8 bits
  Destination ID=0 (Luminance)
    DQT, Row #0:   8   6   5   8  12  20  26  31 
    DQT, Row #1:   6   6   7  10  13  29  30  28 
    DQT, Row #2:   7   7   8  12  20  29  35  28 
    DQT, Row #3:   7   9  11  15  26  44  40  31 
    DQT, Row #4:   9  11  19  28  34  55  52  39 
    DQT, Row #5:  12  18  28  32  41  52  57  46 
    DQT, Row #6:  25  32  39  44  52  61  60  51 
    DQT, Row #7:  36  46  48  49  56  50  52  50 
    Approx quality factor = 74.75 (scaling=50.51 variance=0.81)

*** Marker: DQT (xFFDB) ***
  Define a Quantization Table.
  OFFSET: 0x00000059
  Table length = 67
  ----
  Precision=8 bits
  Destination ID=1 (Chrominance)
    DQT, Row #0:   9   9  12  24  50  50  50  50 
    DQT, Row #1:   9  11  13  33  50  50  50  50 
    DQT, Row #2:  12  13  28  50  50  50  50  50 
    DQT, Row #3:  24  33  50  50  50  50  50  50 
    DQT, Row #4:  50  50  50  50  50  50  50  50 
    DQT, Row #5:  50  50  50  50  50  50  50  50 
    DQT, Row #6:  50  50  50  50  50  50  50  50 
    DQT, Row #7:  50  50  50  50  50  50  50  50 
    Approx quality factor = 74.74 (scaling=50.52 variance=0.19)

*** Marker: SOF0 (Baseline DCT) (xFFC0) ***
  OFFSET: 0x0000009E
  Frame header length = 17
  Precision = 8
  Number of Lines = 256
  Samples per Line = 256
  Image Size = 256 x 256
  Raw Image Orientation = Landscape
  Number of Img components = 3
    Component[1]: ID=0x01, Samp Fac=0x22 (Subsamp 1 x 1), Quant Tbl Sel=0x00 (Lum: Y)
    Component[2]: ID=0x02, Samp Fac=0x11 (Subsamp 2 x 2), Quant Tbl Sel=0x01 (Chrom: Cb)
    Component[3]: ID=0x03, Samp Fac=0x11 (Subsamp 2 x 2), Quant Tbl Sel=0x01 (Chrom: Cr)

...

For comparison, here's the output from the old file. No JFIF marker at the start and 4 components.

JPEGsnoop 1.7.3 by Calvin Hass
  http://www.impulseadventure.com/photo/
  -------------------------------------

  Filename: [C:\Users\Administrator\Desktop\sample.jpg]
  Filesize: [18488] Bytes

Start Offset: 0x00000000
*** Marker: SOI (xFFD8) ***
  OFFSET: 0x00000000

*** Marker: DQT (xFFDB) ***
  Define a Quantization Table.
  OFFSET: 0x00000002
  Table length = 67
  ----
  Precision=8 bits
  Destination ID=0 (Luminance)
    DQT, Row #0:   8   6   5   8  12  20  26  31 
    DQT, Row #1:   6   6   7  10  13  29  30  28 
    DQT, Row #2:   7   7   8  12  20  29  35  28 
    DQT, Row #3:   7   9  11  15  26  44  40  31 
    DQT, Row #4:   9  11  19  28  34  55  52  39 
    DQT, Row #5:  12  18  28  32  41  52  57  46 
    DQT, Row #6:  25  32  39  44  52  61  60  51 
    DQT, Row #7:  36  46  48  49  56  50  52  50 
    Approx quality factor = 74.75 (scaling=50.51 variance=0.81)

*** Marker: DQT (xFFDB) ***
  Define a Quantization Table.
  OFFSET: 0x00000047
  Table length = 67
  ----
  Precision=8 bits
  Destination ID=1 (Chrominance)
    DQT, Row #0:   9   9  12  24  50  50  50  50 
    DQT, Row #1:   9  11  13  33  50  50  50  50 
    DQT, Row #2:  12  13  28  50  50  50  50  50 
    DQT, Row #3:  24  33  50  50  50  50  50  50 
    DQT, Row #4:  50  50  50  50  50  50  50  50 
    DQT, Row #5:  50  50  50  50  50  50  50  50 
    DQT, Row #6:  50  50  50  50  50  50  50  50 
    DQT, Row #7:  50  50  50  50  50  50  50  50 
    Approx quality factor = 74.74 (scaling=50.52 variance=0.19)

*** Marker: SOF0 (Baseline DCT) (xFFC0) ***
  OFFSET: 0x0000008C
  Frame header length = 20
  Precision = 8
  Number of Lines = 256
  Samples per Line = 256
  Image Size = 256 x 256
  Raw Image Orientation = Landscape
  Number of Img components = 4
    Component[1]: ID=0x01, Samp Fac=0x22 (Subsamp 1 x 1), Quant Tbl Sel=0x00 (C)
    Component[2]: ID=0x02, Samp Fac=0x11 (Subsamp 2 x 2), Quant Tbl Sel=0x01 (M)
    Component[3]: ID=0x03, Samp Fac=0x11 (Subsamp 2 x 2), Quant Tbl Sel=0x01 (Y)
    Component[4]: ID=0x04, Samp Fac=0x22 (Subsamp 1 x 1), Quant Tbl Sel=0x00 (K)

...

jyutzler commented 9 years ago

The file has been updated. Same URL. https://portal.opengeospatial.org/files/?artifact_id=63156 The old version is still available at https://portal.opengeospatial.org/files/?artifact_id=63156&version=1

There is a problem with access control on the new file. It should be completely open to the public. Stay tuned.

jyutzler commented 9 years ago

Access control fixed.

opengeospatial / geopackage

Color issue in Whitehorse GPKG #115