Large images

[frank-actialuna]

Large images

Some formats (webtoons...) are laid out as a single vertically scrolling page. Therefore we need to support image strips with a large dimension.

The JPEG file format limits the image size to 64k pixels in each dimension. Furthermore, unless encoded specifically (progressive JPEG...), the whole file is required in order to start decoding the image.

HEIF/HEIC

HEIF provides a way to circumvent the image size limitation (virtually unlimited image size), and the format is intrinsically tiled (default tile size = 512x512 pixels). This approach has a few drawbacks:

• a recent OS (or an external lib) is needed.
• HEIF decoding may require hardware support to achieve reasonable speed.
• even though the file format is tiled internally, all decoding libs expect random access to a local file. That makes streaming more difficult: the tiles are to be extracted and served separately, typically by using offsets in HTTP requests. A solution to this issue could involve developing a streaming layer for HEIF images, similar to the HTTP Live Streaming File Segmenter.
• patents on HEVC: HEIF itself (as a container) seems to be exempt of patents, but some codecs (HEVC) are encumbered with patents.

[
  {
    "type": "image/heic",
    "width": 1200,
    "height": 25426,
    "href": "page1.heic"
  }
]

Google WebP

• Raw WebP limit: 16k pixels.
• WebP-Mux (RIFF-based) for tiling support.
• Similar issues as with HEIF regarding tile streaming.
• no patents, open source implementation under BSD license.
• support in browsers other than Chrome?

Explicit tiles

Another way to support large images is with explicit tiling.

[
  {
    "type": "multipart/related",
    "rel": "tiles",
    "width": 1200,
    "height": 25426,
    "children": [
      {
        "href": "page1_1.jpg",
        "type": "image/jpeg",
        "top": 0,
        "left": 0,
        "width": 1200,
        "height": 600
      },
      {
        "href": "page1_2.jpg",
        "type": "image/jpeg",
        "top": 600,
        "left": 0,
        "width": 1200,
        "height": 600
      },
      ...
    ]
  }
]

Fallback image

Could be useful even with explicit tiling, to help generate thumbnails.

    "poster": {
      "href": "page1_poster.jpg",
      "type": "image/jpeg",
      "width": 120,
      "height": 2543
    }

[llemeurfr]

I can't imagine that we can impose an emerging file format, as HEIF. In order to get the desired effet (a long vertical strip, people reading sequentially from top to bottom), lazy loading of images seems a good trick. A look at google images or https://unsplash.com/ seems convincing enough to me. Re. authoring, an explicit tiling like the one proposed by Frank would fit IMO.

[HadrienGardeur]

Regarding streaming do we really need something specific like HTTP live streaming (which means serving multiple resolutions, server specific settings and index files) intsead of a simple HTTP byte range?

If Apple and Google are truly pushing for these formats they should IMO follow a fairly standard behaviour. Large media files are usually streamed in a browser context using byte ranges.

[HadrienGardeur]

My preference would be to rely on existing bitmap formats and include alternate representations using the latest/upcoming image formats:

[
  {
    "href": "page1.jpg",
    "type": "image/jpeg",
    "alternate": [{
      "type": "image/heic",
      "href": "page1.heic"
    }]
  }
]

[JayPanoz]

I’d like to point out that memory is a technical issue which should probably be discussed if web browsers/views are involved at some point.

Browser vendors indeed tend to consider images a hidden cost (Apple even pointed that in a WWDC session) and what’s super important to them in the size of the image decoded in memory. Now, things can go insanely wrong in some cases (web browsers/view crashing, image not being entirely decoded on low-end mobile devices, etc.), which is why there are image sizing limitations in EPUB Readers for instance.

One of their advice is to not use “images larger than necessary”, which reminds me a lot of JS slideshows plugins/libs heavily rely on picture/srcset (responsive options being mapped to this spec), in addition to lazy-loading, which is not sufficient – it appears that in this case, if too many large images are to be displayed for instance, decoded images won’t necessarily be trashed as expected in browsers’ vanilla scrolled view.

In the case of JPG, PNG and GIF, the formula is width × height × 4. Don’t know what it is for HEIF or WebP but I guess it’s an issue worth-addressing early on if, once again, web browsers/web views end up involved at some point. That’s not necessarily an edge-case by the way (a significant amount of web articles with a lot of images/GIFs can crash iOS Safari for instance).

I can already imagine web browser vendors scream at the 1200px × 25426px images example, which would be 122 044 800 bytes each, decoded in memory – it would actually crash an iPad Mini 2 for instance. Which is also a reason why explicit titling is worth exploring.

So, for political reasons, etc.