OSD IIIF does not attempt all sizes on profile level 0 if non-power of 2

[janhoy]

This bug relates to #1561 but I create a separate issue instead of commenting on that old one.

I have a similar issue with profile level0. One of our image sources (we have several) is a photo mgmt system that serves many different pre-cached resolutions, but they don't support IIIF. So we have a backend that acts as middle man and exposes an IIIF endpoint for that service by constructing a valid IIIF level0 info.json to OSD. Here is a sample:

{
    "protocol": "http://iiif.io/api/image",
    "sizes": [{
        "width": 200,
        "height": 177
    }, {
        "width": 300,
        "height": 266
    }, {
        "width": 400,
        "height": 354
    }, {
        "width": 600,
        "height": 532
    }, {
        "width": 800,
        "height": 709
    }, {
        "width": 1000,
        "height": 886
    }, {
        "width": 1200,
        "height": 1063
    }, {
        "width": 1600,
        "height": 1417
    }, {
        "width": 2400,
        "height": 2126
    }],
    "profile": ["http://iiif.io/api/image/2/level0.json"],
    "width": 2400,
    "@id": "https://backend.example.com/v2/images/ra_sb:GL2ZvdG93ZWIvYXJjaGl2ZXMvNTAzMy1TYWtzYmVoYW5kbGVyYmlsZGVyL1Nha3NiZWhhbmRsZXJiaWxkZXIvQUFNL1Ryb25kaGVpbS8yMDIwYXVnL05pZGFyb3Nkb21lbl8yMDIwMDgyN18wMi50aWYuaW5mbw==",
    "X-Spor-Scaled": true,
    "@context": "http://iiif.io/api/image/2/context.json",
    "height": 2128
}

This apparently works well, our OSD first request three resolutions:

/full/600,/0/default.jpg
/full/200,/0/default.jpg
/full/300,/0/default.jpg

Then, when user starts clicking and zooming, OSD requests the 800x709 size (size number 5):

/full/800,/0/default.jpg

However, OSD stops the zoom there. No further requests are sent and you cannot zoom deeper. I'd expect it to continue with the 1000x886, 1200x1063 etc. Also, the 800x709 image is blurry, obviously a too low resolution for that zoom level.

We use OSD version 4.0.0 in a React app. No errors in the browser developer console.

I have not inspected OSD souce code, but it appears that it caps the size array at 5?

[pearcetm]

Hmm, not sure why that would be happening. You could try setting a breakpoint in iiiftilesource.js here:

https://github.com/openseadragon/openseadragon/blob/640526b444536dccb2c78eec7a0b4b7eeb1eb43c/src/iiiftilesource.js#L114-L131

This will let you look at how many levels are being created, which would define maxLevel as well.

One more thing, and I'm not sure if/why this would make a difference, but in your config, height is not the same as the height of the largest size. Just something to look into...

[janhoy]

Thanks for answering. I debugged and stepped through the code, and indeed it creates 9 levels and maxLevel=8. I tested the same image some more, zoomed some out and in, and suddenly it fetched the 1000px version, and then after some more time it fetched the 1200px one. But never 1600 or 2400, even if I try to zoom more. This suggests a bug somewhere else in the handling of level0 or the emulateLegacyImagePyramid logic. I see there are 7 if-clauses handling emulateLegacyImagePyramid in iiiftilesource.js, so guess there is room for bugs?

Next step would be for another developer to reproduce. You don't need an IIIF server, only a server in nodejs or python that listens to the URL provided in @id, and serves a dummy image for any sub-path requested.

[janhoy]

Do you think you have enough to reproduce this? I'm not a frontend developer so I don't think there will be a PR from me :)

[pearcetm]

I'm not very knowledgeable about IIIF. Perhaps an expert like @ahankinson @ruven @azaroth42 or @tomcrane could take a look?

[janhoy]

I made a small reproduction repository at https://github.com/janhoy/osd-repro

It will easily let you point your OSD instance of choice to tileSource "http://localhost:3000/iiif/2/demo/info.json" to test

[pearcetm]

@janhoy the link to your repro repository takes me to a 404 page. Perhaps its inadvertently set to private?

[janhoy]

Ah, sorry. I created the repo from my IntelliJ and did not know that it would be private. Just made it public.

[pearcetm]

Thanks, that's working now.

Looks like the issue is that the pyramid you're using has non-power-of-two increments in the sizes of the levels, and the calculations in TiledImage assume power of two:

https://github.com/openseadragon/openseadragon/blob/640526b444536dccb2c78eec7a0b4b7eeb1eb43c/src/tiledimage.js#L1153-L1166

In particular, the max level to request (calculated base on the pixel ratio) is done purely based on powers of two from the dimensions of level zero.

I think this issue is related to https://github.com/openseadragon/openseadragon/issues/1344 too. If a TileSource implementation has fewer levels than OSD expects (because of the powers of two assumption) it will load high-res tiles too early. In your case here, there are more levels than OSD expects (or to put it another way, all levels with level >= 1 are smaller than it expects them to be). So, it's requesting the level it thinks should give it the right pixel ratio, but it's not.

I'm not 100% sure what the best approach is here, but it seems like perhaps the calculation of which level to use for a given zoom should be delegated to the TileSource implementation, so that when level sizes are explicitly available they can be used appropriately.

In the mean time for this specific case at least, you can set the minPixelRatio to a lower value so OSD will request larger levels sooner. In your repro in my browser, minPixelRatio: 0.24 or less does the trick in getting the viewer to request the highest-resolution image. Of course, this would vary by image and is definitely not as robust as an actual fix!

[janhoy]

Thanks for the advice and workaround. In my case I have an image software that does not support IIIF or deepzoom, but they support a number of pre-cached sizes, which are 200, 300, 400, 600... i.e. not the typical doubling between each size.

The IIIF image spec for “sizes” (https://iiif.io/api/image/2.0/#image-information) says:

“A set of height and width pairs the client should use in the size parameter to request complete images at different sizes that the server has available. This may be used to let a client know the sizes that are available when the server does not support requests for arbitrary sizes, or simply as a hint that requesting an image of this size may result in a faster response. A request constructed with the w,h syntax using these sizes must be supported by the server, even if arbitrary width and height are not.”

I.e. the spec does not limit the "sizes" objects to any rule of ratio between them, rather it just lets the client know what sizes are available.

I'm not 100% sure what the best approach is here, but it seems like perhaps the calculation of which level to use for a given zoom should be delegated to the TileSource implementation, so that when level sizes are explicitly available they can be used appropriately.

That sounds fair.

In the meantime I'll try your minPixelRatio.

[janhoy]

I just tested the workaround with minPixelRatio: 0.24 on our real data, and it managed to climb from level 6 to level 7 of our 9-levels pyramid, but unfortunately not to the top.

So I'll try a backend workaround instead, by removing from the sizes arry all sizes that are not power of 2 of each other, starting from the largest number. Hopefully this will work until a permanent fix is ready in OSD.

[iangilman]

I agree this is related to https://github.com/openseadragon/openseadragon/issues/1344. We should support non-power of 2 levels, and in fact we already do with the LegacyTileSource, but that one doesn't support tiled levels.

Whoever wants to work on this should look at LegacyTileSource to see how it works.

[pearcetm]

in fact we already do with the LegacyTileSource

@iangilman I'm not so sure that LegacyTileSource actually works any better, because the code in TiledImage that calculates which level to request ignores the explicit width defined by each level of the tile source. It's hardcoded there to use powers of two.

[janhoy]

If each zoom level needs double resolution, then it should be fairly simple to traverse the (sorted) sizes array and pick the next size in the list that is equal to or larger than what you request. And I guess you could calculate max zoom level as well from such a formula? Have not looked at the code, so it may be harder than it seems...

[iangilman]

@pearcetm Oh, interesting! I guess I haven't looked closely at that code. Thank you for mentioning it.

@janhoy That's an interesting approach. Clearly it's working for LegacyTileSource. I wonder if it'll run into problems because the levels in IIIF are tiled, unlike LegacyTileSource. Worth a try, though.

And I don't really know how necessary the resolution doubling is... It's just how the system was built originally (and if you have control over the data, it seems like a pretty good way to build your pyramid). It may be possible to rework the system to not have that assumption, as deeply embedded as it is.

[pearcetm]

For every frame that is to be drawn, the best/highest resolution that the viewer will request is calculated based on powers of two from the lowest-res level (level 0). This is calculated here:

https://github.com/openseadragon/openseadragon/blob/640526b444536dccb2c78eec7a0b4b7eeb1eb43c/src/tiledimage.js#L1164

I.e. "I need level X" where X is calculated purely on the ratio of the desired size versus the size of level 0.

If a tile source (including a legacy tile source) has large jumps in the sizes of each level (> 2x width from the previous level), the viewer will end up requesting the higher-res images earlier than it needs to. If your level 0 has width=200" then OSD will request level 1 when it wants something withwidth>= 400, level 2 when it wantswidth>=800` etc. If your "pyramid" has just three images and the highest-res image (level 2) has width of 8000 pixels, it will still request level 2 as soon as it wants just 800 pixels. My guess is the reason this hasn't cropped up as an issue is that, other than increased network/memory load, this really wouldn't cause visual problems since the image would visually appear appropriate (once scaled down) at all zoom levels.

@janhoy's problem is the opposite case - since the scale between levels is less than 2x per level, when OSD requests (for example) level 4, it expects the tilesource to give it an image appropriate for 4 powers of two (i.e. 16x the level 0 width), but instead it gets an image with just 800px width (4x resolution), and thus is visually apparent as too low of a resolution.

That's an interesting approach. Clearly it's working for LegacyTileSource. I wonder if it'll run into problems because the levels in IIIF are tiled, unlike LegacyTileSource. Worth a try, though.

You could in theory implement this in a TileSource of any type (tiled or not) - in getTileUrl you'd just need to use the level parameter to calculate the size that OSD is requesting based on powers-of-two above level 0, and instead of using the requested level replace it with the computed most appropriate level for your pyramid.

It may be possible to rework the system to not have that assumption, as deeply embedded as it is.

I think ultimately this would be a better (cleaner, more understandable) solution. It would require refactoring the code in TiledImage and in TileSource to allow OSD to request a certain width, and let TileSource implementations decide which level of their pyramid that should correspond to.

[iangilman]

@pearcetm Well explained... Thank you for the analysis!

You could in theory implement this in a TileSource of any type (tiled or not) - in getTileUrl you'd just need to use the level parameter to calculate the size that OSD is requesting based on powers-of-two above level 0, and instead of using the requested level replace it with the computed most appropriate level for your pyramid.

That seems like a good solution for the IIIF issue... don't assume the sizes align with our levels.

I think ultimately this would be a better (cleaner, more understandable) solution. It would require refactoring the code in TiledImage and in TileSource to allow OSD to request a certain width, and let TileSource implementations decide which level of their pyramid that should correspond to.

I don't know... I think the powers of 2 thing is a good foundation, and I think these tilesets that don't match that are rare enough that we shouldn't be redoing our basics, especially if we have another way to accommodate them.

Otherwise, I suppose we could be a little more efficient with memory/transfer in cases where we would otherwise be loading too big of an image, but we would also be less efficient in cases where we would be loading tons of images that are close to each other in size. I think it is a wash.