Cover image/thumbnail discussions

[shermp]

Opening this to follow on discussions from #3 & #9 regarding cover images and thumbnails.

[pgaskin]

From https://github.com/shermp/Kobo-UNCaGED/pull/9#pullrequestreview-239750172 regarding my findings about image size in the firmware:

Type	Size
N3_LIBRARY_GRID	149x223
N3_LIBRARY_LIST	60x90
N3_LIBRARY_FULL	355x530
MediumGridList	135x205
iPhoneThumbnail	104x158
NickelBookCover	600x600
Default	600x800

Device Class	N3_FULL
Storm	1264x1680
Daylight	1404x1872
Alyssum / Nova	1072x1448
Dragon	1080x1440
Kraken / Star	758x1024
Phoenix	758x1014
Default	600x800

The images are then scaled to fit within those bounds using Qt::SmoothTransformation (bilinear) and Qt::KeepAspectRatioQt::KeepAspectRatioByExpanding (see above comment thread).

Note that device class does not directly correspond to the device with that codename; it is the first in it's series.

[pgaskin]

From https://github.com/shermp/Kobo-UNCaGED/issues/3#issuecomment-493825841:

Here are the results of the image resize library benchmark:

goos: linux
goarch: amd64
pkg: imgbench
BenchmarkImagingLanczos-4                    500           3932481 ns/op
BenchmarkImagingNearestNeighbor-4           5000            215358 ns/op
BenchmarkImagingCatmullRom-4                 500           2911818 ns/op
BenchmarkImagingMitchellNetravali-4          500           2895267 ns/op
BenchmarkImagingLinear-4                    1000           1996054 ns/op
BenchmarkResizeNearestNeighbor-4             500           2853196 ns/op
BenchmarkResizeBilinear-4                    500           3720607 ns/op
BenchmarkResizeBicubic-4                     300           5141554 ns/op
BenchmarkResizeLanczos2-4                    300           4969040 ns/op
BenchmarkResizeLanczos3-4                    200           6253797 ns/op
BenchmarkResizeMitchellNetravali-4           300           4721741 ns/op
BenchmarkBildLinear-4                        200           8192214 ns/op
BenchmarkBildBox-4                           200           5673657 ns/op
BenchmarkBildGaussian-4                       20          52835551 ns/op
BenchmarkBildMitchellNetravali-4             100          17115462 ns/op
BenchmarkBildCatmullRom-4                    100          17024602 ns/op
BenchmarkBildLanczos-4                        30          59347798 ns/op
BenchmarkRezBicubic-4                       2000            500082 ns/op
BenchmarkRezBilinear-4                      5000            296590 ns/op
BenchmarkRezLanczos2-4                      2000            659241 ns/op
BenchmarkRezLanczos3-4                      2000           1069625 ns/op
PASS
ok      imgbench        38.815s

goos: linux
goarch: arm
pkg: imgbench
BenchmarkImagingLanczos                    1    14652688042 ns/op
BenchmarkImagingNearestNeighbor           20      53550435 ns/op
BenchmarkImagingCatmullRom                 1    9939826377 ns/op
BenchmarkImagingMitchellNetravali          1    9939386627 ns/op
BenchmarkImagingLinear                     1    5314377750 ns/op
BenchmarkResizeNearestNeighbor             1    3381681000 ns/op
BenchmarkResizeBilinear                   10     133334445 ns/op
BenchmarkResizeBicubic                     5     200040091 ns/op
BenchmarkResizeLanczos2                    5     313570208 ns/op
BenchmarkResizeLanczos3                    3     432721861 ns/op
BenchmarkResizeMitchellNetravali           5     204855525 ns/op
BenchmarkBildLinear                        1    12568487042 ns/op
BenchmarkBildBox                           1    6992410877 ns/op
BenchmarkBildGaussian                      1    68470815591 ns/op
BenchmarkBildMitchellNetravali             1    35156644546 ns/op
BenchmarkBildCatmullRom                    1    33081780504 ns/op
BenchmarkBildLanczos                       1    98406429387 ns/op
BenchmarkRezBicubic                       10     158502129 ns/op
BenchmarkRezBilinear                      20      63737918 ns/op
BenchmarkRezLanczos2                       3     382409819 ns/op
BenchmarkRezLanczos3                       2     572310833 ns/op
PASS

It seems that rez is significantly faster on amd64 due to SIMD, imaging's nearestneighbour is the fastest in general, but rez's bilinear is quite close. Bilinear is slightly better quality though.

[shermp]

It looks like to handle covers properly, we really need to save the full cover as well, according to the investigations @NiLuJe made.

This was something I originally had, but canned due to performance concerns, With a faster library though, this could still probably be done.

[NiLuJe]

Do double-check that, as it may just be temporary insanity on the part of my device ;).

[NiLuJe]

If it turns out we do need the full cover, I understand the source is requested from Calibre? What happens if we fudge that and request a fullscreen-sized "thumbnail"? Only having to do downscaling might be a tad faster?

EDIT: In which case, NearestNeighbor is not completely terrible if it's downscale only. It's what CRe had been using since basically forever in KOReader, and it took a random test on a high-dpi non-eInk screen to realize that it was less than great when upscaling and/or downscaling in some circumstances ;p.

I'd obviously prefer something else from a quality standpoint, but, to be fair, on eInk, and for downscaling only, it's really not completely awful ;).

[shermp]

If it turns out we do need the full cover, I understand the source is requested from Calibre? What happens if we fudge that and request a fullscreen-sized "thumbnail"? Only having to do downscaling might be a tad faster?

That's what I was originally doing. But yes, we request a cover size from Calibre, and it delivers. Although I'm not certain what calibre does when the cover image available is smaller than we request...

[NiLuJe]

I'd imagine a QImage scale to honor the requested dimensions, so, smooth & fast, and on the server's side. Hopefully while honoring AR ;).

[shermp]

I'd imagine a QImage scale to honor the requested dimensions, so, smooth & fast, and on the server's side. Hopefully while honoring AR ;).

I wish. It seems the cover dimensions that we send are treated as a maximum. I just checked.

[NiLuJe]

Gah. Well, that's not ideal... Nickel will scale it at display time, though, so it's not a complete loss...

On Tue, May 21, 2019, 06:31 Sherman Perry notifications@github.com wrote:

I'd imagine a QImage scale to honor the requested dimensions, so, smooth & fast, and on the server's side. Hopefully while honoring AR ;).

I wish. It seems the cover dimensions that we send are treated as a maximum. I just checked.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/shermp/Kobo-UNCaGED/issues/16?email_source=notifications&email_token=AAA3KZRZMNDLVIKIM6TS7DTPWN3I5A5CNFSM4HOHEFPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV2W5CI#issuecomment-494235273, or mute the thread https://github.com/notifications/unsubscribe-auth/AAA3KZQIOTR5EHN5EWGSYXTPWN3I5ANCNFSM4HOHEFPA .

[davidfor]

For the record, as it was mentioned in the pull request discussion, the cover image size in calibre KoboTouch driver comes from observation. Sideload a book and let the device generate the covers and check the sizes. I check them occasionally, but not for a few firmware versions, The one I do need to check is the Forma as I suspect I have the screen size rather than the image size.

Also, be careful if you don't create the full screen cover but do create the others. The last time I checked, the firmware won't generate the full screen cover if the smaller covers exist. One of the smaller ones gets used and it looks terrible. Of course, the space saving might be considered a good thing.

[NiLuJe]

@davidfor : Thanks for the confirmation about the missing thumbnails, I knew this sounded familiar ;).

Spoiler alert about the Forma: Nickel is doing it wrong, and generating (and downloading for store-bought KePubs) stuff that's KA1 sized, which is just plain wrong.

[NiLuJe]

@geek1011 : Okay, confirmed that, at least for downloaded thumbnails for KePubs, that's indeed Qt::KeepAspectRatioByExpanding ;).

def thumb_scale_qt(orig_dims, dims, expand=False):
    w = orig_dims[0]
    h = orig_dims[1]
    scaled_width = dims[0]
    scaled_height = dims[1]

    rw = int(float(scaled_height) * float(w) / float(h) + 0.5)

    if expand:
        useHeight = (rw >= scaled_width)
    else:
        useHeight = (rw <= scaled_width)

    if useHeight:
        return (rw, scaled_height)
    else:
        return (scaled_width, int(float(scaled_width) * float(h) / float(w) + 0.5));

print
cover_dims = (1197, 1872)
print("FULL: {} -> {}".format(cover_dims, thumb_scale_qt(cover_dims, (1080, 1429), False)))
print("LIB_FULL: {} -> {} -> {}".format(cover_dims, thumb_scale_qt(cover_dims, (355, 530), False), thumb_scale_qt(cover_dims, (355, 530), True)))
print("GRID_FULL: {} -> {} -> {}".format(cover_dims, thumb_scale_qt(cover_dims, (149, 223), False), thumb_scale_qt(cover_dims, (149, 223), True)))

Yields

FULL: (1197, 1872) -> (914, 1429)
LIB_FULL: (1197, 1872) -> (339, 530) -> (355, 555)
GRID_FULL: (1197, 1872) -> (143, 223) -> (149, 233)

Which matches that 355x555 I mentioned earlier ;).

I'm going to sideload that book manually to see if Nickel internally behaves differently when it's generating the thumbnails instead of downloading them, but, as we said before, I prefer this behavior anyway, so, it's basically just FOR SCIENCE§! :D.

EDIT: Yeah, judging from the thumbnails generated from our trigger PNGs, I'd say it's using Qt::KeepAspectRatio when generating (+/- rounding errors), which would be consistent with my previous observations that downloaded thumbnails tended to be larger than generated thumbnails ;).

i.e.,

FULL: (1440, 1920) -> (1072, 1429)
LIB_FULL: (1440, 1920) -> (355, 473) -> (398, 530)
GRID_FULL: (1440, 1920) -> (149, 199) -> (167, 223)

While the on-device files are 355x473 & 149x198 ;).

[shermp]

I got around to putting @geek1011 benchmark results (ARM) into spreadsheet to sort them, and convert to units I can better understand (milliseconds).

First, I'm really glad I decided not to use Bild!

Second, compared to Bild, the algorithm currently used is way faster. Still took over 5000 ms/op though, which is definitely unacceptable.

Assuming the benchmark is representative, and not an outlier, I would be inclined to use nfnt/resize. In the grand scheme of things, it isn't that much slower than rez, and the API appears simpler.

[shermp]

So, I ran the benchmark provided by @geek1011 on my H2O, with a random, relatively high resolution cover from my Calibre library. Note, I canceled my first benchmark run halfway through and restarted without benching Bild, cause I didn't want to spend all night benchmarking :p

cover.jpg    1512x2200

goos: linux
goarch: arm
BenchmarkImagingLanczos                        1        122358313999 ns/op
BenchmarkImagingNearestNeighbor               20          65517306 ns/op
BenchmarkImagingCatmullRom                     1        79850543250 ns/op
BenchmarkImagingMitchellNetravali              1        80190022250 ns/op
BenchmarkImagingLinear                         1        41102400250 ns/op
BenchmarkResizeNearestNeighbor                 1        24202448750 ns/op
BenchmarkResizeBilinear                        2         967996437 ns/op
BenchmarkResizeBicubic                         1        1630968375 ns/op
BenchmarkResizeLanczos2                        1        2006728000 ns/op
BenchmarkResizeLanczos3                        1        2764296250 ns/op
BenchmarkResizeMitchellNetravali               1        1647441000 ns/op
BenchmarkRezBicubic                            2         883429687 ns/op
BenchmarkRezBilinear                           3         451669125 ns/op
BenchmarkRezLanczos2                           1        1627825750 ns/op
BenchmarkRezLanczos3                           1        2414422625 ns/op

[shermp]

And here's the sorted results:

Benchmark	Ops	ns/ops	ms/ops
BenchmarkImagingNearestNeighbor	20	65517306.00	65
BenchmarkRezBilinear	3	451669125.00	451
BenchmarkRezBicubic	2	883429687.00	883
BenchmarkResizeBilinear	2	967996437.00	967
BenchmarkRezLanczos2	1	1627825750.00	1627
BenchmarkResizeBicubic	1	1630968375.00	1630
BenchmarkResizeMitchellNetravali	1	1647441000.00	1647
BenchmarkResizeLanczos2	1	2006728000.00	2006
BenchmarkRezLanczos3	1	2414422625.00	2414
BenchmarkResizeLanczos3	1	2764296250.00	2764
BenchmarkResizeNearestNeighbor	1	24202448750.00	24202
BenchmarkImagingLinear	1	41102400250.00	41102
BenchmarkImagingCatmullRom	1	79850543250.00	79850
BenchmarkImagingMitchellNetravali	1	80190022250.00	80190
BenchmarkImagingLanczos	1	122358313999.00	122358

With a bigger and/or more complex image, the performance gap between rez and nfnt/resize definitely increases.

Whatever library/algorithm gets chosen, I think we will need to generate N3_LIBRARY_GRID from N3_LIBRARY_FULL, and not from the original cover.

[pgaskin]

Do you want to generate each image from the previous one?

[NiLuJe]

I'm afraid scaling LIB_GRID from LIB_FULL risks generating mush, no matter the algorithm ;).

EDIT: Or not! See below :).

[NiLuJe]

To answer an earlier question from @shermp, I'm clearly not opposed to settings ;).

In fact, having an option to disable thumbnail generation entirely could be desirable, in cases you want to let Nickel handle it, our you're using another tool to do it for you later anyway (KoboUtilities plugin, for instance).

[NiLuJe]

@shermp: Okay, that insane 41s does look familiar ;p.

May 22 15:28:47 KoboUNCaGED[1350]: 2019/05/22 15:28:47 Starting Calibre Connection
May 22 15:29:04 KoboUNCaGED[1350]: 2019/05/22 15:29:04 Resizing (903,1429) cover to (903,1429) (target (1080,1429)) for N3_FULL . . .
May 22 15:29:04 KoboUNCaGED[1350]: 2019/05/22 15:29:04  -- Skipped resize: already correct size
May 22 15:29:05 KoboUNCaGED[1350]: 2019/05/22 15:29:05 Resizing (903,1429) cover to (355,561) (target (355,530)) for N3_LIBRARY_FULL . . .
May 22 15:29:31 KoboUNCaGED[1350]: 2019/05/22 15:29:31  -- Resized to (355,561)
May 22 15:29:31 KoboUNCaGED[1350]: 2019/05/22 15:29:31 Resizing (903,1429) cover to (149,235) (target (149,223)) for N3_LIBRARY_GRID . . .
May 22 15:29:49 KoboUNCaGED[1350]: 2019/05/22 15:29:49  -- Resized to (149,235)

---

For fun, here's how long the FULL -> LIB_FULL scaling takes with a couple other things:

FBInk (which, okay, does a rendering pass instead of an encoding pass ;p):

190ms to render the FULL, 210ms to render the downscaled LIBRARY_FULL

From my earlier experiments, much of that time is actually spent in the decoding pass, not the scaling or rendering one ;p.

IM (with a classic -filter Triangle -resize 355x, not a bells'n whistle EWA Distort in linear light, because that's hilariously slow):

1000ms

(Triangle should be Bilinear. This jumps to roughly 1.5s if we switch to Mitchell, and 2s w/ Lanczos. It drops to 590ms with Point (NearestNeighbor)).

(Displaying that with FBInk then takes 90ms, as another data point).

---

IM should pretty much be considered a worst-case, as its goal is never speed ;).

But from my various experiences using it on Kindle for the ScreenSaver hack, I rarely saw it spinning for more than 4 or 5s on a heavy screen-sized rescale, and that was w/ Lanczos + error-diffusion dithering. Most of the time it's in the 2s range.

[NiLuJe]

I may have been a bit harsh in my earlier comment ;).

Scaling GRID from LIB_FULL is indeed noticeably faster, without being noticeably worse looking.

For ref., with IM, Triangle:

From Full (750ms):

From Lib (235ms):

[NiLuJe]

The only quirk is potentially additional rounding errors because of the lack of rounding in resizeKeepAspectRatio ;).

[pgaskin]

In fact, having an option to disable thumbnail generation entirely could be desirable, in cases you want to let Nickel handle it, our you're using another tool to do it for you later anyway (KoboUtilities plugin, for instance).

That should be pretty easy to implement, if @shermp agrees.

Scaling GRID from LIB_FULL is indeed noticeably faster, without being noticeably worse looking.

That seems pretty useful. I'm going to add this when I get back to a proper computer. It'll probably be as simple as directly editing img instead of copying to nimg.

The only quirk is potentially additional rounding errors because of the lack of rounding in resizeKeepAspectRatio ;).

Hey, I'm just copying Qt's implementation. 😈 Edit: Also, note that the target sizes could be all calculated from the original, then you only have the rounding errors from one resize.

I'm going to test the quality of the different libs and algorithms with multiple resizes later today (or tomorrow).

[shermp]

@geek1011 do you still want to run your experiments? Or should I just implement rez and call it a day?

[pgaskin]

I won't be able to get to it today, so I'd probably suggest just switching to rez for now, and we can make more improvements later if necessary.

[shermp]

Sounds good. Thanks for the update.

[pgaskin]

We'll need values for the Libra. If @NiLuJe doesn't find them first, I'll do them later this week.

[NiLuJe]

Oh, that's easy, as we now only need the screen's resolution ;).

And that's 1264x1680 ;).

[pgaskin]

And that's 1264x1680 ;).

Confirmed: I've updated the table at the top, and will open a PR.

[pgaskin]

I decided to check the firmware for values again, and here is the first part (I'm putting them here for now for lack of a better place):

Here are the values in code order for various device-specific stuff in nickel as of 13737. These are exactly as they appear in libnickel, without adding missing values to make it usable.

There seem to be two codename types: one for identifying individual devices, and an abbreviated one for things like CSS.

Device::getDeviceClassString(), mapping from Device::is* to a string, falls back to Device::codeName when not matching anything
trilogy, Kobo Touch
kraken, Kobo Glo
pixie, Kobo Mini
phoenix, Kobo Aura
dragon, Kobo Aura HD
dahlia, Kobo Aura H2O
alyssum, Kobo Glo HD
pika, Kobo Touch 2.0
daylight, Kobo Aura ONE
star, Kobo Aura
storm, Kobo Libra H2O
snow, Kobo Aura H2O Edition 2
nova, Kobo Clara HD
frost, Kobo Forma
Device::codeName == desktop, Kobo Desktop
Device::codeName == vox, Kobo Vox
Device::codeName == merch, Literati / LookBook eReader
Device::codeName == nickel1, Kobo eReader
Device::codeName == nickel2, Kobo Wireless eReader

Device::codeName(QString), mapping from device id to string
1 2 3 4 5 10 11 12, desktop
210, nickel2
100 200 201, nickel1
300 301, merch
310 320 340 372, trilogy
350 370 371 374 376 384 378, dragon
373 380 381 377, daylight
330 360 375 379, phoenix
511, vox
*, unknown

Device::codeName(bool), idk what the bool is for, mapping from device id to string, seems to be missing values, Device::is* matches up with this when assuming the missing values to be the remainders of Device::codeName(QString) (for example 377 and 380 are frost, but the 373 and 381 are daylight)
330, kraken
340, pixie
370, dahlia
371, alyssum
372, pika
374 378, snow
375 379, star
376, nova
377 380, frost
384, storm

Image::sizeForType(Device), mapping from Device::is* to cover size for N3_FULL, fixed for others
N3_LIBRARY_FULL, 355x530
N3_LIBRARY_LIST, 60x90
N3_LIBRARY_GRID, 149x223
MediumGridList, 135x205
iPhoneThumbnail, 104x158
NickelBookCover, 600x600
Default, 600x800
N3_FULL
storm, 1264x1680
daylight, 1404x1872
alyssum nova, 1072x1448
dragon, 1080x1440
kraken star, 758x1024
phoenix, 758x1014
*, 600x800

I'll sort through these in a bit to put them into a usable form, and I'll probably add them to my koboutils repo (and KoboStuff whenever I get around to it).

[pgaskin]

Here's what I have so far (everything to do with codenames, classnames, familynames, deviceids, and devicenames is done, I'll be adding specs and so on as I need them):

1. create top-level classes from Device::codeName(QString)
2. split into legacy and current based on if class in Device::getDeviceClassString() is based on Device::codeName
3. add legacy families under top-level classes, codename=classname and familyname is from Device::getDeviceClassString()
4. add legacy device ids under families from Device::codeName(QString)
5. skip current families for now, add device ids from Device::codeName(QString)
6. add device names from known values to those devices
7. split current devices into families based on Device::codeName(bool), remaining ones have a matching classname
8. add family names from Device::getDeviceClassString()

LEGACY DEVICES

classname=desktop
    codename=desktop    familyname=Kobo Desktop
        deviceid=1
        deviceid=2
        deviceid=3
        deviceid=4
        deviceid=5
        deviceid=10
        deviceid=11
        deviceid=12

classname=nickel1
    codename=nickel1    familyname=Kobo eReader
        deviceid=100
        deviceid=200
        deviceid=201

classname=nickel2
    codename=nickel2    familyname=Kobo Wireless eReader
        deviceid=210

classname=merch
    codename=merch      familyname=Literati / LookBook eReader
        deviceid=300
        deviceid=301

classname=vox
    codename=vox        familyname=Kobo Vox
        deviceid=511

CURRENT DEVICES

classname=trilogy
    codename=trilogy    familyname=Kobo Touch
        deviceid=310    devicename=Kobo Touch A/B
        deviceid=320    devicename=Kobo Touch C
    codename=pixie      familyname=Kobo Mini
        deviceid=340    devicename=Kobo Mini
    codename=pika       familyname=Kobo Touch 2.0
        deviceid=372    devicename=Kobo Touch 2.0

classname=dragon
    classname=dragon    familyname=Kobo Aura HD
        deviceid=350    devicename=Kobo Aura HD
    codename=dahlia     familyname=Kobo Aura H2O
        deviceid=370    devicename=Kobo Aura H2O
    codename=alyssum    familyname=Kobo Glo HD
        deviceid=371    devicename=Kobo Glo HD
    codename=snow       familyname=Kobo Aura H2O Edition 2
        deviceid=374    devicename=Kobo Aura H2O Edition 2 v1
        deviceid=378    devicename=Kobo Aura H2O Edition 2 v2
    codename=nova       familyname=Kobo Clara HD
        deviceid=376    devicename=Kobo Clara HD
    codename=storm      familyname=Kobo Libra H2O
        deviceid=384    devicename=Kobo Libra H2O

classname=daylight
    codename=daylight   familyname=Kobo Aura ONE
        deviceid=373    devicename=Kobo Aura ONE
        deviceid=381    devicename=Kobo Aura ONE Limited Edition
    codename=frost      familyname=Kobo Forma
        deviceid=380    devicename=Kobo Forma 32GB
        deviceid=377    devicename=Kobo Forma

classname=phoenix
    codename=phoenix    familyname=Kobo Aura
        deviceid=360    devicename=Kobo Aura
    codename=kraken     familyname=Kobo Glo
        deviceid=330    devicename=Kobo Glo
    codename=star       familyname=Kobo Aura
        deviceid=375    devicename=Kobo Aura Edition 2 v1
        deviceid=379    devicename=Kobo Aura Edition 2 v2

By doing this, I think I finally completely understand how the hierarchy of Kobo devices works. @NiLuJe, you might find this interesting.

[shermp]

Wow, that's a bit of a tangled web!

[pgaskin]

Here's a nicer version, which I'm working on updating: https://gist.github.com/geek1011/613b34c23f026f7c39c50ee32f5e167e

Update: and it turns out there are third-level codenames too, like frost32 and superDaylight

[pgaskin]

Oh, and this will probably also interest @jackiew1, as the top-level device classes I found are used for the CSS too.