jonsneyers / jonsneyers.github.io

0 stars 0 forks source link

Discussion about the future of image codecs #1

Open LifeIsStrange opened 2 years ago

LifeIsStrange commented 2 years ago

Hi @jonsneyers :) Feel free to close the issue, I am simply curious about your perspective on future possible enhancements for an hypothetical sucessor to JPEG XL. Notably I think machine learning super resolution/compression has a lot of potential see e.g https://github.com/bilibili/ailab/blob/main/Real-CUGAN/README_EN.md

jonsneyers commented 2 years ago

Sorry, completely missed this one.

Yes, AI based codecs are very promising, but also somewhat scary in the sense that the whole notion of image quality and fidelity gets challenged and the impact of bias in training sets is hard to estimate but probably not safe to ignore. In that sense, I think classical codecs will remain to have a place, even when AI based codecs become more feasible (in terms of computational resources). On the other hand, besides compression, AI based codecs could also bring other advantages, e.g. image processing and analysis in the latent domain as opposed to the pixel domain.

JPEG XL was designed to be pretty future-proof in the sense that it is ready for HDR, wide gamut, new kinds of channels besides color and alpha (e.g. depth, thermal, etc), supports layers, etc. In terms of functionality, I do not currently anticipate any new needs that will not already be covered by JPEG XL — if there would have been anything I could think of, I would have put it in JPEG XL :). But of course innovation cannot be predicted, so we'll see what the future will bring. For now though, I think it's a bit early to think about potential successors to JPEG XL, at least as far as I'm concerned. If you or anyone else does see new use cases and requirements, then of course feel free to share them so we can see what could be missing in jxl and perhaps be something for a successor.

There are also several extension mechanisms in jxl, so ideally a successor would use those and gracefully degrade when decoded by existing jxl decoders (that don't implement the new extensions).

LifeIsStrange commented 2 years ago

Those are excellent points ! Indeed AI super resolution challenge has downsides such as tendency to bias/generalization for out of domain textures, and challenge current metrics of quality assesment. It is great to hear that JPEG XL is "feature complete" and it is amazing to hear that there are extension mechanisms.

LifeIsStrange commented 2 years ago

@jonsneyers The future of photography is the encoding of light fields information, it allows as a post processing step to e.g choose the depth of field (blurry background or blurry foreground or no blur) https://en.wikipedia.org/wiki/Light_field_camera https://en.wikipedia.org/wiki/Light_field_camera#/media/File:Lytro_Illum_light_field_camera_demonstration.jpg

I really hope that JPEG XL can currently properly encode those data/metadata and set the desired blur mode at runtime dynamically or at least that it is on your roadmap, as it is the biggest advance in photography of the decade.