ImageMagick / ImageMagick

🧙‍♂️ ImageMagick 7
https://imagemagick.org
Other
12.03k stars 1.35k forks source link

Add support for HDR gain maps #6377

Open gregbenz opened 1 year ago

gregbenz commented 1 year ago

Is your feature request related to a problem? Please describe.

HDR (high dynamic range) is a significant leap forward in image quality on supporting monitors.

One of the challenges is to create an image which shows as an ideal HDR where possible, but safely falls back to showing SDR (standard dynamic range) where not supported. A new "gain map" standard (https://helpx.adobe.com/camera-raw/using/gain-map.html) offers this and is already supported by Chrome Canary v116 (including both MacOS / Windows desktop clients and the Android 14 beta). Enable the dev flag for gain maps via chrome://flags/#gainmap-hdr-images.

When images are reprocessed (such as by ImageMagick for posting on a Wordpress site via the media library), it is important that the gain map meta data and auxiliary image are retained. And if the image is resized/cropped, the secondary image should be altered in a consistent way.

When a JPG with a gain map is processed with ImageMagick now (ImageMagick 6.9.12-86 Q16 x86_64 17814), the primary image metadata (XMP-hdrgm) is retained, but the secondary image is discarded.

Describe the solution you'd like

Describe alternatives you've considered

Direct posting of the image without support for derivative assets like thumbnails or use of the media library. This could broadly affect a wide range of sites managing such images, so lack of support is very limiting and would require exporting multiple versions of images (if the platform even supports bypassing asset management which may reprocess the file).

Additional context

See https://gregbenzphotography.com/hdr/ for examples, monitor tests, and more info on HDR images.

See https://helpx.adobe.com/camera-raw/using/gain-map.html for sample gain map images in JPG, AVIF, and JXL formats.

snibgo commented 1 year ago

I can't find any image files with gain maps on the Greg Benz site, or elsewhere. Can you point to some?

When an image has an associated gain map (GM), I can see that cropping and resizing the GM in parallel to the main image would be useful. For ImageMagick (IM), one difficulty is that the architecture is designed to process individual images, or lists of images. IM doesn't have a concept of "associated" images.

Another difficulty is: why limit parallel operations to cropping and resizing? How about general -distort operations? Sharpening? And many others?

Perhaps as a first step, IM could include an option to extract a gain map from a file, adding the gain map to its usual list of images. It could have another option to insert a gain map into a file. It would also read/write the GM metadata. The next step is the fairly simple image processing: given 2 out of 3 images (SDR, HDR, GM), make the third.

gregbenz commented 1 year ago

@snibgo There are some gain map examples under the Adobe link (direct download is https://www.adobe.com/go/gainmap_sample_photos).

Your question on other operations is a good one. I would say the support I outlined above would probably cover most needs (certainly the most critical ones for sharing images on the web). However, those additional operations would likely be useful to others (especially sharpening if some service wants to use that to offset softness caused by resizing). Personally, I would probably not invoke such options. My emphasis is simply on derivative versions of a finished image, especially for use with the WordPress media library.

I would imagine that internally extracting the gain map, scaling/cropping, retaining HDR metadata, and inserting back into the main image would work well. The scaling should be proportional rather than absolute pixels, as the specification allows for gain maps to be lower resolution than the primary image (they get scaled by the viewer to match the primary image, which ought to be a very good compromise in the interest of size reduction outside of some edge cases with high frequency detail in the HDR range). If the tool works that way internally, I would say it would be important that the tool automates that step rather than require the external management of that process to (1) simplify implementation, (2) ensure widespread adoption, and (3) avoid mistakes during 3rd-party implementation.

Auxiliary images seem to be showing up in other places, the iPhone appears to use it for at least some HEIC images: https://gist.github.com/kiding/fa4876ab4ddc797e3f18c71b3c2eeb3a. There are also examples of alpha channels like depth masks. To be clear, I don't personally see an obvious use case for HEIC / depth masks those with ImageMagick (as those are probably going to be used for inputs to editing tools like Photoshop), but just want to note there seems to be an increasing list of potential uses for associated images.

snibgo commented 1 year ago

Thanks for the samples.

IM has an option for extracting depth maps, "-define heic:depth-image=true", but it can't write depth maps into files, and doesn't treat then as "associated" images, to process in parallel with main images.

I suspect HDR will increase in popularity. The "gain map" system seems to be a useful bridge between SDR and HDR.

Parallel processing of associated images could be very messy. However, something like this might be feasible (bash syntax):

magick \
  -define hdr:read-gain-map=true \
  in.jpx \
  -crop 2000x1000+100+200 +repage \
  -resize 600x600 \
  -define hdr:write-gain-map=true \
  out.jpx  

This would read images from in.jpx, including GMs. Each GM would be automatically resized to the same size as its parent image. -crop will crop all images (including GMs) to the same dimensions, then -resize will resize all images to fit in a 600x600 box. Finally, each image with its GM is written to out.jpx.

When IM reads the GM, it records any GM metadata as image properties. IM would also record the scale of the GM (eg 25%). When writing the GM, IM resizes the DM by the same scale, and writes the associated metadata.

Under this scheme, -crop and -resize etc would not need any modification. We would need changes to the coder for jpx, and coders for any filetype that might contain GMs.

How does that sound?

fmw42 commented 1 year ago

If the gain map is pixel data (same size as input), then could it not be put into a meta channel?

gregbenz commented 1 year ago

If the gain map is pixel data (same size as input), then could it not be put into a meta channel?

What's a beta channel?

One thing to note with the spec is that the child metadata varies by format, at least in the Adobe samples. In some cases (JPG if I recall correctly), critical gain map metadata is in the auxiliary image, so that probably needs to remain (not sure if the spec allows flexibility between that and the parent image metadata). There is always at least the gain map version in the parent (for quick parsing), but that may be it for some images.

fmw42 commented 1 year ago

meta channel (not beta channel) is an extra channel apart from RGBA for example. TIFF supports extra channels that it uses for example as masks. But they can be used for anything. A depth channel could be stored in Imagemagick as a meta channel, for example. I am not sure how it is stored now in HEIC images.

snibgo commented 1 year ago

fmw42 wrote:

If the gain map is pixel data (same size as input), then could it not be put into a meta channel?

Ah, yes, thanks, I think that's a better idea. Using meta channels would mean the GM would have to be stored in memory at full scale, ie the same size as the main image.

The "Gain Map Specification (1.0 draft 12, May 2023)" https://helpx.adobe.com/content/dam/help/en/camera-raw/using/gain-map/jcr_content/root/content/flex/items/position/position-par/table/row-3u03dx0-column-4a63daf/download_section/download-1/Gain_Map_1_0d12.pdf says the GM will have either 1 or 3 channels. Hence there would be 1 or 3 meta channels.

IMagick is an interface between PHP and ImageMagick. (It isn't developed or supported by the same people as ImageMagick.) When ImageMagick has the capability to read and write GMs, IMagick could inherit that capability.

I don't have access to an HDR display. If I do the development work, I would rely on other people for testing.

gregbenz commented 1 year ago

@snibgo I would be happy to help test any gain maps you convert. You can reach me directly via gregbenzphotography.com/contact/

The Adobe AVIF images would be a also good option to test on an SDR system. They are built with an HDR base image, so they will show as tone mapped if gain maps are not supported and show the proper SDR image if the gain map is used. So if the converted AVIF matches the source AVIF on your screen, the gain map should be ok (the JPGs are built with an SDR base image and therefore would show the same result on your SDR display whether the gain map was valid or not).

The Adobe materials include an image browser to view gain maps, which is a helpful reference.

gregbenz commented 1 year ago

meta channel (not beta channel) is an extra channel apart from RGBA for example. TIFF supports extra channels that it uses for example as masks. But they can be used for anything. A depth channel could be stored in Imagemagick as a meta channel, for example. I am not sure how it is stored now in HEIC images.

Ah, makes sense. I'm used to referring to it as "alpha" channel (or "layer mask"), per Photoshop lingo.

snibgo commented 1 year ago

Update on this:

The supplied sample JPEG files with gainmaps are stored in JPEG MPF (Multi-Picture Format) files. (Similarly JPEGs with depth maps.) Exiftool can extract the multiple images, but can't write them.

So a subproblem is to read and write JPEG MPF files. These have multiple instances of SOI/EOI markers.

It would be useful if IM had a feature that could read all the images in the file. Similarly, when IM has a list of images, it should be able to write all these into a single JPEG MPF file.

A complication is that within the first instance of SOI/EOI, there is (or should be) an APP2 segment labelled "MPF", with pointers to the images.

If we are reading a JPEG MPF and splitting it into separate images, we shouldn't save that APP2 MPF segment. Similarly, when we are writing multiple images to a JPEG MPF, we need to create a APP2 MPF segment and write it to the appropriate place in the file.

As far as I can see, jpeglib etc has no facility for correctly handling MPF. It simply reads the first image (in the first SOI/EOI markers), including the APP2 MPF segment, and ignores the other images. I am happy to be corrected on this.

The best solution would be for jpeglib etc to handle MPF. Failing that, IM could have a fairly simple coder to read and write JPG MPF files.

Thoughts, anyone?

madmanchan commented 1 year ago

Relevant CIPA document on Multi-Picture Format: https://www.cipa.jp/std/documents/e/DC-X007_E.pdf Note that while gain maps are an exciting direction for HDR imaging and formats, this is a swiftly moving area with many changes happening. There is an effort to standardize the approach, but that will take time. MPF is likely to be the mechanism used for JPEG (mostly for lack of a better option), but details like metadata and the math may change.

gregbenz commented 1 year ago

@snibgo Does jpeglib or the info from madmanchan cover what would be needed for IM to add support for JPG gain maps? (I can't offer much on your question on jpeglib, outside my expertise).

Anything I can do to help test?

gregbenz commented 1 year ago

JPG gain maps (aka "Ultra HDR JPG") are now supported in 4 browsers:

They look amazing on all these browsers under a variety of SDR and HDR viewing conditions (and still look great as SDR on other older browsers that lack support since this format is automatically backwards compatible, the gain map data for HDR is just ignored when not understood or not applicable).

Sample gain maps: https://www.adobe.com/go/gainmap_sample_photos

fmw42 commented 1 year ago

Gain maps are being researched but have not been implemented in IM. The first step of supporting MPO JPGs has been implemented. That is as far as I know has been done. It is not an easy implementation for Gain Maps.

gregbenz commented 1 year ago

@fmw42 Thank you for the update! I'm sure there's a lot of detail to process two images in the same file.

fmw42 commented 1 year ago

That part is implemented in IM to read the MPO JPG structure, so I understand. Though it may change as more work is done on Gain Maps

gregbenz commented 1 year ago

@fmw42 Please forgive my ignorance here if this question sounds silly... If IM can now read and write MPO/MPF/auxiliary images, wouldn't the rest of the support for gain maps be fairly straight forward? It should be a matter of retaining the existing metadata and resizing/cropping the images.

Does the resizing cause edge artifacts with the interaction of the two images? I can imagine the encoding of the map itself might risk high frequency details, haven't dug into it.

Or is there some other transformation being done? Conversions of color profile aren't needed (and should probably be avoided and just managed by the content creator).

fmw42 commented 1 year ago

Sorry, I am not a developer, so do not know the code, only what I have heard from the developers discussions. So I could be wrong. They would have to comment.

gregbenz commented 1 year ago

FYI that JPG gain maps are now supported in 4 browsers (representing ~75% of internet usage if all were at latest version - Safari support would bring this to 95%):

See https://www.adobe.com/go/gainmap_sample_photos for sample gain map images in Ultra HDR JPG format.

gregbenz commented 1 year ago

Chrome Canary v119 now includes support for AVIF gain maps as well. Enable via chrome://flags/#avif-gainmap-hdr-images

gregbenz commented 1 year ago

Browser support for gain maps is now mainstream. Chrome v116+, MS Edge v116+, Brave v1.58+, and Opera v102+ all support JPG gain maps by default.

fmw42 commented 1 year ago

What about Firefox and Safari?

gregbenz commented 1 year ago

I don't know if Safari 17 does, will test when MacOS releases next week. That would be the biggest missing piece (the others are 75% of browsing and Safari would get to 95% coverage). They added a lot of core gain map support which was highlighted in public sessions at WWDC.

FireFox share is very low and I'm not holding my breath for support soon, as I've seen no indications of progress.

The nice thing is that JPG gain maps fallback gracefully. So there is no penalty for sharing one of these images and you get the benefit if supported by both browser and the monitor.

Android 14 has support and looks awesome on a Pixel 7 Pro.

martinschmidt-leica commented 1 year ago

Hi @gregbenz , can you share a picture of Pixel 7 Pro that contains a gain map?

gregbenz commented 1 year ago

@martinschmidt-leica I'm confused, what's the purpose? Any JPG gain map will work on the Android 14 and the rendering is similar to what would show on a computer.

You can get a sample JPG gain map from the Adobe link above.

martinschmidt-leica commented 1 year ago

@gregbenz It's not about the map, but about the metadata. I would expect Google to implement Ultra HDR format according to the spec https://developer.android.com/guide/topics/media/platform/hdr-image-format. While the map itself is the same format, Android uses a slightly different container

gregbenz commented 1 year ago

@martinschmidt-leica thought you were asking for a picture "of" not created on the phone.

I haven't explored enough, but haven't found a way to create one with the Android camera yet. I'm creating my images on a computer.

I'll see if I can't find an example using GContainer as the auxiliary image container.

https://developer.android.com/guide/topics/media/platform/hdr-image-format

gregbenz commented 1 year ago

I've been told that JPG can be written with headers for both MPF and GContainer such that regardless of which is in the file, a reader could handle it if it only knows one or the other. That said, it isn't required or recommended in the spec as far as I know, so I don't believe you can assume you'll have that option if the goal is 100% compatibility.

martinschmidt-leica commented 1 year ago

FYI https://github.com/google/libultrahdr

gregbenz commented 1 year ago

@martinschmidt-leica After futher digging, my understanding is that an image can write both types of headers, that MPF is more common, and that when GContainer is used both headers are written.

So as far as ImageMagick goes, it only needs to support MPF (and would still automatically support images which may include GContainer).

gregbenz commented 1 year ago

I've just posted a tutorial video and more details on how to create JPG gain maps using Adobe Camera RAW v15.5: https://gregbenzphotography.com/hdr-images/jpg-hdr-gain-maps-in-adobe-camera-raw/

gregbenz commented 12 months ago

Android 14 was just released today and now officially supports HDR gain maps (Ultra HDR JPG) on Chrome, Edge, and Brave (Opera is not supported but works on desktop and uses same code base, so likely to follow soon). https://blog.google/products/android/android-14/

I've posted an image gallery with HDR gain maps at https://gregbenzphotography.com/hdr-gain-map-gallery/.

I also added a test (#7) to determine if an HDR-capable browser supports gain maps: https://gregbenzphotography.com/hdr/#tests

gregbenz commented 11 months ago

Adobe just released extensive HDR support to every version of Lightroom, including the ability to export HDR JPG gain maps from any device (Mac, PC, Android, iPad, and iPhone). https://gregbenzphotography.com/hdr-images/lightroom-adds-hdr-edit-and-export/

maryla-uc commented 11 months ago

Chrome Canary v119 now includes support for AVIF gain maps as well. Enable via chrome://flags/#avif-gainmap-hdr-images

Hi, I would just like to point out that the gain map format for AVIF is still in flux, so current AVIF+gainmap images might not work in the future (actually, they most definitely won't). I'm glad to see there is interest for this, but make sure to treat it as a demo for now.

gregbenz commented 11 months ago

@maryla-uc Thanks for that info. To confirm, you mean gain maps for AVIF are in flux, you aren't saying JPG gain maps are, correct? As far as I know, JPG gain maps are stable and the Adobe-published proposal is what's in the ISO draft.

maryla-uc commented 11 months ago

Indeed, I was talking about AVIF gain maps only. I'm not an expert on JPEG gain maps but I am also under the impression that they're stable.

gregbenz commented 11 months ago

@maryla-uc Thanks. That has been my understanding, that JPG gain maps are set. They are supported in the Adobe standard by ACR, LR, Chrome, Brave, Edge, and Opera now in production release by default.

I have been under the impression that AVIF might change and hence no encoder support and decoding is only under a dev flag.

gregbenz commented 11 months ago

The Google library for Ultra HDR JPG (JPG gain maps) is now at an official v1.0.0 release and the discussions section has been turned on in the repository. https://github.com/google/libultrahdr/releases/tag/1.0.0

urban-warrior commented 11 months ago

We are currently in the process of enhancing ImageMagick's support for HDR, but there's still a substantial amount of work ahead of us. Let's begin by addressing a fundamental feature: the ability to read JPEG images that contain both SDR (Standard Dynamic Range) and gain map images.

A straightforward approach to create an HDR image from the SDR and Gain map could involve the following command:

magick mpo:01.jpg -colorspace RGB -compose multiply -composite -define quantum:format=floating-point -colorspace sRGB 01.tiff

However, we are exploring the possibility of introducing a new composite operation, perhaps named -compose gain-map, to accurately convert SDR to HDR while considering the gain map metadata specified in the XMP profile.

We're seeking guidance from experts in Gain map handling to help define this new composite operation. What should the formulation look like, taking into account metadata parameters such as hdrgm:GainMapMax and hdrgm:Gamma?

Alternatively, if you have access to code-sets, preferably in C or C++, that demonstrate how to perform SDR to HDR conversion with gain maps, we would greatly appreciate it if you could point us in that direction. Once we've completed this initial step, we will also explore the possibility of converting HDR back to SDR while accounting for the gain map. Your expertise and guidance in this endeavor would be invaluable.

You can find a gain map specification plus sample images (including 01.jpg) here.

gregbenz commented 11 months ago

@urban-warrior That's great to hear.

The issues you've just raised, if I understand correctly, would be expanded capability to generate an HDR with gain map from just an SDR input - correct? If so, perhaps a new issue should be raised for that (inverse tone mapping with gain map output), and then link back to this issue as a dependency. Or are you saying that you don't believe direct manipulation of the gain map would produce a quality result and that a temporary HDR result should be created in order to generate a new gain map?


The intention of my original request here is simply to support keeping the image qualitatively the same when it is resized, compressed, or cropped. The goal being to support derivative files, such as thumbnails and small sizes in the WordPress media library. I don't believe that functionality would require any real understanding of the metadata or image content, but that it could serve as a foundation on which to build additional capability which does involve qualitative alteration of the content.

I believe the core functionality required to support the original request above includes:

I don't believe this request requires ImageMagick to interpret HDR metadata, understand HDR, or construct an HDR image from the gain map. But perhaps I'm over-simplifying. I am unclear if the gamma encoding of the gain map falls outside normal bounds of processing SDR JPGs (ie, is there a quality benefit to converting any of the data to linear space for cropping/scaling, if that isn't already done when handling standard JPG now).

Please let me know if you need some image samples. I could create some gain maps with the same image output a few different ways to use as a reference to evaluate the impacts to the gain map when the same image is scaled or cropped.

Note that the Adobe Gain Map Demo App is very helpful for understanding gain maps conceptually. It represents the converted gain map (the RGB sampler does not list the encoded values, but rather the scalar encoded in linear space to get from SDR to HDR with the interpreted map). See the 12:10+ mark of this tutorial video: https://gregbenzphotography.com/hdr-images/jpg-hdr-gain-maps-in-adobe-camera-raw/

arifdikic commented 10 months ago

Hi All, we are happy to help adding ultra hdr support to imagemagick via libultrahdr. urban-warrior please let me know how we can sync up on the effort (libultrahdr originates from my team)).

urban-warrior commented 10 months ago

@arifdikic, At a minimum, if libultrahdr offers applications for encoding and decoding ultra HDR images, we could explore calling these apps from ImageMagick. Alternatively, the ideal scenario would be to compile libultrahdr on both Linux and Windows, enabling us to invoke its API from C. With thorough documentation, this integration would seamlessly incorporate libultrahdr into ImageMagick.

In the best-case scenario, your team could develop an ultrahdr coder specifically for ImageMagick, releasing it under the ImageMagick license. This would allow us to include it in the official ImageMagick distribution. What is the preference from your team?

To kickstart the process, could you share links to the libultra source, build instructions, API documentation, and an example codeset demonstrating the reading and writing of ultrahdr image files to/from disk?

grapeot commented 10 months ago

I happen to have a reference implementation based on Swift here: https://github.com/grapeot/AppleJPEGGainMap. It's still based on Apple's API, which doesn't have the greatest transparency, but hope it's somehow useful.

gregbenz commented 10 months ago

@grapeot I believe the Apple implementation is not compliant with the ISO spec (ISO 21496-1). Last I checked, it was using slightly different metadata and a 1D map (the ISO spec uses a 3D spec which should enable better color adaptation from HDR to SDR). https://helpx.adobe.com/camera-raw/using/gain-map.html#resources

arifdikic commented 10 months ago

Hi @urban-warrior ,

Ultra HDR format is defined here: https://developer.android.com/guide/topics/media/platform/hdr-image-format (it's compatible with Adobe's format provided above, and we hope to standardize it soon as part of ISO and AOM efforts.)

libultrahdr is located here: https://github.com/google/libultrahdr , it only depends on libjpegturbo.

libultrahdr provides multiple encoding and decoding APIs as documented here:

https://github.com/google/libultrahdr/blob/main/lib/jpegr.h

Sample app is here: https://github.com/google/libultrahdr/blob/main/examples/ultrahdr_app.cpp

Now it's important to emphasize that, Ultra HDR is a dual intent format meaning that the image content to be encoded comes with an SDR-intent (read look) and an HDR-intent. The gainmap is normally generated from these two intents.

Please take a look at the API surface for decoding and encoding to kick this off, and we can discuss the command line changes required and the integration work after clarifying any questions you might have. Again, my team is happy to give a hand for the actual integration work as well.

arifdikic commented 9 months ago

Quick update , we are looking into the code and working on a change proposal.

gregbenz commented 8 months ago

Sounds like JPG gain map support now is making its way to Instagram: https://www.theverge.com/2024/1/18/24043138/instagram-is-working-to-expand-hdr-photo-support-to-more-devices

neckaros commented 7 months ago

Does this request cover gain map support from HEIF/HEIC photos (mainly coming from iPhone)?Especially conversion from HEIF to Ultra JPG or Avif with gain map.

gregbenz commented 7 months ago

I recommend the emphasis be on formats which are on an ISO-standards track. Right now that is JPG gain maps.

AVIF should follow the same path, but appears potentially subject to minor revisions before it may be more final / official. It's already supported in Chrome-based browsers (under a dev flag), so I would expect the risk for at least preliminary support (such as for encoding secondary images) won't be affected - I would expect it's more of a risk to things like metadata. When ready, this will be a very important format as it offers both smaller files and higher quality than JPG.

iOS HEIF have gain maps. I am unaware of a public spec, but the details seem straight-forward and your point on conversion accepting a HEIF source for conversion to another format like JPG is well taken. That would seem to be valuable to enable direct uploads of iPhone images to be shared in a format more widely supported by browsers (ie JPG gain map now - which would add HDR support for Chrome/Edge/Brave/Opera as well as SDR JPG fallback on FireFox vs failing to render anything at all).

neckaros commented 7 months ago

Agreed with you.

Most information I did found is under this gist which you already now about 😁 https://gist.github.com/kiding/fa4876ab4ddc797e3f18c71b3c2eeb3a

and the repo mentioned in the last comment https://github.com/grapeot/AppleJPEGGainMap