Why is ImageMagick 7.0.x suddenly creating corrupted HEIC images?

[JackSzwergold]

I am a fan of HEIC image files! Same quality images at around 10x smaller file size. And I am happy ImageMagick can handle HEIC conversions.

In July 2020 — after some basic quality tests — I went ahead and converted tons of high resolution JPEG scans I made over the years to HEIC format. Using a simple Bash script I wrote that uses ImageMagick — as well as ExifTool to retain metadata — it converted all of the images with no problems.

Flash forward to now, and somehow the same exact Bash script is creating HEIC images that seem to be corrupted; the preview on the file doesn’t change and attempting to open it results in this message:

The file “filename.heic” could not be opened.

It may be damaged or use a file format that Preview doesn’t recognize.

This happens to seemingly arbitrary images (not random images) but consistently. Meaning, if I give the script a pile of images to convert — such as one.jpg, two.jpg, three.jpg, etc… — and two.jpg is converted to two.heic which is corrupt, that image will always convert as a corrupt HEIC.

Why the heck is this happening? Is this an ImageMagick bug? And yes, ImageMagick has definitely been updated on my system since July 2020 thanks to regular Homebrew updates but still, this is not expected behavior. Things I have tried:

• Converting the JPEG images to TIFF or PNG and then converting those derivative images to HEIC. The resulting HEIC is corrupt.
• Upscaling the images to something like 1200 to 1920 pixels in width or height (whichever is larger) and the resulting HEIC is corrupt.
• Stripping out all metadata tags with ExifTool prior to HEIC conversion. The resulting HEIC is still corrupt.
• Commenting out the ExifTool line in my script doesn’t change anything. The resulting HEIC is corrupt.
• Uninstalling and re-installing ImageMagick — and even purging all Homebrew installs — didn’t help.

I am using ImageMagick 7.0.10-29 Q16 x86_64 2020-09-06 (installed via Homebrew) on macOS Catalina 10.5.6 with this simple Bash script:

find -E 'Desktop/Pics' -type f -iregex '.*\.(JPG|JPEG|PNG|TIF|TIFF)$' |\
  while read FULL_IMAGE_PATH
  do
    PATH_SANS_EXTENSION="${FULL_IMAGE_PATH%.*}"
    convert "${FULL_IMAGE_PATH}" "${PATH_SANS_EXTENSION}".heic
    exiftool -overwrite_original_in_place -tagsFromFile "${FULL_IMAGE_PATH}" "${PATH_SANS_EXTENSION}".heic
  done

And here is the full output of convert --version:

Version: ImageMagick 7.0.10-29 Q16 x86_64 2020-09-06 https://imagemagick.org
Copyright: © 1999-2020 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC HDRI Modules OpenMP(3.1) 
Delegates (built-in): bzlib freetype gslib heic jng jp2 jpeg lcms lqr ltdl lzma openexr png ps tiff webp xml zlib

And here are some test images I found (via the Wikimedia commons) that recreates the issue; 1280px-Test_(student_assessment).jpeg always results in a corrupted and inoperable HEIC image:

• 1599px-Test_(student_assessment).jpeg
• 1156px-Blue_Mountains_NationalPark(AU),_ThreeSisters--2019--_1987-9.jpg

[snibgo]

As you have IM v7, I suggest you use "magick", not "convert".

You say:

And here is the full output of convert --version: Version: ImageMagick 7.0.10-29 Q16 x86_64 2020-09-06 https://imagemagick.org Copyright: © 1999-2020 ImageMagick Studio LLC License: https://imagemagick.org/script/license.php Features: Cipher DPC HDRI Modules OpenMP(3.1)

If that really is the full output, that explains the problem. There should be a final line like:

Delegates (built-in): ... heic ...

[JackSzwergold]

@snibgo Thanks for the info. And sorry about the ImageMagick version info being incorrect. Edited my initial question to show the full output with the delegates line as follows:

Delegates (built-in): bzlib freetype gslib heic jng jp2 jpeg lcms lqr ltdl lzma openexr png ps tiff webp xml zlib

But regardless, I do not believe this is a delegate issue: I can do bulk conversions using convert for all kinds of images without issues. And JPEG to HEIC had no issues a few months back. But now, it seems that seemingly arbitrary HEIC images are being generated that are corrupt. So I can convert some images to HEIC. But a handful just result in inoperable files. To me, that is not a delegate issue.

[snibgo]

And JPEG to HEIC had no issues a few months back.

What has changed since then? A new installation of IM, I suppose.

You should also look at "-list format", eg:

f:\web\im>%IMG7%magick -list format |grep -i heic

     HEIC* HEIC      r--   High Efficiency Image Format (1.4.0)

My usual version of IM can read, but not write, HEIC format.

Confusingly, this command appears to work:

%IMG7%magick rose: t.heic

However, my IM can't write HEIC, so the format is actually taken from the input:

%IMG7%magick  t.heic info:
t.heic PPM 70x46 70x46+0+0 8-bit sRGB 9673B 0.000u 0:00.007

The input format is PPM, so the output is also PPM. Any software that expects t.heic to be HEIC format will report an error.

[fmw42]

There has been some recent work in heic.c, I believe, from some of the code contributions that I have seen. So perhaps something in that module has been changed and introduced this issue.

[JackSzwergold]

@snibgo Preface, but essentially all of the answers to your questions are provided in my original post.

Yes, ImageMagick has been updated since back in July 2020 on my machine. I use Homebrew as a package manager for macOS and I routinely update my installed packages and it seems a recent update to ImageMagick contains this error.

As for checking as to whether HEIC is supported in my install, that is pretty much a moot point. But here are the relevant lines from magick -list format:

AVIF* HEIC      rw+   AV1 Image File Format (1.8.0)
HEIC* HEIC      rw+   High Efficiency Image Format (1.8.0)

I can clearly convert some images to HEIC but not all and I cannot see any clear pattern as two which arbitrary image becomes a corrupted HEIC. Let me clearly restate what I said in the original post:

This happens to seemingly arbitrary images (not random images) but consistently. Meaning, if I give the script a pile of images to convert — such as one.jpg, two.jpg, three.jpg, etc… — and two.jpg is converted to two.heic which is corrupt, that image will always convert as a corrupt HEIC.

I strongly believe this is not an issue on my side but a bug in HEIC implementation in ImageMagick (7.0.10-29 Q16 x86_64 2020-09-06).

@fmw42 I believe you are correct here. I am not too sure which commit might be related to this issue, but the commit history here definitely shows a lot of tweaks from July 2020 to September 2020. For some reason this commit here — with the pull request “Using RGBA colorspace for sRGB source images when encoding HEIC images #2487” — seems to be the most likely place to look given the sheer number of changes and additions it contains.

[dooman87]

I did some tests on the example images and could confirm that the latest IM version doesn't seem to work. I've also tested with heif-enc utility from libheif-1.7.0 and it worked. And when I tested with IM 7.0.10-23 with libheif-1.7.0 it worked as well.

From the above, it looks like an issue introduced in libheif-1.8.0, but to be 100% sure we would need to test with heif-enc from 1.8.0. I don't have it handy now, but should be able to test tonight.

[JackSzwergold]

@dooman87 Thanks for the update and confirmation that something in the code chain of something connected to ImageMagick is choking HEIC output. Appreciate the assistance.

And just an update that the version of heif-enc I have installed via Homebrew is version 1.8.0:

Encoder: x265 = x265 HEVC encoder (0.0)
heif-enc  libheif version: 1.8.0

And attempting to convert the test image 1599px-Test_(student_assessment).jpeg results in the same thing I get with ImageMagick: 1599px-Test_(student_assessment).heic is created but seemingly corrupt and unopenable while 1156px-Blue_Mountains_National_Park_(AU),_Three_Sisters_--_2019_--_1987-9.jpg converts to 1156px-Blue_Mountains_National_Park_(AU),_Three_Sisters_--_2019_--_1987-9.heic without a hitch. So I guess this is not an ImageMagick issue but rather a libheif-1.8.0 issue?

[JackSzwergold]

The issue seems to be based on odd numbered pixel widths/heights.

If anyone here cares — since this issue seems to be deeper in the library hole than ImageMagick specific — the only consistent pattern I have found based on this comment here is that the images that are not able to open in macOS have odd number widths.

Like in the examples I have provided, 1599px-Test_(student_assessment).jpeg fails due to the width being an odd number width (1599 pixels) while 1156px-Blue_Mountains_National_Park_(AU),Three_Sisters--2019--_1987-9.jpg converts and is openable since it has an even number width (1156 pixels).

Randomly sampling some of the hundreds of HEIC images I converted in July 2020 shows only even numbered dimensions. So I wonder if back then the algorithm either added or shaved off one pixel to get even numbers? Or maybe stretched or compressed the image by one pixel?

Regardless, one thing I have noticed is these images I cannot open via “Preview” or even “Pixelmator” can still be converted back to JPEG images cleanly via ImageMagick. So ImageMagick has no problems processing these images with odd number dimensions, but if there is an odd dimension macOS cannot open it up in “Preview” and other apps.

[farindk]

Here my analysis: the h265 compression codec (as used in HEIF) cannot encode images with odd width/height. libheif <=1.7.0 applied the simple hack to round down the image size to an even number (e.g. 1599x1001 becomes 1598x1000). However, this was not optimal because we lose image content. Starting with libheif 1.8.0, the encoded image size is rounded up to the next even number and then a crop transform is added in the HEIF image to get the correct output size.

Now, the problem with macOS seems to be that macOS cannot read images with clap crop transforms. macOS always saves images a tiled grid images. The total grid size can also be an odd number, hence macOS does not use any crop transforms.

You cannot open those images with macOS because it apparently does not support the (mandatory) 'clap' image transformations.

[JackSzwergold]

@farindk Excellent explanation. For my own purposes I will adjust my ImageMagick scripts to do cropping of 1 pixel from odd numbered widths/heights and then do an HEIC conversion. As for the larger macOS issue, might report this to Apple. Thanks again for the help in tracking this down!

[JackSzwergold]

To all ImageMagick specific developers here: libheif 1.9.0 now has a -E/--even-size option that shaves off one pixel (rounds down) from odd numbered dimensions to make the dimensions even so HEIC files created using it can be opened by macOS (and possibly other OSes) without issue.

Is there a way for ImageMagick to use that option? Should a new issue be opened?

[dlemstra]

We could probably introduce a heic specific define for that. We don't need a separate issue for that. I will take a look at how the -E option works inside heif-enc. Any suggestion how we should call that option?

[JackSzwergold]

@dlemstra Looking at the list of ImageMagick options here perhaps -heic-even would work? Unless you want to go the more format agnostic/universal way and perhaps -even-dimensions could work with specific cases handled in different ways if other formats need/require such an accommodation.

[urban-warrior]

Recommended development practices is to use a define for all coder specific options. Using your example, we would support -define heic:even-dimensions=true or -define heic:pad=true.

[JackSzwergold]

@urban-warrior Hey… That makes sense to me! Whatever works and is most in line with established ImageMagick development practices is good. As long as the new option can be cleanly passed from an ImageMagick command I think that will work.

That said, I would therefore recommend it be called as follows:

-define heic:even-size=true

That uses the same wording as the longer --even-size option value that libheif already uses.

[farindk]

We could probably introduce a heic specific define for that. ... I will take a look at how the -E option works inside heif-enc. ...

-E is very simple: if the image has an odd width or height, simply crop away one column/row of pixels. It really is just a quick workaround and no long-term solution.

The best option would probably be that I change the way libheif saves images to a format that Apple readers can understand. I.e. I'll have to wrap each image into a virtual tiled grid image (even if it only consists of one image) and use the grid dimensions to crop away the extra pixels instead of using the more obvious crop transform.

If you want a quick intermediate solution for ImageMagick, one option could be to print a warning when odd-sized images are encoded and print the ImageMagick options that the user can simply copy paste into the command line to crop the image to even size.

[dlemstra]

I think it would be better to crop the image yourself @JackSzwergold instead of us adding an option for this. Also not sure if we should print a warning...

[JackSzwergold]

@dlemstra Fair enough. The convert command I am using now to deal with this rounds up using -extent and a nested called to convert again using -format; works fine for my purposes:

convert source.jpg -background white -extent  $(convert source.jpg -format '%[fx:2*int((w+1)/2)]x%[fx:2*int((h+1)/2)]!' info:) destination.heic

[quantumgolem]

@JackSzwergold not really sure what's going on, but mine only creates corrupt HEIC files if the image width is even (it works if the image width is odd). So your command created all corrupted HEIC files for me, but using 2*int(w/2)+1 for the width solved the issue.

Is there a better way to create the images instead of using this command? It seems kind of like a hacky workaround, since a white line is added to the right side of the image

[Lorne0]

@sn0wyfall I also encountered the same problem as you mentioned too. However after upgrading the package, the problem fixed.

platform: macOS Catalina 10.15.6

my command: magick a.jpg -compress lossless a.heic

the package version I used and it would be broken: libheif == 1.10.0 imagemagick == 7.0.10-60

after upgrading packages by homebrew: libheif == 1.12.0 imagemagick == 7.0.11-11

It works fine now :) Hope this comment can help you~