enabling jpg generation generates heic files

[tcomputerchip]

I am using the latest docker container to generate jpg images. After checking the files on disk the header seams to indicate they are still heic files. I am using the unraid community docker container that points to this repo. From what I can tell from looking at the logs everything looks to running as expected just the output file is not JPG.

The data is larger in the JPG file but the string at the top shows: typheic

Here is the HEIC file up to !hdlr: ftypheic mif1heicmiaf ©meta !hdlr Here is the JPG file up to I hope the same spot: (ftypheic mif1MiHEMiPrmiafMiHBheic meta !hdlr

Since the files contain sensitive metadata (location, etc) I wont' be able to upload them for inspection.

Opening the images on my iPad I was able to convert to JPEG from the iPad by saving to another location. I compared the files (and header information). The JPEG file looks correct and I am able to open the file on my windows machine.

Maybe I am missing something here? Anything I can do to further investigate?

Thanks for any help you can provide. I would really like to archive my images a JPG in my backups.

[timrettop]

It looks from the code that ImageMagick convert utility is used here: https://github.com/boredazfcuk/docker-icloudpd/blob/5ea5060f65169b1b4df55e57031c4e93caacc50a/sync-icloud.sh#L1432

using the file utility, the heic source file I tested has the following header info: IMG_5746.HEIC: ISO Media, HEIF Image HEVC Main or Main Still Picture Profile

after using the convert tool convert IMG_5746.HEIC -quality 90% IMG_5746.jpg, the file header is described as: IMG_5746.jpg: JPEG image data, JFIF standard 1.01, resolution (DPI), density 72x72, segment length 16, Exif Standard: [TIFF image data, big-endian, direntries=11, manufacturer=Apple, model=iPhone 14 Pro, orientation=upper-left, xresolution=166, yresolution=174, resolutionunit=2, software=17.2.1, datetime=2024:01:01 10:15:57, hostcomputer=iPhone 14 Pro, GPS-Data], baseline, precision 8, 3024x4032, components 3

Does that not match up with what you're seeing?

[timrettop]

Oh interesting, that test I ran with convert was on my local machine, however I do see the same erroneous file descriptor on the converted jpeg's from icloudpd. file 2024/01/01/IMG_5746.JPG 2024/01/01/IMG_5746.JPG: ISO Media, HEIF Image HEVC Main or Main Still Picture Profile

I went back to an older image that icloudpd downloaded and converted some time in the past and do see the proper descriptor for the converted files, maybe there is a bug in the version of ImageMagick.

# docker exec -it icloudpd /bin/sh / # convert --version Version: ImageMagick 7.1.1-22 Q16-HDRI x86_64 21779 https://imagemagick.org Copyright: (C) 1999 ImageMagick Studio LLC License: https://imagemagick.org/script/license.php Features: Cipher DPC HDRI Modules OpenMP(4.5) Delegates (built-in): bzlib cairo fftw fontconfig freetype gslib heic jng jpeg jxl lcms ltdl lzma pangocairo png ps raw rsvg tiff webp x xml zlib Compiler: gcc (13.2)

I was using 7.1.1-23 locally in my first test producing the correct filetype, but don't see anything in their changelog to indicate a fix related to this.

I used convert in the container and reproduced the bad file version, and also got some set_mempolicy not permitted errors during the convert. I'm testing now after enabling privileged mode on the container and will report back

[tcomputerchip]

I set the quality level to 100%. I’ll try dialing it back to 90% to see if I see anything different. But so far I have not been able to produce a valid JPEG image. Thanks for taking a look.

[tcomputerchip]

Sorry about the close, when I commented on the phone it also closed the ticket.

[timrettop]

I tested with Priveledged mode and although it does seem to run faster, it doesn't change the error.

I downloaded alpine-3.19 and installed imagemagick and dependencies and tried the convert and it also produces the bad version.
I compiled imagemagick 7.1.1-22 (same as in alpine 3.19) and it also produces the bad version.

Alpine 3.18.5 package versions (imagemagick 7.1.1-13 and libheif 1.16.2-r0) do not have this issue and result in correct JPEG files.

@boredazfcuk would you accept a PR to roll back the dockerfile alpine source to 3.18.5?

[timrettop]

I submitted this issue to see if there's anything others recommend besides building or awaiting repo updates.

[timrettop]

Seems the issue is due to the container not having the jpeg package. You can confirm by removing one of the bad files, restarting the container and while it is starting the icloud file checks, access the container's shell and run apk --update add jpeg. When the conversion is run, it will convert successfully.

I've submitted a PR to add the package, I think that might take care of future conversions, but I need to go through and delete all the conversions done since I updated.

[timrettop]

I ran this command to search the 2023 and 2024 directories for JPG files that contain the HEIF file descriptor and subsequently trashed them. find 2023 -type f -iname "*.JPG" -exec bash -c 'file="$1"; if $(file $file | grep HEIF > /dev/null); then echo "$file";fi' _ {} \; nothing in that command changes anything, to be safe.

[tcomputerchip]

Thanks @timrettop for your quick investigation. I will attempt to add the package to the container for now and see if that fixes it. I have done this for other containers before so very familiar with adding packages.

[tcomputerchip]

Just another update to confirm your fix. After adding the jpg dependency everything converted properly. I’ll need to purge everything. Not a big deal since I just recently setup these backups. Thanks again.

[boredazfcuk]

Just another update to confirm your fix. After adding the jpg dependency everything converted properly. I’ll need to purge everything. Not a big deal since I just recently setup these backups. Thanks again.

If you run sync-icloud.sh --ConvertAllHEICs the script will search for all .HEIC files in your download path and convert them to JPG. This should overwrite all the JPG files that are in your library already.

Edit: Oh yeah, I forgot there's a sync-icloud.sh --RemoveAllJPGs parameter too. It will search for all the HEIC files in your downloads directory and then see if there is a matching JPG. If it finds one, it will remove it. I used this to remove all the JPGs once my Windows laptop was capable of generating HEIC thumbnails. This should purge all the JPGs converted by the app and leave behind any others... such as JPGs saved from other apps like Safari, Signal, Telegram, WhatsApp etc. After that, run again with the --ConvertAllHEICs parameter to re-convert your library.

Doing it this way should be a fair bit faster than purging your entire library and re-downloading.