search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
43.21k stars 2.1k forks source link

The jpg images saved to Photos on iOS are not the original images from the server. #8815

Open ngtrio opened 4 months ago

ngtrio commented 4 months ago

The bug

I have noticed that the JPG images downloaded from the server in the iOS app slightly vary in size (they look the same visually), which results in duplicate images on the server The PNG images are not affected.

The OS that Immich Server is running on

unraid 6.12.6

Version of Immich Server

1.101.0

Version of Immich Mobile App

1.101.0

IOS 17.4.1

Platform with the issue

Your docker-compose.yml content

I don't feel like this is particularity relevant but I can provide if needed

Your .env content

I don't feel like this is particularity relevant but I can provide if needed

Reproduction steps

1.upload a JPG image to server via web page
2.click the download cloud-shaped button in the iOS app
3.you will see an extra image in the gallery that looks identical visually, and the cloud-shaped download button has not been checked as expected.
4.comparing the downloaded image with the original uploaded image will reveal that they are of different sizes.

IMG_37FFB81694BB-1

Relevant log output

No response

Additional information

No response

mmomjian commented 4 months ago

Can you be more specific with the filesize? How are you determining this size and what changes are you seeing?

I believe the backup symbol is supposed to indicate images that were backed up from that device, which the downloaded one was not, since it was uploaded on Web.

waclaw66 commented 4 months ago

This happens (https://github.com/immich-app/immich/issues/6209) when you upload file with the same filename but different content than you have downloaded to device. Seems the file is moddified during transfer to device. Please compare it binary.

ngtrio commented 4 months ago

@mmomjian I exported two images, and renamed them to "upload" and "download" for convenience. The uploaded one is 100842 bytes, and the downloaded one is 101518 bytes. Moreover, this change occurs only between the steps of uploading from the web and downloading from the app.

image

I think the change in file size causes the hash of the two images to differ, so duplicate images exist on the server.

BTW, I am unable to reproduce this issue with PNG images. When I download the image from the app, the backup button is correctly checked and there is only one image, even if the image was uploaded from the web.

So, is anyone else able to reproduce this issue with JPG images, or is it a configuration issue on my end?

ngtrio commented 4 months ago

This happens (#6209) when you upload file with the same filename but different content than you have downloaded to device. Seems the file is moddified during transfer to device. Please compare it binary.

It doesn't appear to be the same issue, as I haven't performed any editing operations on the images.

waclaw66 commented 4 months ago

I think the change in file size causes the hash of the two images to differ, so duplicate images exist on the server.

Binary comparison of those two would help to track the cause of file size change. It could be many reasons, reverse proxy missconfiguration, unintentional modification some app on your phone, etc.

ngtrio commented 4 months ago

I have tried the following steps again with the help of a few friends:

  1. Upload the 693-byte JPG image below to the immich server. image.
  2. Save this image from the immich iOS app.
  3. Save this image from the ios "Share".

Now I have the following three images, corresponding to the three steps above by name:
image.
As you can see, the sizes of the last two images are the same(16983 bytes), both larger than the original image(693 bytes).

Then I used exiftool to compare the three images, the latter two are consistent and both contain more information than the original image, especially this one: Thumbnail Image : (Binary data 16004 bytes, use -b option to extract):

Therefore, I speculate that the way immich's iOS app saves images may be similar to the way images are saved in the iOS Share, which adds some additional data to the original image, rather than using the system API to save original images (if there is one). Could iOS developers please help confirm this?

florihupf commented 4 months ago

Same here. I reported this way back when and this issue is one of the main things i run into as it prevents to download images that are only on the server as it creates duplicates that are re uploaded.

mokahless commented 3 months ago

Same issue here. Discovered it while playing around with what I thought was another issue.

  1. Used the cloud button to download a 3MB photo.
  2. Didn't realize that that added it to my roll in ios photos app, so used the share button > download (unwittingly this time on the duplicate photo because I'd exited and gone back in in between).
  3. A 20MB photo appears stripped of metadata in my ios photos app roll at the top with today's date.
  4. I search the web for that problem.
  5. I come across this thread so start checking. I discover the duplicate, 11MB version I had tried to share, not stripped of metadata.

So a note, whatever is causing the alternate-filesize duplicate, is also affecting the share button.

Seems to me this should be pretty high priority given it breaks the ability to download images in iOS, in a silent manner of inflated duplicates.

So just to be sure, I tried it again:

  1. Clicked the cloud to download.
  2. Says download started, says download success.
  3. no checkmark appears
  4. a new photo appears with a cloud crossed out because it is on my phone but hasn't been synced yet. It's a larger photo. This time not nearly as big a difference as the other photo but 2.2MB > 2.3MB. Metadata is still there so the date is right.

Like others have tried, I then tried with a PNG and there was no issue.

I tried it with an internet meme file that was a jpg and the creation date was changed to today. I guess expected since it is a new different sized file.

I'm on the latest release as of this comment.

florihupf commented 3 months ago

Thanks for the report and I agree, I think this impacts usability a bit. As soon as most of your pics and not on device, you can't share / download those pics anymore because it implicitly creates duplicates.

pawel-szopinski commented 3 months ago

I have the same issue with jpeg files. You mentioned that PNG files are not affected. I can also add that HEIC (the format in which iPhone camera photos are stored) are not affected either. Unfortunately, for me, jpeg files are the most often redownloaded ones (wallpapers, memes), so it is a bit of a hassle to deal with the duplicates.

ngxson commented 3 months ago

Another suggestion would be to allow immich to save downloaded photos into a dedicated album (on iOS), then we can exclude that album from uploading. It still requires changes in the code, but I imagine that it should be easier than fixing de-duplicated problem (please correct me if I'm wrong).

Edit: Maybe also have a look on photo_manager.editor.saveImage() function

wr1williams commented 3 months ago

Just chiming in to report I've experienced this as well, although it doesn't seem to impact every single jpg file for me. I have an external library with 863 images in it, all of which I tried to download to my phone, and the automatic sync re-uploaded roughly a third of them. I can also echo that the sizes of the files downloaded to the phone through the app are ever so slightly larger, as seen in the snippet below


PS C:\Users\wwilliams\Downloads\immich-20240530_114856> ls

    Directory: C:\Users\wwilliams\Downloads\immich-20240530_114856

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----         5/30/2024  11:49 AM        1298546 000001510024.jpg
-a----         5/30/2024  11:49 AM        1299941 IMG_5877.JPG