search

ente-io / ente

Fully open source, End to End Encrypted alternative to Google Photos and Apple Photos
https://ente.io
GNU Affero General Public License v3.0
14.38k stars 743 forks source link

Importing Google Takeout create lots of duplicated image (live vs still) #2477

Open lacasseio opened 1 month ago

lacasseio commented 1 month ago

Description

I'm currently in the process of migrating about 200 Gb of pictures from Google to Ente, but I noticed there are a lot of duplicates (or import issues) when it comes to live images. I see one live and one still image for each live picture. The deduplicate feature doesn't mark any pictures as duplicates (most likely because of the format difference). It seems as-if the still image is the "thumbnail" for the live picture that Google exported as two different entries. Ente should be able to reconcile these pictures.

Version

v1.7.2

What product are you using?

Ente Photos

What platform are you using?

Desktop - macOS

ua741 commented 1 month ago

Hey @lacasseio,

Can you please help us with the following information?

  1. Were these live images taken from an iPhone or an Android device?

  2. I see one live and one still image for each live picture. Just to ensure that we are on the same page, does "live" here mean that both the video and image are part of the same file?

  3. If the answer to the second question is No, can you tell us the file names you see on Google Photos for one such live photo? We combine a still image and video as a live photo in our system based on the file name.

  4. If the answer to the second question is Yes, can you check the file name on Ente for both the live and still images, and search for the same inside your Google Photos? There might be a duplicate copy in your export for the image, which could have resulted in this behavior.

PS: As you have guessed, the current de-duplication system only detects files that have the same hash. If your Google export contains three individual files for a live photo, only two of them will be combined as a live image, and the third file will be uploaded as an individual file.

lacasseio commented 1 month ago

I don't see the same result on my phone (Android) vs desktop (macOS). I will give feedback according to my Android phone and add another comment when I get in front of my desktop.

All pictures from the import were taken using an Android phone. On Android, the duplicate entries both have the live icon (at the bottom right corner). I took a screenshot shoot of what it looks like in my library: Screenshot_20240718-054142.png

Next, here is the different metadata of two duplicate: Screenshot_20240718-055603.png

VS

Screenshot_20240718-055710.png

The only difference is they come from two different album (Camera vs Photo from 2024). My feeling is Camera comes from Ente sync and Photo from 2024 comes from Google Photo.

ua741 commented 1 month ago

Thank you for the clarification. This issue is primarily occurring due to how Google Photos handles Android Live/Motion Photos.

In Android Live photos, there's usually just a single file that contains both the image and video (you can read more details here).

During export, Google Photos breaks[1] the Android live photo and exports it as two separate files. If you try to copy the files from "Photos from 2024" to your Android device, you will see two different files instead of a single live photo. As a workaround to solve this issue, our desktop app uses heuristics (primarily based on file name and time) to link the files into a single live photo on Ente. If you try to download such imported files on Web/desktop, they will be downloaded as two separate files.

When uploading the file directly from the device, we don't break the Android Live photo. The photo is uploaded as a single file, which you can easily download on any other Android device as a Live photo.

PS: Our web/desktop app doesn't support Android Motion Photo. That's why you might not be seeing the motion/live photo indicator on the web. https://github.com/ente-io/ente/discussions/561

[1] https://www.reddit.com/r/googlephotos/comments/o0qnll/google_takeout_motionlive_pictures_2_files/ [2] https://www.reddit.com/r/googlephotos/comments/n5vj4e/separate_motion_photos_in_takeout_data/