search

simulot / immich-go

An alternative to the immich-CLI command that doesn't depend on nodejs installation. It tries its best for importing google photos takeout archives.
GNU Affero General Public License v3.0
1.18k stars 35 forks source link

Command duplicate, option `-ignore-extension` doesn't always work #315

Open toteto opened 1 week ago

toteto commented 1 week ago

I have two images being reported as duplicates from Immich but not from immich-go

image

Left (original)

{
  "id": "a36a22be-a149-4bf4-9382-935f84406ae2",
  "deviceAssetId": "1000006383",
  "ownerId": "910a47e4-bedc-4f1c-bbf2-f0d07268a1f7",
  "deviceId": "b45cc9ab602dac461c6284bdb7015c44c9f8270dee1ffbd03cab01c1005fea8a",
  "libraryId": null,
  "type": "IMAGE",
  "originalPath": "upload/library/antonio/2024/2024-04-22/20240422_163537.heic",
  "originalFileName": "20240422_163537.heic",
  "originalMimeType": "image/heic",
  "resized": true,
  "thumbhash": "XxgKDQJwaIaZeHd3qJiJZx9d12BX",
  "fileCreatedAt": "2024-04-22T14:35:37.525Z",
  "fileModifiedAt": "2024-04-22T14:35:38.000Z",
  "localDateTime": "2024-04-22T16:35:37.525Z",
  "updatedAt": "2024-06-19T09:32:34.001Z",
  "isFavorite": false,
  "isArchived": false,
  "isTrashed": false,
  "duration": "0:00:00.00000",
  "exifInfo": {
    "make": "samsung",
    "model": "Galaxy S24+",
    "exifImageWidth": 4000,
    "exifImageHeight": 3000,
    "fileSizeInByte": 4652170,
    "orientation": "6",
    "dateTimeOriginal": "2024-04-22T14:35:37.525Z",
    "modifyDate": "2024-04-22T14:35:37.525Z",
    "timeZone": "UTC+2",
    "lensModel": null,
    "fNumber": 2.2,
    "focalLength": 2.2,
    "iso": 1250,
    "exposureTime": "1/50",
    "latitude": 42.0109921997222,
    "longitude": 21.3718387997222,
    "city": "Krušopek",
    "state": "Grad Skopje",
    "country": "The Republic of North Macedonia",
    "description": "",
    "projectionType": null
  },
  "livePhotoVideoId": null,
  "people": [],
  "checksum": "ltAFmoRS9V/BD/17XcuGnkCRrs0=",
  "stackCount": null,
  "isOffline": false,
  "hasMetadata": true,
  "duplicateId": "021ce122-723b-4461-8b95-f1cb063b1cb0"
}

Right (Google Photos)

{
  "id": "42c9d832-94aa-48e6-b4ff-45e310419855",
  "deviceAssetId": "20240422_163537.jpg-2414264",
  "ownerId": "910a47e4-bedc-4f1c-bbf2-f0d07268a1f7",
  "deviceId": "asus",
  "libraryId": null,
  "type": "IMAGE",
  "originalPath": "upload/library/antonio/2024/2024-04-22/20240422_163537.jpg",
  "originalFileName": "20240422_163537.jpg",
  "originalMimeType": "image/jpeg",
  "resized": true,
  "thumbhash": "XxgKDQJwaIaZeHd3qJiJZx9d12BX",
  "fileCreatedAt": "2024-04-22T14:35:37.525Z",
  "fileModifiedAt": "2024-04-22T16:35:37.000Z",
  "localDateTime": "2024-04-22T16:35:37.525Z",
  "updatedAt": "2024-06-19T09:36:30.906Z",
  "isFavorite": false,
  "isArchived": false,
  "isTrashed": false,
  "duration": "0:00:00.00000",
  "exifInfo": {
    "make": "samsung",
    "model": "Galaxy S24+",
    "exifImageWidth": 4000,
    "exifImageHeight": 3000,
    "fileSizeInByte": 2414264,
    "orientation": "6",
    "dateTimeOriginal": "2024-04-22T14:35:37.525Z",
    "modifyDate": "2024-04-22T14:35:37.525Z",
    "timeZone": "UTC+2",
    "lensModel": null,
    "fNumber": 2.2,
    "focalLength": 2.2,
    "iso": 1250,
    "exposureTime": "1/50",
    "latitude": 42.0109921997222,
    "longitude": 21.3718387997222,
    "city": "Krušopek",
    "state": "Grad Skopje",
    "country": "The Republic of North Macedonia",
    "description": "",
    "projectionType": null
  },
  "livePhotoVideoId": null,
  "people": [],
  "checksum": "0TxA/Xw3FsR/k6zS6LJfM+cqf1g=",
  "stackCount": null,
  "isOffline": false,
  "hasMetadata": true,
  "duplicateId": "021ce122-723b-4461-8b95-f1cb063b1cb0"
}

I am running immich-go with:

./immich-go -server=http://localhost:2283 -key=<my key> duplicate -yes -ignore-tz-errors -ignore-extension

There are quite few more instances of this where duplicate is not detected. Some are videos.

Let me know if you need more samples/data.

toteto commented 4 days ago

Here is another example just for testing purposes

{
  "id": "a09d31f4-4739-4388-bb59-eb2e709f04a6",
  "deviceAssetId": "20240428_162300.heic-1098177",
  "ownerId": "910a47e4-bedc-4f1c-bbf2-f0d07268a1f7",
  "deviceId": "casaos",
  "libraryId": null,
  "type": "IMAGE",
  "originalPath": "upload/library/antonio/2024/2024-04-28/20240428_162300.heic",
  "originalFileName": "20240428_162300.heic",
  "originalMimeType": "image/heic",
  "resized": true,
  "thumbhash": "ZLYFMwpwnneMeXPleHCABzc=",
  "fileCreatedAt": "2024-04-28T14:23:00.937Z",
  "fileModifiedAt": "2024-04-28T14:23:00.000Z",
  "localDateTime": "2024-04-28T16:23:00.937Z",
  "updatedAt": "2024-06-20T14:44:12.741Z",
  "isFavorite": false,
  "isArchived": false,
  "isTrashed": false,
  "duration": "0:00:00.00000",
  "exifInfo": {
    "make": "samsung",
    "model": "SM-N985F",
    "exifImageWidth": 1868,
    "exifImageHeight": 4000,
    "fileSizeInByte": 1098177,
    "orientation": "1",
    "dateTimeOriginal": "2024-04-28T14:23:00.937Z",
    "modifyDate": "2024-04-28T14:23:00.937Z",
    "timeZone": "UTC+2",
    "lensModel": null,
    "fNumber": 1.8,
    "focalLength": 7,
    "iso": 50,
    "exposureTime": "1/229",
    "latitude": null,
    "longitude": null,
    "city": null,
    "state": null,
    "country": null,
    "description": "",
    "projectionType": null
  },
  "livePhotoVideoId": "b2f9e882-ced5-4d85-be2e-7e7408811629",
  "people": [],
  "checksum": "U0STAk9xu7EyQ/7Bl7j97jtMGAA=",
  "stackCount": null,
  "isOffline": false,
  "hasMetadata": true,
  "duplicateId": "0544b3d3-d141-4058-8e92-90e997011ed9"
}
{
  "id": "2b2efb7e-92e4-467b-a44b-c7cea8b8ab7c",
  "deviceAssetId": "20240428_162300.jpg-4494797",
  "ownerId": "910a47e4-bedc-4f1c-bbf2-f0d07268a1f7",
  "deviceId": "asus",
  "libraryId": null,
  "type": "IMAGE",
  "originalPath": "upload/library/antonio/2024/2024-04-28/20240428_162300.jpg",
  "originalFileName": "20240428_162300.jpg",
  "originalMimeType": "image/jpeg",
  "resized": true,
  "thumbhash": "ZLYFMwpwnneMeXPleHCABzc=",
  "fileCreatedAt": "2024-04-28T14:23:00.937Z",
  "fileModifiedAt": "2024-04-28T16:23:00.000Z",
  "localDateTime": "2024-04-28T16:23:00.937Z",
  "updatedAt": "2024-06-20T14:44:12.741Z",
  "isFavorite": false,
  "isArchived": false,
  "isTrashed": false,
  "duration": "0:00:00.00000",
  "exifInfo": {
    "make": "samsung",
    "model": "SM-N985F",
    "exifImageWidth": 4000,
    "exifImageHeight": 1868,
    "fileSizeInByte": 4494797,
    "orientation": "6",
    "dateTimeOriginal": "2024-04-28T14:23:00.937Z",
    "modifyDate": "2024-04-28T14:23:00.937Z",
    "timeZone": "UTC+2",
    "lensModel": null,
    "fNumber": 1.8,
    "focalLength": 7,
    "iso": 50,
    "exposureTime": "1/229",
    "latitude": null,
    "longitude": null,
    "city": null,
    "state": null,
    "country": null,
    "description": "",
    "projectionType": null
  },
  "livePhotoVideoId": "4843608a-fbe5-415e-bb5e-76b42f94bba5",
  "people": [],
  "checksum": "oE3/8nkj43R3QK5hvnos4RL30M4=",
  "stackCount": null,
  "isOffline": false,
  "hasMetadata": true,
  "duplicateId": "0544b3d3-d141-4058-8e92-90e997011ed9"
}
Screenshot 2024-06-28 at 15 45 33

One issue I see is that the JPEG image is larger in size then the HEIC image

simulot commented 3 days ago

The duplicate command is meant to remove duplicates caused by the compressed version of the photo stored by Google Photo. Same name, same date, but different file size. Immich-go will remove the most compressed files: the smallest file.

In your case the heic version would be discarded. Maybe you would prefer keeping the HEIC one, knowing that HEIC is better thant JPG.

One issue I see is that the JPEG image is larger in size then the HEIC image

HEIC format is reputed to produce smaller files than JPG for the same quality. That's your 1st case that is unusual.