search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
51.09k stars 2.7k forks source link

Microservices error - Possible Duplicates causing Duplicate Key Value? #9900

Closed undaunt closed 4 months ago

undaunt commented 5 months ago

The bug

I previously uploaded my iCloud library via bulk API and have been keeping it updated within Immich via the mobile app. I expand on this use case here (https://github.com/immich-app/immich/discussions/9899) but I'm seeing a lot of errors right now while my external library copy of my content is being scanned into Immich.

I'm wondering if this is because some of the exact same images already exist in the upload library, but since it's a separate library Immich doesn't like the potential hash collisions, but its not auto excluded the same as if I had two of the same photo and the mobile app was scanning for upload? Just throwing out possible ideas, I'm not sure.

The OS that Immich Server is running on

Ubuntu 24.04 LTS

Version of Immich Server

v1.105.1

Version of Immich Mobile App

N/A

Platform with the issue

Your docker-compose.yml content

services:
  immich-server:
    container_name: immich-server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    restart: unless-stopped
    networks:
      - ${PROXY_NETWORK}
      - ${APP_NETWORK}
    depends_on:
      - immich-redis
      - immich-db
    command: [ "start.sh", "immich" ]
    volumes:
      - ${PHOTOSDIR}/immich:/usr/src/app/upload
      - ${PHOTOSDIR}/icloud:/usr/src/app/external
      - /etc/localtime:/etc/localtime:ro
    environment:
      DB_HOSTNAME: ${POSTGRES_IMMICH_HOST}
      DB_DATABASE_NAME: ${POSTGRES_IMMICH_DB}
      DB_USERNAME: ${POSTGRES_IMMICH_USER}
      DB_PASSWORD: ${POSTGRES_IMMICH_PASS}
      REDIS_HOSTNAME: ${IMMICH_REDIS_HOSTNAME}

  immich-microservices:
    container_name: immich-microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    restart: unless-stopped
    networks:
      - ${APP_NETWORK}
    depends_on:
      - immich-redis
      - immich-db
    cpus: 3
    devices:
      - /dev/dri:/dev/dri
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${PHOTOSDIR}/immich:/usr/src/app/upload
      - ${PHOTOSDIR}/icloud:/usr/src/app/external
      - /etc/localtime:/etc/localtime:ro
    environment:
      DB_HOSTNAME: ${POSTGRES_IMMICH_HOST}
      DB_DATABASE_NAME: ${POSTGRES_IMMICH_DB}
      DB_USERNAME: ${POSTGRES_IMMICH_USER}
      DB_PASSWORD: ${POSTGRES_IMMICH_PASS}
      REDIS_HOSTNAME: ${IMMICH_REDIS_HOSTNAME}

  immich-machine-learning:
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    container_name: immich-machine-learning
    volumes:
      - ${APPDATA}/photos/immich/models:/cache
    restart: unless-stopped
    networks:
      - ${APP_NETWORK}

  immich-redis:
    image: redis:6.2-alpine@sha256:b6124ab2e45cc332e16398022a411d7e37181f21ff7874835e0180f56a09e82a
    container_name: immich-redis
    restart: unless-stopped
    networks:
      - ${APP_NETWORK}
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping | grep PONG"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 3s
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

  immich-db:
    image: tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    container_name: immich-db
    restart: unless-stopped
    networks:
      - ${APP_NETWORK}
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_IMMICH_PASS}
      POSTGRES_USER: ${POSTGRES_IMMICH_USER}
      POSTGRES_DB: ${POSTGRES_IMMICH_DB}
    volumes:
      - ${APPDATA}/photos/immich/db:/var/lib/postgresql/data

Your .env content

See above

Reproduction steps

1. Import iCloud content via API or mobile app to upload library
2. Add external library with copy of same content
3. Scan external library

Relevant log output

05/30/2024 02:31:31 PM 2024-05-30 21:31:31.940 UTC [357] ERROR:  duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef"
05/30/2024 02:31:31 PM 2024-05-30 21:31:31.940 UTC [357] DETAIL:  Key ("livePhotoVideoId")=(14cee637-f8e9-4406-ba16-40674d08849f) already exists.
05/30/2024 02:31:31 PM 2024-05-30 21:31:31.940 UTC [357] STATEMENT:  UPDATE "assets" SET "id" = $1, "livePhotoVideoId" = $2, "updatedAt" = CURRENT_TIMESTAMP WHERE "id" IN ($3)
05/30/2024 02:31:31 PM 2024-05-30 21:31:31.957 UTC [446] ERROR:  duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef"
05/30/2024 2:31:31 PM 2024-05-30 21:31:31.957 UTC [446] DETAIL:  Key ("livePhotoVideoId")=(18f20df4-9819-499c-9d26-e0fde703fc5d) already exists.
05/30/2024 02:31:31 PM 2024-05-30 21:31:31.957 UTC [446] STATEMENT:  UPDATE "assets" SET "id" = $1, "livePhotoVideoId" = $2, "updatedAt" = CURRENT_TIMESTAMP WHERE "id" IN ($3)
LOG [ImmichMicroservices] [MediaService] Successfully generated JPEG image preview for asset 879e2dbd-a63e-4c3f-97d4-b5fd017b2ad2
2024-05-30T21:23:46.601505859Z [Nest] 7  - 05/30/2024, 9:23:46 PM     LOG [ImmichMicroservices] [MediaService] Successfully generated JPEG image preview for asset e073dc5b-bd13-4e05-aa88-8bb3a1185294
2024-05-30T21:23:46.820207260Z [Nest] 7  - 05/30/2024, 9:23:46 PM   ERROR [ImmichMicroservices] [JobService] Unable to run job handler (metadataExtraction/link-live-photos): QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef"
2024-05-30T21:23:46.820328158Z [Nest] 7  - 05/30/2024, 9:23:46 PM   ERROR [ImmichMicroservices] [JobService] QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef"
2024-05-30T21:23:46.820342819Z     at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
2024-05-30T21:23:46.820347411Z     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-05-30T21:23:46.820352164Z     at async UpdateQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/UpdateQueryBuilder.js:83:33)
2024-05-30T21:23:46.820356296Z     at async AssetRepository.update (/usr/src/app/dist/repositories/asset.repository.js:183:9)
2024-05-30T21:23:46.820360399Z     at async MetadataService.handleLivePhotoLinking (/usr/src/app/dist/services/metadata.service.js:155:9)
2024-05-30T21:23:46.820364637Z     at async /usr/src/app/dist/services/job.service.js:145:36
2024-05-30T21:23:46.820368572Z     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
2024-05-30T21:23:46.820372526Z     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
2024-05-30T21:23:46.820376565Z [Nest] 7  - 05/30/2024, 9:23:46 PM   ERROR [ImmichMicroservices] [JobService] Object:
2024-05-30T21:23:46.820382675Z {
2024-05-30T21:23:46.820386810Z   "id": "686ee129-e364-4616-b14d-0e09fad7b987",
2024-05-30T21:23:46.820391084Z   "source": "upload"
2024-05-30T21:23:46.820395487Z }

Additional information

No response

davidvpe commented 4 months ago

I am also experiencing this issue

[Nest] 17  - 06/12/2024, 7:52:41 AM   ERROR [Api:AssetMediaService~ajxhyi4p] Error uploading file QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef"
QueryFailedError: duplicate key value violates unique constraint "UQ_16294b83fa8c0149719a1f631ef"
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async SubjectExecutor.executeInsertOperations (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:260:42)
    at async SubjectExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:92:9)
    at async EntityPersistExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/EntityPersistExecutor.js:140:21)
    at async AssetMediaService.create (/usr/src/app/dist/services/asset-media.service.js:292:23)
    at async AssetMediaService.uploadAsset (/usr/src/app/dist/services/asset-media.service.js:131:27)
    at async AssetMediaController.uploadAsset (/usr/src/app/dist/controllers/asset-media.controller.js:39:29)

A bit of a back story. Sometimes the app logs out by itself forcing it to resync all assets again and again, but since the last update this error started appearing, It's not very concerning as I know that the assets are already on the server, but if they are there then this's not exactly an error, just something that needs to be skipped next time.

alextran1502 commented 4 months ago

Fixed in #10637