search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
33.05k stars 1.59k forks source link

unhandled Error: 500 when importing an Apple Photos library with the CLI #9115

Open anderbubble opened 2 weeks ago

anderbubble commented 2 weeks ago

The bug

I am importing an Apple Photos library with the CLI.

[username]@immich1:/srv/immich/[library_name].photoslibrary/originals$ docker run -it -v "$(pwd)":/import:ro -e IMMICH_INSTANCE_URL=http://[ip_address]:2283/api -e IMMICH_API_KEY=[redacted] ghcr.io/immich-app/immich-cli:latest upload --album [album_name] --recursive /import

This proceeds successfully to import the images, but then fails when "Adding assets to albums."

Crawling for assets...
Checking files | ████████████████████████████████████████ | 100% | ETA: 0s | 21920/21920 assets
Found 0 new files and 21920 duplicates
All assets were already uploaded, nothing to do.
Creating albums | ████████████████████████████████████████ | 100% | ETA: 0s | 0/0 albums
Successfully created 0 new albums
Successfully updated 21920 assets
Adding assets to albums | ██████████████████████████████████████░░ | 93% | ETA: 1s | 20578/21920 assets
file:///usr/src/app/dist/index.js:3064
  throw new y(e.status, e.data, e.headers);
        ^

y [Error]: Error: 500
    at Object.l [as ok] (file:///usr/src/app/dist/index.js:3064:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async updateAlbums (file:///usr/src/app/dist/index.js:18020:9)
    at async upload (file:///usr/src/app/dist/index.js:17812:3) {
  status: 500,
  data: {
    message: 'Failed to add assets to album',
    error: 'Internal Server Error',
    statusCode: 500
  },
  headers: Headers {
    [Symbol(headers list)]: HeadersList {
      cookies: null,
      [Symbol(headers map)]: Map(8) {
        'x-powered-by' => { name: 'X-Powered-By', value: 'Express' },
        'x-immich-cid' => { name: 'x-immich-cid', value: '4oso6f2n' },
        'content-type' => {
          name: 'Content-Type',
          value: 'application/json; charset=utf-8'
        },
        'content-length' => { name: 'Content-Length', value: '92' },
        'etag' => { name: 'ETag', value: '"5c-lyaVQAHBg1LfX0s5U5+HqLAWa+A"' },
        'date' => { name: 'Date', value: 'Fri, 26 Apr 2024 03:50:15 GMT' },
        'connection' => { name: 'Connection', value: 'keep-alive' },
        'keep-alive' => { name: 'Keep-Alive', value: 'timeout=5' }
      },
      [Symbol(headers map sorted)]: null
    },
    [Symbol(guard)]: 'immutable',
    [Symbol(realm)]: null
  }
}

Node.js v20.11.1

On the server, this is produced at the time of the error:

[Nest] 6  - 04/26/2024, 3:50:15 AM   ERROR [QueryFailedError: duplicate key value violates unique constraint "PK_c67bc36fa845fb7b18e0e398180"
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async AlbumRepository.addAssetIds (/usr/src/app/dist/repositories/album.repository.js:205:9)
    at async addAssets (/usr/src/app/dist/utils/asset.util.js:29:9)
    at async AlbumService.addAssets (/usr/src/app/dist/services/album.service.js:150:25)] Failed to add assets to album
[Nest] 6  - 04/26/2024, 3:50:15 AM   ERROR [QueryFailedError: duplicate key value violates unique constraint "PK_c67bc36fa845fb7b18e0e398180"
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async AlbumRepository.addAssetIds (/usr/src/app/dist/repositories/album.repository.js:205:9)
    at async addAssets (/usr/src/app/dist/utils/asset.util.js:29:9)
    at async AlbumService.addAssets (/usr/src/app/dist/services/album.service.js:150:25)] QueryFailedError: duplicate key value violates unique constraint "PK_c67bc36fa845fb7b18e0e398180"

The same error occurs if I restart the server and re-run the same import. (The logs above are from such a re-run, which is why all the assets are duplicates.)

The OS that Immich Server is running on

Debian GNU/Linux 12 (bookworm)

Version of Immich Server

v1.102.3

Version of Immich Mobile App

1.102.2 build.151

Platform with the issue

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672
    restart: always

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    restart: always

volumes:
  model-cache:

### Your .env content

```Shell
# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/srv/immich/library

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=[redacted]

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
DB_DATA_LOCATION=./postgres

REDIS_HOSTNAME=immich_redis

### Reproduction steps

```bash
1. `docker run -it -v "$(pwd)":/import:ro -e IMMICH_INSTANCE_URL=http://[ip_address]:2283/api -e IMMICH_API_KEY=[redacted] ghcr.io/immich-app/immich-cli:latest upload --album [album_name] --recursive /import`

Relevant log output

[username]@immich1:/srv/immich/[library_name].photoslibrary/originals$ docker run -it -v "$(pwd)":/import:ro -e IMMICH_INSTANCE_URL=http://[ip_address]:2283/api -e IMMICH_API_KEY=[redacted] ghcr.io/immich-app/immich-cli:latest upload --album [album_name] --recursive /import
Crawling for assets...
Checking files | ████████████████████████████████████████ | 100% | ETA: 0s | 21920/21920 assets
Found 0 new files and 21920 duplicates
All assets were already uploaded, nothing to do.
Creating albums | ████████████████████████████████████████ | 100% | ETA: 0s | 0/0 albums
Successfully created 0 new albums
Successfully updated 21920 assets
Adding assets to albums | ██████████████████████████████████████░░ | 93% | ETA: 1s | 20578/21920 assets
file:///usr/src/app/dist/index.js:3064
  throw new y(e.status, e.data, e.headers);
        ^

y [Error]: Error: 500
    at Object.l [as ok] (file:///usr/src/app/dist/index.js:3064:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async updateAlbums (file:///usr/src/app/dist/index.js:18020:9)
    at async upload (file:///usr/src/app/dist/index.js:17812:3) {
  status: 500,
  data: {
    message: 'Failed to add assets to album',
    error: 'Internal Server Error',
    statusCode: 500
  },
  headers: Headers {
    [Symbol(headers list)]: HeadersList {
      cookies: null,
      [Symbol(headers map)]: Map(8) {
        'x-powered-by' => { name: 'X-Powered-By', value: 'Express' },
        'x-immich-cid' => { name: 'x-immich-cid', value: '4oso6f2n' },
        'content-type' => {
          name: 'Content-Type',
          value: 'application/json; charset=utf-8'
        },
        'content-length' => { name: 'Content-Length', value: '92' },
        'etag' => { name: 'ETag', value: '"5c-lyaVQAHBg1LfX0s5U5+HqLAWa+A"' },
        'date' => { name: 'Date', value: 'Fri, 26 Apr 2024 03:50:15 GMT' },
        'connection' => { name: 'Connection', value: 'keep-alive' },
        'keep-alive' => { name: 'Keep-Alive', value: 'timeout=5' }
      },
      [Symbol(headers map sorted)]: null
    },
    [Symbol(guard)]: 'immutable',
    [Symbol(realm)]: null
  }
}

Node.js v20.11.1
[Nest] 6  - 04/26/2024, 3:50:15 AM   ERROR [QueryFailedError: duplicate key value violates unique constraint "PK_c67bc36fa845fb7b18e0e398180"
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async AlbumRepository.addAssetIds (/usr/src/app/dist/repositories/album.repository.js:205:9)
    at async addAssets (/usr/src/app/dist/utils/asset.util.js:29:9)
    at async AlbumService.addAssets (/usr/src/app/dist/services/album.service.js:150:25)] Failed to add assets to album
[Nest] 6  - 04/26/2024, 3:50:15 AM   ERROR [QueryFailedError: duplicate key value violates unique constraint "PK_c67bc36fa845fb7b18e0e398180"
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async AlbumRepository.addAssetIds (/usr/src/app/dist/repositories/album.repository.js:205:9)
    at async addAssets (/usr/src/app/dist/utils/asset.util.js:29:9)
    at async AlbumService.addAssets (/usr/src/app/dist/services/album.service.js:150:25)] QueryFailedError: duplicate key value violates unique constraint "PK_c67bc36fa845fb7b18e0e398180"


### Additional information

_No response_
anderbubble commented 2 weeks ago

Looks like this might be the same as #8608.

FlashStopFall commented 2 weeks ago

I've had the exact same problem. (Immich v1.103.1, Immich CLI v2.2.0, Node.Js v21.7.3).

FlashStopFall commented 2 weeks ago

For me, the upload fails with or without the "--recursive" flag. It did successfully upload 171 JPG images via specifying a file name range "file[0-171].JPG", but failed to upload a different batch of about 100 photos. It also uploaded two 1.3GB .AVI files together and a single 1.8GB .MOV file successfully, but failed to upload 1.1GB and 1.8GB .MOV files together.

In my case, I didn't run the command through docker, I just used immich upload -A "iCloud Photo Library" ./*.AVI.

My system never seems to add any assets to an album before failing. If it is going to fail, it always fails at Adding assets to albums | ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 0% | ETA: 0s | 0/80 assets

anderbubble commented 2 weeks ago

I used fdupes -rn --delete . to find and remove duplicate files in the library, which let me finish an upload successfully, so I'm no longer stuck. I also realized that --album wasn't doing what I intended, but it was --album-name that I wanted.

But I still think that either immich or the CLI needs to handle this case better.

Olimiya commented 1 day ago

Windows solution reference:

# This is to output first to see which duplicates are present
$folderPath = "C:\path\to\your\folder"; $hashes = Get-ChildItem -Path $folderPath -Recurse -File | Get-FileHash; $duplicates = $hashes | Group-Object -Property Hash | Where-Object { $_.Count -gt 1 }; $duplicates | ForEach-Object { Write-Output "Duplicate files found:"; $_.Group | ForEach-Object { Write-Output $_.Path }; Write-Output "---" }

# THEN DELETE
$folderPath = "C:\path\to\your\folder"; $hashes = Get-ChildItem -Path $folderPath -Recurse -File | Get-FileHash; $duplicates = $hashes | Group-Object -Property Hash | Where-Object { $_.Count -gt 1 }; $duplicates | ForEach-Object { $_.Group | Select-Object -Skip 1 | Remove-Item -Force }