Crash adding assets to albums using CLI upload

mpoplaw commented 3 months ago

The bug

I tried uploading 41k images inside 364 folders using the immich cli, specifying that I want it to create albums for each folder. Looks like it was successful uploading the files, but then crashed adding assets to the new albums it created. This is a brand new immich installation, running 1.101.0.

The OS that Immich Server is running on

TrueNAS Scale

Version of Immich Server

1.101.0

Version of Immich Mobile App

N/A

Platform with the issue

[X] Server
[ ] Web
[ ] Mobile

Your docker-compose.yml content

NA

Your .env content

NA

Reproduction steps

1. Using IMMICH CLI,
> immich upload -c 8 -a -r /mnt/exlib

Stack Trace:

Crawling for assets...
Checking files | ████████████████████████████████████████ | 100% | ETA: 0s | 41267/41267 assets
Found 41267 new files and 0 duplicates
Uploading assets | ████████████████████████████████████████ | 100% | ETA: 0s | 162.1 GB/162.1 GB
Successfully uploaded 41267 assets (162.1 GB)
Creating albums | ████████████████████████████████████████ | 100% | ETA: 0s | 364/364 albums
Successfully created 364 new albums
Successfully updated 41267 assets
Adding assets to albums | ██░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 5% | ETA: 34s | 2424/41267 assets
file:///usr/local/lib/node_modules/@immich/cli/dist/index.js:3064
  throw new y(e.status, e.data, e.headers);
        ^

y [Error]: Error: 500
    at Object.l [as ok] (file:///usr/local/lib/node_modules/@immich/cli/dist/index.js:3064:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async updateAlbums (file:///usr/local/lib/node_modules/@immich/cli/dist/index.js:18020:9)
    at async upload (file:///usr/local/lib/node_modules/@immich/cli/dist/index.js:17812:3) {
  status: 500,
  data: {
    message: 'Failed to add assets to album',
    error: 'Internal Server Error',
    statusCode: 500
  },
  headers: Headers {
    [Symbol(headers list)]: HeadersList {
      cookies: null,
      [Symbol(headers map)]: Map(7) {
        'x-powered-by' => { name: 'X-Powered-By', value: 'Express' },
        'content-type' => {
          name: 'Content-Type',
          value: 'application/json; charset=utf-8'
        },
        'content-length' => { name: 'Content-Length', value: '92' },
        'etag' => { name: 'ETag', value: '"5c-lyaVQAHBg1LfX0s5U5+HqLAWa+A"' },
        'date' => { name: 'Date', value: 'Sun, 07 Apr 2024 21:22:36 GMT' },
        'connection' => { name: 'Connection', value: 'keep-alive' },
        'keep-alive' => { name: 'Keep-Alive', value: 'timeout=5' }
      },
      [Symbol(headers map sorted)]: null
    },
    [Symbol(guard)]: 'immutable',
    [Symbol(realm)]: null
  }
}

Node.js v20.11.1

Additional information

No response

mpoplaw commented 3 months ago

Before I tried to add this larger library of pictures and videos, I tested it with a smaller set of files and folders and it did work as expected, but that was on v1.100.0.

bo0tzz commented 3 months ago

The error message indicates a serverside error. Can you post the corresponding server logs, and the version of the CLI that you're using?

libra146 commented 3 months ago

The error message indicates a serverside error. Can you post the corresponding server logs, and the version of the CLI that you're using?

I have the same problem, my cli version

root@debian:~# immich -V
2.2.0

libra146 commented 3 months ago

The error message indicates a serverside error. Can you post the corresponding server logs, and the version of the CLI that you're using?

Oh, I almost forgot the server log

[Nest] 6  - 04/08/2024, 2:35:09 AM     LOG [EventRepository] Websocket Connect:    5uQVKsXjT8wKUtoZAAAD
[Nest] 6  - 04/08/2024, 3:52:48 AM   ERROR [QueryFailedError: duplicate key value violates unique constraint "PK_c67bc36fa845fb7b18e0e398180"
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async AlbumRepository.addAssetIds (/usr/src/app/dist/repositories/album.repository.js:205:9)
    at async addAssets (/usr/src/app/dist/utils/asset.util.js:29:9)
    at async AlbumService.addAssets (/usr/src/app/dist/services/album.service.js:148:25)] Failed to add assets to album
[Nest] 6  - 04/08/2024, 3:52:48 AM   ERROR [QueryFailedError: duplicate key value violates unique constraint "PK_c67bc36fa845fb7b18e0e398180"
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async AlbumRepository.addAssetIds (/usr/src/app/dist/repositories/album.repository.js:205:9)
    at async addAssets (/usr/src/app/dist/utils/asset.util.js:29:9)
    at async AlbumService.addAssets (/usr/src/app/dist/services/album.service.js:148:25)] QueryFailedError: duplicate key value violates unique constraint "PK_c67bc36fa845fb7b18e0e398180"
[Nest] 6  - 04/08/2024, 4:41:52 AM   ERROR [QueryFailedError: duplicate key value violates unique constraint "PK_c67bc36fa845fb7b18e0e398180"
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async AlbumRepository.addAssetIds (/usr/src/app/dist/repositories/album.repository.js:205:9)
    at async addAssets (/usr/src/app/dist/utils/asset.util.js:29:9)
    at async AlbumService.addAssets (/usr/src/app/dist/services/album.service.js:148:25)] Failed to add assets to album
[Nest] 6  - 04/08/2024, 4:41:52 AM   ERROR [QueryFailedError: duplicate key value violates unique constraint "PK_c67bc36fa845fb7b18e0e398180"
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async AlbumRepository.addAssetIds (/usr/src/app/dist/repositories/album.repository.js:205:9)
    at async addAssets (/usr/src/app/dist/utils/asset.util.js:29:9)
    at async AlbumService.addAssets (/usr/src/app/dist/services/album.service.js:148:25)] QueryFailedError: duplicate key value violates unique constraint "PK_c67bc36fa845fb7b18e0e398180"

libra146 commented 3 months ago

I searched the history of the problem and found that it was similar to this problem. https://github.com/immich-app/immich/issues/8028 The reason was that some files were uploaded repeatedly.

There's a PR closing it, but it doesn't seem to work https://github.com/immich-app/immich/pull/8199

mpoplaw commented 3 months ago

immich -V 2.2.0

Where do I grab the server log?

I am going to try calling immich upload for each asset directory and pass the album name with -A in hope that it might work and maybe I'll see which dir it's having an issue with.

libra146 commented 3 months ago

immich -V 2.2.0

Where do I grab the server log?

I am going to try calling immich upload for each asset directory and pass the album name with -A in hope that it might work and maybe I'll see which dir it's having an issue with.

You can run docker logs -n 100 immich_server to view the server logs

mpoplaw commented 3 months ago

What I found out is that I had duplicate images and videos in my directories and after removing the dups, I was able to workaround this problem. So maybe there is an issue detecting duplicates and rejecting them using the CLI?

melhakim commented 3 months ago

Can confirm. Have the same issue. Tried both adding an external library and from the cli. Getting PK constraint validation error. Also possible I have duplicate images.

Cli output:

file:///usr/local/lib/node_modules/@immich/cli/dist/index.js:3064
  throw new y(e.status, e.data, e.headers);
        ^

y [Error]: Error: 500
    at Object.l [as ok] (file:///usr/local/lib/node_modules/@immich/cli/dist/index.js:3064:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async updateAlbums (file:///usr/local/lib/node_modules/@immich/cli/dist/index.js:18020:9)
    at async upload (file:///usr/local/lib/node_modules/@immich/cli/dist/index.js:17812:3) {
  status: 500,
  data: {
    message: 'Failed to add assets to album',
    error: 'Internal Server Error',
    statusCode: 500
  },
  headers: Headers {
    [Symbol(headers list)]: HeadersList {
      cookies: null,
      [Symbol(headers map)]: Map(7) {
        'x-powered-by' => { name: 'X-Powered-By', value: 'Express' },
        'content-type' => {
          name: 'Content-Type',
          value: 'application/json; charset=utf-8'
        },
        'content-length' => { name: 'Content-Length', value: '92' },
        'etag' => { name: 'ETag', value: '"5c-lyaVQAHBg1LfX0s5U5+HqLAWa+A"' },
        'date' => { name: 'Date', value: 'Sat, 13 Apr 2024 07:54:13 GMT' },
        'connection' => { name: 'Connection', value: 'keep-alive' },
        'keep-alive' => { name: 'Keep-Alive', value: 'timeout=5' }
      },
      [Symbol(headers map sorted)]: null
    },
    [Symbol(guard)]: 'immutable',
    [Symbol(realm)]: null
  }
}

Node.js v20.11.1

Microservice container output:

    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async AlbumRepository.addAssetIds (/usr/src/app/dist/repositories/album.repository.js:205:9)
    at async addAssets (/usr/src/app/dist/utils/asset.util.js:29:9)
    at async AlbumService.addAssets (/usr/src/app/dist/services/album.service.js:148:25)] Failed to add assets to album
[Nest] 7  - 04/13/2024, 7:54:13 AM   ERROR [QueryFailedError: duplicate key value violates unique constraint "PK_c67bc36fa845fb7b18e0e398180"
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async AlbumRepository.addAssetIds (/usr/src/app/dist/repositories/album.repository.js:205:9)
    at async addAssets (/usr/src/app/dist/utils/asset.util.js:29:9)
    at async AlbumService.addAssets (/usr/src/app/dist/services/album.service.js:148:25)] QueryFailedError: duplicate key value violates unique constraint "PK_c67bc36fa845fb7b18e0e398180"

jeanmatthieud commented 3 months ago

I also had this issue. I used a software to detect duplicated images, deleted them and it worked!

liyboy commented 3 months ago

I also had this issue. I deleted the duplicate file, then it worked!

immich-app / immich