[BUG] thumbnails missing for non-existent files under non-existent paths

HoldOnBro commented 9 months ago

The bug

generating thumbnail failed and I checked logs found:

    at async /usr/src/app/dist/domain/job/job.service.js:113:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Object:
{
  "id": "419f1f20-de59-4d79-878a-6339dc206357"
}
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file is missing: /Media/家庭/照片/2023/10.3 台州+绍兴（台州原图待恢复）/绍兴/DSCF4859.RAF
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Error: Input file is missing: /Media/家庭/照片/2023/10.3 台州+绍兴（台州原图待恢复）/绍兴/DSCF4859.RAF
    at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
    at MediaRepository.resize (/usr/src/app/dist/infra/repositories/media.repository.js:38:14)
    at MediaService.generateThumbnail (/usr/src/app/dist/domain/media/media.service.js:116:44)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MediaService.handleGenerateJpegThumbnail (/usr/src/app/dist/domain/media/media.service.js:103:28)
    at async /usr/src/app/dist/domain/job/job.service.js:113:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Object:
{
  "id": "f13f03f2-e15e-47c8-a172-75cfed145430"
}
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file is missing: /Media/家庭/照片/2023/10.3 台州+绍兴（台州原图待恢复）/绍兴/DSCF4860.RAF
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Error: Input file is missing: /Media/家庭/照片/2023/10.3 台州+绍兴（台州原图待恢复）/绍兴/DSCF4860.RAF
    at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
    at MediaRepository.resize (/usr/src/app/dist/infra/repositories/media.repository.js:38:14)
    at MediaService.generateThumbnail (/usr/src/app/dist/domain/media/media.service.js:116:44)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MediaService.handleGenerateJpegThumbnail (/usr/src/app/dist/domain/media/media.service.js:103:28)
    at async /usr/src/app/dist/domain/job/job.service.js:113:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Object:
{
  "id": "19c94660-68bd-4a24-9f71-19535da5ef72"
}
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file is missing: /Media/家庭/照片/2023/10.3 台州+绍兴（台州原图待恢复）/绍兴/DSCF4858.RAF
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Error: Input file is missing: /Media/家庭/照片/2023/10.3 台州+绍兴（台州原图待恢复）/绍兴/DSCF4858.RAF
    at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
    at MediaRepository.resize (/usr/src/app/dist/infra/repositories/media.repository.js:38:14)
    at MediaService.generateThumbnail (/usr/src/app/dist/domain/media/media.service.js:116:44)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MediaService.handleGenerateJpegThumbnail (/usr/src/app/dist/domain/media/media.service.js:103:28)
    at async /usr/src/app/dist/domain/job/job.service.js:113:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Object:
{
  "id": "57137594-2fe3-4cec-9f3f-43deb530bd8c"
}
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file is missing: /Media/家庭/照片/2023/10.3 台州+绍兴（台州原图待恢复）/绍兴/DSCF4856.RAF
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Error: Input file is missing: /Media/家庭/照片/2023/10.3 台州+绍兴（台州原图待恢复）/绍兴/DSCF4856.RAF
    at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
    at MediaRepository.resize (/usr/src/app/dist/infra/repositories/media.repository.js:38:14)
    at MediaService.generateThumbnail (/usr/src/app/dist/domain/media/media.service.js:116:44)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MediaService.handleGenerateJpegThumbnail (/usr/src/app/dist/domain/media/media.service.js:103:28)
    at async /usr/src/app/dist/domain/job/job.service.js:113:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Object:
{
  "id": "a775c9c4-1daa-463c-87a3-c5a9be9ea839"
}
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file is missing: /Media/家庭/照片/2023/10.3 台州+绍兴（台州原图待恢复）/绍兴/DSCF4857.RAF
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Error: Input file is missing: /Media/家庭/照片/2023/10.3 台州+绍兴（台州原图待恢复）/绍兴/DSCF4857.RAF
    at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
    at MediaRepository.resize (/usr/src/app/dist/infra/repositories/media.repository.js:38:14)
    at MediaService.generateThumbnail (/usr/src/app/dist/domain/media/media.service.js:116:44)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MediaService.handleGenerateJpegThumbnail (/usr/src/app/dist/domain/media/media.service.js:103:28)
    at async /usr/src/app/dist/domain/job/job.service.js:113:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Object:
{
  "id": "06f38a64-fcae-4421-95a7-e4a0f4bd1a97"
}
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input image exceeds pixel limit
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Error: Input image exceeds pixel limit
    at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
    at MediaRepository.resize (/usr/src/app/dist/infra/repositories/media.repository.js:38:14)
    at MediaService.generateThumbnail (/usr/src/app/dist/domain/media/media.service.js:116:44)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MediaService.handleGenerateJpegThumbnail (/usr/src/app/dist/domain/media/media.service.js:103:28)
    at async /usr/src/app/dist/domain/job/job.service.js:113:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)
[Nest] 7  - 12/20/2023, 1:28:06 AM   ERROR [JobService] Object:
{
  "id": "a1c6bc57-a5be-4281-bab2-386cd4897a38"
}
terminate called after throwing an instance of 'LibRaw_exceptions'
terminate called after throwing an instance of 'LibRaw_exceptions'

the path "10.3 台州+绍兴（台州原图待恢复）" is non-existent right now

I changed it to "10.3 台州+绍兴" couple weeks ago and accidentally found this missing thumbnails problem yesterday.

The OS that Immich Server is running on

Openwrt

Version of Immich Server

v1.91.4

Version of Immich Mobile App

v1.91.4

Platform with the issue

[X] Server
[ ] Web
[ ] Mobile

Your docker-compose.yml content

immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "immich"]
    ports:
      - 2283:3001
    volumes:
      - '${UPLOAD_LOCATION}:/usr/src/app/upload'
      - '${HDD_RAID1}/Media:/Media'
      - '/etc/localtime:/etc/localtime:ro'
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always
    labels:
      com.centurylinklabs.watchtower.depends-on: /redis,/database

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
      file: ./hwaccel.yml
      service: hwaccel
    command: ["start.sh", "microservices"]
    volumes:
      - '${UPLOAD_LOCATION}:/usr/src/app/upload'
      - '${HDD_RAID1}/Media:/Media'
      - '/etc/localtime:/etc/localtime:ro'
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always
    labels:
      com.centurylinklabs.watchtower.depends-on: /redis,/database

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - './Immich/model-cache:/cache'
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:80cc8518800438c684a53ed829c621c94afd1087aaeb59b0d4343ed3e7bcf6c5
    restart: always
    labels:
      - "com.centurylinklabs.watchtower.enable=false"

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - './Immich/pgdata:/var/lib/postgresql/data'
    restart: always
    labels:
      - "com.centurylinklabs.watchtower.enable=false"

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=${HDD_RAID1}/Immich

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=mx1Q)/`We+sS<=
DB_PASSWORD=sBm2Xu8AzSwPjQ

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1.look for missing thumbnails
2.found 5 failed
3.check log

Additional information

No response

o10s commented 9 months ago

I have I believe the same issue after a mass import of files with the cli it looks like this (see picture) but when I look at the actual files on disk I don't have that ma y pictures for this day but only 2 or 3 (this happens on a lot of days) I'm at 1.91.3. Screenshot_20231220-070749

dekiesel commented 9 months ago

I am not sure but the issue I am experiencing might be related to this one

immich_microservices     | [Nest] 8  - 12/20/2023, 7:06:09 PM   ERROR [JobService] Error: Input file is missing: upload/thumbs/aab9fdfc-7fd9-416e-b6a2-3639710baa2c/c7/e2/c7e2bd06-620d-4268-92e8-fdc84973dbc1.jpeg
immich_microservices     |     at Sharp.toBuffer (/usr/src/app/node_modules/sharp/lib/output.js:161:17)
immich_microservices     |     at MediaRepository.crop (/usr/src/app/dist/infra/repositories/media.repository.js:28:14)
immich_microservices     |     at PersonService.handleGeneratePersonThumbnail (/usr/src/app/dist/domain/person/person.service.js:326:58)
immich_microservices     |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
immich_microservices     |     at async /usr/src/app/dist/domain/job/job.service.js:113:37immich_microservices     |     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
immich_microservices     |     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)
immich_microservices     | [Nest] 8  - 12/20/2023, 7:06:09 PM   ERROR [JobService] Object:immich_microservices     | {
immich_microservices     |   "id": "e7f62f8d-b995-4f5d-a057-04ef7e8cf34c"
immich_microservices     | }

This is a log excerpt from when I try to create missing thumbnails.

It seems that a missing thumbnail is used as the input for thumbnail creation?

Please let me know if this issue isn't related, I'll open a new issue then.

jrasm91 commented 2 months ago

Some bugs in the past could lead to orphaned files that led to these issues. These have been fixed in subsequent versions. If there is a set of reproducible steps to reproduce this issue today, we can take another look though.

immich-app / immich