search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
47.2k stars 2.39k forks source link

old people thumbnails broken after swiching to webp #9944

Closed heikomat closed 1 month ago

heikomat commented 4 months ago

The bug

When 1.102.0 released, which introduced webp support for thumbnails and previews, i switched to webp. I had issues with thumbnails and read https://github.com/immich-app/immich/issues/8981. I deleted all jpeg files and reran about every job immich provides.

Most things work fine, but the thumbnails for people are not showing. Logs show, that immich tries to load .jpeg files like this.

server-1                             | [Nest] 7  - 06/02/2024, 12:48:30 PM   ERROR [ImmichServer] [LoggerRepository] Unable to send file: Error
server-1                             | Error: ENOENT: no such file or directory, access 'upload/thumbs/cb765e07-ebd7-4903-a0c7-41598e43d1cf/e5/24/e524e25a-fda9-4636-a5af-3e40d3f6ed40.jpeg'
server-1                             |     at async access (node:internal/fs/promises:603:10)
server-1                             |     at async sendFile (/usr/src/app/dist/utils/file.js:51:9)
server-1                             |     at async PersonController.getPersonThumbnail (/usr/src/app/dist/controllers/person.controller.js:52:9)

If i look into that folder, a file with than name exists, but it has the (correct) .webp ending.

Interestingly enough, thumbnails of new people from photos from two weeks ago and yesterday do show up correctly. When i look at the person-table, there is no entry with .webp-path (althouth there are thousands with .jpeg-path)

SELECT * FROM person WHERE "thumbnailPath" ilike '%.webp';

 id | createdAt | updatedAt | ownerId | name | thumbnailPath | isHidden | birthDate | faceAssetId
----+-----------+-----------+---------+------+---------------+----------+-----------+-------------
(0 rows)

In the code, i found a handlePersonMigration, which calls movePersonFile which in turn uses getPersonThumbnailPath as target-path.

getPersonThumbnailPath has the .jpeg-ending hardcoded. Could that be an issue?

How do i tell immich to update the paths of the thumbnails for the persons to webp?

The OS that Immich Server is running on

Ubuntu 22.04

Version of Immich Server

v1.105.1

Version of Immich Mobile App

v1.105.1

Platform with the issue

Your docker-compose.yml content

version: "3.8"

services:
  server:
    image: ghcr.io/immich-app/immich-server:release
    command: ["start.sh", "immich"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      # upload libraries
      - /quick-vault/main-storage/Bilder/smartphone-camera-roll:/usr/src/app/upload/library/heiko
      - /quick-vault/main-storage/Backups/<redacted>/smartphone/camera-roll:/usr/src/app/upload/library/<redacted>
      # external libraries
      - /quick-vault/main-storage/Bilder/Oer-Erkenschwicker\ bis\ 2018:/mnt/media/heiko/old-pics-til-2018:ro
      - /quick-vault/main-storage/Bilder/seit_2017:/mnt/media/heiko/albums-since-2017:ro
      - /fat-vault/Media/Backups/<redacted>/backup_hdd_<redacted>:/mnt/media/<redacted>/old-hdd:ro

    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always
    networks:
      - default
      - traefik
    labels:
      - traefik.enable=true
      - traefik.http.routers.immich.rule=Host(`<redacted>`)

  microservices:
    image: ghcr.io/immich-app/immich-server:release
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      # upload libraries
      - /quick-vault/main-storage/Bilder/smartphone-camera-roll:/usr/src/app/upload/library/heiko
      - /quick-vault/main-storage/Backups/<redacted>/smartphone/camera-roll:/usr/src/app/upload/library/<redacted>
      # external libraries
      - /quick-vault/main-storage/Bilder/<redacted>\ bis\ 2018:/mnt/media/heiko/old-pics-til-2018:ro
      - /quick-vault/main-storage/Bilder/seit_2017:/mnt/media/heiko/albums-since-2017:ro
      - /fat-vault/Media/Backups/<redacted>/backup_hdd_<redacted>:/mnt/media/<redacted>/old-hdd:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  machine-learning:
    image: ghcr.io/immich-app/immich-machine-learning:release
    volumes:
      - ./immich-data/model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    image: redis:6.2-alpine
    volumes:
      - ./immich-data/redis:/data
    restart: always

  database:
    image: tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    restart: always

networks:
  traefik:
    external: true

Your .env content

DB_HOSTNAME=database
DB_USERNAME=postgres
DB_PASSWORD=postgres
DB_DATABASE_NAME=immich
DB_DATA_LOCATION=./immich-data/pgdata

REDIS_HOSTNAME=redis

UPLOAD_LOCATION=/docker/immich/immich-data

TYPESENSE_API_KEY=some-random-text

PUBLIC_LOGIN_PAGE_MESSAGE=

IMMICH_WEB_URL=http://web:3000
IMMICH_SERVER_URL=http://server:3001
IMMICH_MACHINE_LEARNING_URL=http://machine-learning:3003

IMMICH_API_URL_EXTERNAL=https://<redacted>, http://192.168.178.20:2283, http://proxy:8080

Reproduction steps

1. be on a version below 1.102.0
2. have people detected in the explore-page
3. upgrade to 1.102.0
4. switch thumbnails and previews from jpeg to webp
5. run necessary jobs to migrate thumbnails
--- optional
6. kill all jpegs as described in https://github.com/immich-app/immich/issues/8981
7. rerun more jobs

Relevant log output

server-1                             | [Nest] 7  - 06/02/2024, 12:48:30 PM   ERROR [ImmichServer] [ExceptionsHandler] ENOENT: no such file or directory, access 'upload/thumbs/cb765e07-ebd7-4903-a0c7-41598e43d1cf/5a/d2/5ad28af5-450b-42ec-83ec-5bec363d8b58.jpeg'
server-1                             | Error: ENOENT: no such file or directory, access 'upload/thumbs/cb765e07-ebd7-4903-a0c7-41598e43d1cf/5a/d2/5ad28af5-450b-42ec-83ec-5bec363d8b58.jpeg'
server-1                             |     at async access (node:internal/fs/promises:603:10)
server-1                             |     at async sendFile (/usr/src/app/dist/utils/file.js:51:9)
server-1                             |     at async PersonController.getPersonThumbnail (/usr/src/app/dist/controllers/person.controller.js:52:9)
server-1                             | [Nest] 7  - 06/02/2024, 12:48:30 PM   ERROR [ImmichServer] [LoggerRepository] Unable to send file: Error
server-1                             | Error: ENOENT: no such file or directory, access 'upload/thumbs/cb765e07-ebd7-4903-a0c7-41598e43d1cf/e5/24/e524e25a-fda9-4636-a5af-3e40d3f6ed40.jpeg'
server-1                             |     at async access (node:internal/fs/promises:603:10)
server-1                             |     at async sendFile (/usr/src/app/dist/utils/file.js:51:9)
server-1                             |     at async PersonController.getPersonThumbnail (/usr/src/app/dist/controllers/person.controller.js:52:9)
server-1                             | [Nest] 7  - 06/02/2024, 12:48:30 PM    WARN [ImmichServer] [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
server-1                             | [Nest] 7  - 06/02/2024, 12:48:30 PM   ERROR [ImmichServer] [ExceptionsHandler] ENOENT: no such file or directory, access 'upload/thumbs/cb765e07-ebd7-4903-a0c7-41598e43d1cf/e5/24/e524e25a-fda9-4636-a5af-3e40d3f6ed40.jpeg'
server-1                             | Error: ENOENT: no such file or directory, access 'upload/thumbs/cb765e07-ebd7-4903-a0c7-41598e43d1cf/e5/24/e524e25a-fda9-4636-a5af-3e40d3f6ed40.jpeg'
server-1                             |     at async access (node:internal/fs/promises:603:10)
server-1                             |     at async sendFile (/usr/src/app/dist/utils/file.js:51:9)
server-1                             |     at async PersonController.getPersonThumbnail (/usr/src/app/dist/controllers/person.controller.js:52:9)
server-1                             | [Nest] 7  - 06/02/2024, 12:48:30 PM   ERROR [ImmichServer] [LoggerRepository] Unable to send file: Error
server-1                             | Error: ENOENT: no such file or directory, access 'upload/thumbs/cb765e07-ebd7-4903-a0c7-41598e43d1cf/d9/5f/d95f9dca-7310-4eee-9fa7-de8766b0c871.jpeg'
server-1                             |     at async access (node:internal/fs/promises:603:10)
server-1                             |     at async sendFile (/usr/src/app/dist/utils/file.js:51:9)
server-1                             |     at async PersonController.getPersonThumbnail (/usr/src/app/dist/controllers/person.controller.js:52:9)
server-1                             | [Nest] 7  - 06/02/2024, 12:48:30 PM   ERROR [ImmichServer] [ExceptionsHandler] ENOENT: no such file or directory, access 'upload/thumbs/cb765e07-ebd7-4903-a0c7-41598e43d1cf/d9/5f/d95f9dca-7310-4eee-9fa7-de8766b0c871.jpeg'
server-1                             | Error: ENOENT: no such file or directory, access 'upload/thumbs/cb765e07-ebd7-4903-a0c7-41598e43d1cf/d9/5f/d95f9dca-7310-4eee-9fa7-de8766b0c871.jpeg'
server-1                             |     at async access (node:internal/fs/promises:603:10)
server-1                             |     at async sendFile (/usr/src/app/dist/utils/file.js:51:9)
server-1                             |     at async PersonController.getPersonThumbnail (/usr/src/app/dist/controllers/person.controller.js:52:9)
server-1                             | [Nest] 7  - 06/02/2024, 12:48:30 PM    WARN [ImmichServer] [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses

Additional information

mertalev commented 4 months ago

Hi, thanks for the report!

Person thumbnails are currently always in JPEG format. We can change this to respect the thumbnail format. However, this isn't problematic in itself and doesn't explain why the thumbnails didn't get generated after running on all assets.

Can you set the log level to verbose, run thumbnail generation on all assets and immediately pause the queue? After it finishes queueing, can you check if there are any errors in the microservices container?

If there are no errors, then unpause it and just let it run. The person thumbnails will be generated last, at which point we can look at the logs relating to them.

heikomat commented 4 months ago

There are indeed some ffprobe-errors. Here is the log-output when pausing right after starting and letting all entries queue:

[Nest] 7  - 06/02/2024, 5:30:21 PM   ERROR [ImmichMicroservices] [JobService] Unable to run job handler (thumbnailGeneration/generate-preview): Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x2f988190180] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x2f988190180] moov atom not found
upload/library/heiko/2020/VID_2020-10-05T22-39-38.mp4: Invalid data found when processing input

[Nest] 7  - 06/02/2024, 5:30:21 PM   ERROR [ImmichMicroservices] [JobService] Error: ffprobe exited with code 1
ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x2f988190180] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x2f988190180] moov atom not found
upload/library/heiko/2020/VID_2020-10-05T22-39-38.mp4: Invalid data found when processing input

    at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
    at ChildProcess.emit (node:events:519:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12)
[Nest] 7  - 06/02/2024, 5:30:21 PM   ERROR [ImmichMicroservices] [JobService] Object:
{
  "id": "00491753-417c-4f0f-8b98-bf6d34526f74"
}

[Nest] 7  - 06/02/2024, 5:30:21 PM     LOG [ImmichMicroservices] [MediaService] Successfully generated WEBP image preview for asset 1eea9563-2157-4cbe-95d6-a4dfccd7a3e8
[Nest] 7  - 06/02/2024, 5:30:21 PM     LOG [ImmichMicroservices] [MediaService] Successfully generated WEBP image preview for asset 2345b53d-d01f-4729-9dc2-2908b9a766d1
[Nest] 7  - 06/02/2024, 5:30:21 PM     LOG [ImmichMicroservices] [MediaService] Successfully generated WEBP image preview for asset 4baaf760-42fa-4ef9-b096-cc8fdb4b212c
[Nest] 7  - 06/02/2024, 5:30:21 PM     LOG [ImmichMicroservices] [MediaService] Successfully generated WEBP image preview for asset 32741a15-91f3-4149-9532-ed899f838bbc
[Nest] 7  - 06/02/2024, 5:30:22 PM     LOG [ImmichMicroservices] [MediaService] Successfully generated WEBP image preview for asset a1de5941-154b-409c-94cc-f723fe913aec
[Nest] 7  - 06/02/2024, 5:30:22 PM     LOG [ImmichMicroservices] [MediaService] Successfully generated WEBP image preview for asset 641caa42-7a49-4175-8030-c422f060debe
[Nest] 7  - 06/02/2024, 5:30:22 PM     LOG [ImmichMicroservices] [MediaService] Successfully generated WEBP image preview for asset 75e67e2b-a35e-473f-9a5e-6ffad9b26d02
[Nest] 7  - 06/02/2024, 5:30:22 PM     LOG [ImmichMicroservices] [MediaService] Successfully generated WEBP image preview for asset 054733fc-0dec-4f09-ba57-028e88e4a7aa
[Nest] 7  - 06/02/2024, 5:30:23 PM     LOG [ImmichMicroservices] [MediaService] Successfully generated WEBP image preview for asset a6276a64-dada-4e8b-98e7-f3ede1d63881
[Nest] 7  - 06/02/2024, 5:30:23 PM     LOG [ImmichMicroservices] [MediaService] Successfully generated WEBP image preview for asset 31d20541-261f-492b-bac3-2dfc74f010af
[Nest] 7  - 06/02/2024, 5:30:23 PM     LOG [ImmichMicroservices] [MediaService] Successfully generated WEBP image preview for asset 62b98e3a-2efd-49e5-9e97-b57e21f7d6d6

I will now let it run and will update you when the logs are available

heikomat commented 4 months ago

The Job is still running (80k thumbnails to go) but there are already errors in the logs that might be of intereset. A lot of these appeared:

[Nest] 7  - 06/02/2024, 7:54:58 PM   ERROR [ImmichMicroservices] [JobService] Unable to run job handler (thumbnailGeneration/generate-person-thumbnail): Error: Expected integer for left but received NaN of type number
[Nest] 7  - 06/02/2024, 7:54:58 PM   ERROR [ImmichMicroservices] [JobService] Error: Expected integer for left but received NaN of type number
    at Object.invalidParameterError (/usr/src/app/node_modules/sharp/lib/is.js:135:10)
    at Sharp.<anonymous> (/usr/src/app/node_modules/sharp/lib/resize.js:478:16)
    at Array.forEach (<anonymous>)
    at Sharp.extract (/usr/src/app/node_modules/sharp/lib/resize.js:473:38)
    at MediaRepository.generateThumbnail (/usr/src/app/dist/repositories/media.repository.js:60:22)
    at PersonService.handleGeneratePersonThumbnail (/usr/src/app/dist/services/person.service.js:427:36)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /usr/src/app/dist/services/job.service.js:145:36
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7  - 06/02/2024, 7:54:58 PM   ERROR [ImmichMicroservices] [JobService] Object:
{
  "id": "25418497-843b-4712-ad95-906d6a6d06b8"
}

I will provide the full logs when the job has finished

mertalev commented 4 months ago

Oh, that's really helpful actually. Could you run this command and share the output?

docker exec immich_postgres psql -U postgres -c "SELECT jsonb_pretty(jsonb_build_object('id', a.id, 'type', a.type, 'height', e.\"exifImageHeight\", 'width', e.\"exifImageWidth\", 'originalPath', a.\"originalPath\", 'previewPath', a.\"previewPath\", 'personThumbnailPath', p.\"thumbnailPath\")) as pretty FROM assets a INNER JOIN exif e ON a.id = e.\"assetId\" INNER JOIN asset_faces af on a.id = af.\"assetId\" INNER JOIN person p on af.id = p.\"faceAssetId\" WHERE p.id = '25418497-843b-4712-ad95-906d6a6d06b8'" immich
heikomat commented 4 months ago

Oh, that's really helpful actually. Could you run this command and share the output?

docker exec immich_postgres psql -U postgres -c "SELECT jsonb_pretty(jsonb_build_object('id', a.id, 'type', a.type, 'height', e.\"exifImageHeight\", 'width', e.\"exifImageWidth\", 'originalPath', a.\"originalPath\", 'previewPath', a.\"previewPath\", 'personThumbnailPath', p.\"thumbnailPath\")) as pretty FROM assets a INNER JOIN exif e ON a.id = e.\"assetId\" INNER JOIN asset_faces af on a.id = af.\"assetId\" INNER JOIN person p on af.id = p.\"faceAssetId\" WHERE p.id = '25418497-843b-4712-ad95-906d6a6d06b8'" immich

I will do that first thing tomorrow morning after making the first coffee :)

heikomat commented 4 months ago

Interesting. After this run about 50% of faces reappeared. Here are the logs: immich_microservices_redacted.log.zip

Will run the query when i'm at my desk in about 1:30h

heikomat commented 4 months ago

@mertalev Here's the query result:

docker compose exec -it database psql -U postgres -c "SELECT jsonb_pretty(jsonb_build_object('id', a.id, 'type', a.type, 'height', e.\"exifImageHeight\", 'width', e.\"exifImageWidth\", 'originalPath', a.\"originalPath\", 'previewPath', a.\"previewPath\", 'personThumbnailPath', p.\"thumbnailPath\")) as pretty FROM assets a INNER JOIN exif e ON a.id = e.\"assetId\" INNER JOIN asset_faces af on a.id = af.\"assetId\" INNER JOIN person p on af.id = p.\"faceAssetId\" WHERE p.id = '25418497-843b-4712-ad95-906d6a6d06b8'" immich
WARN[0000] /docker/immich/compose.yaml: `version` is obsolete
                                                              pretty
----------------------------------------------------------------------------------------------------------------------------------
 {                                                                                                                               +
     "id": "20242acd-cc7e-46bc-9978-fc12def291e2",                                                                               +
     "type": "IMAGE",                                                                                                            +
     "width": 4160,                                                                                                              +
     "height": 3120,                                                                                                             +
     "previewPath": "upload/thumbs/cb765e07-ebd7-4903-a0c7-41598e43d1cf/20/24/20242acd-cc7e-46bc-9978-fc12def291e2-preview.webp",+
     "originalPath": "upload/library/heiko/2016/IMG_2016-07-05T11-22-15.jpg",                                                    +
     "personThumbnailPath": "upload/thumbs/cb765e07-ebd7-4903-a0c7-41598e43d1cf/25/41/25418497-843b-4712-ad95-906d6a6d06b8.jpeg" +
 }
(1 row)

I checked, the file in "originalPath" and "previewPath" do exist. The jpeg in "personThumbnailPath" does not exist. The Folder is there (upload/thumbs/cb765e07-ebd7-4903-a0c7-41598e43d1cf/25/41) but it's empty (no webp, no jpeg)

mertalev commented 4 months ago

Interesting! I was expecting either a video or an image with dimensions that don't scale cleanly, but this asset doesn't seem like an edge case. Unless it's maybe stemming from the face bounding box? I'll take a closer look at this tomorrow.

heikomat commented 4 months ago

@mertalev if it helps, this is the original image you told me to query (asset 20242acd-cc7e-46bc-9978-fc12def291e2, IMG_2016-07-05T11-22-15.jpg) IMG_2016-07-05T11-22-15

mertalev commented 3 months ago

This is probably fixed with #10382. At the very least, I've tried changing the preview for this image to different resolutions and regenerating the person thumbnail and it's worked for every combination. Tentatively closing, but feel free to re-open if you still have the issue.

heikomat commented 3 months ago

This sounds like that could be the cause of my issues here. Will test this as soon it is available in an immich version

heikomat commented 3 months ago

@mertalev sadly, even after updating to 1.7.2 and rerunning the generation of all thumbnails, still about half of the face previews are missing (just like before the update) :(

Could you please re-open the issue? I don't have the rights to do so.

mertalev commented 3 months ago

Are the errors in the logs the same as before?

heikomat commented 3 months ago

Thanks for reopening! I had no time to check the logs yet. I will do so soon

heikomat commented 3 months ago

@mertalev I checked the logs now. The Error seems to be the same. When visiting the "Explore" page, there are 11 faces as preview. 7 of them are missing for me. When looking at the logs just after visiting, i see 7 errors that look like this:

Error: ENOENT: no such file or directory, access 'upload/thumbs/cb765e07-ebd7-4903-a0c7-41598e43d1cf/37/82/37826645-0eca-4d44-a56b-3cb78aa45088.jpeg'
    at async access (node:internal/fs/promises:603:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:51:9)
    at async PersonController.getPersonThumbnail (/usr/src/app/dist/controllers/person.controller.js:52:9)
[Nest] 17  - 07/04/2024, 6:56:07 PM    WARN [Api:ExpressAdapter~syd3irtb] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 17  - 07/04/2024, 6:56:07 PM   ERROR [Api:ExceptionsHandler~syd3irtb] ENOENT: no such file or directory, access 'upload/thumbs/cb765e07-ebd7-4903-a0c7-41598e43d1cf/37/82/37826645-0eca-4d44-a56b-3cb78aa45088.jpeg'
Error: ENOENT: no such file or directory, access 'upload/thumbs/cb765e07-ebd7-4903-a0c7-41598e43d1cf/37/82/37826645-0eca-4d44-a56b-3cb78aa45088.jpeg'
    at async access (node:internal/fs/promises:603:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:51:9)
    at async PersonController.getPersonThumbnail (/usr/src/app/dist/controllers/person.controller.js:52:9)
[Nest] 17  - 07/04/2024, 6:56:07 PM   ERROR [Api:LoggerRepository~gj798lhh] Unable to send file: Error

When seaerching for one of the file-ids in the whole log, i only see one other Error that happened while generating the thumbnails:

[Nest] 7  - 07/03/2024, 12:35:40 PM   ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-person-thumbnail): Error: Expected integer for left but received NaN of type number
[Nest] 7  - 07/03/2024, 12:35:40 PM   ERROR [Microservices:JobService] Error: Expected integer for left but received NaN of type number
    at Object.invalidParameterError (/usr/src/app/node_modules/sharp/lib/is.js:135:10)
    at Sharp.<anonymous> (/usr/src/app/node_modules/sharp/lib/resize.js:483:16)
    at Array.forEach (<anonymous>)
    at Sharp.extract (/usr/src/app/node_modules/sharp/lib/resize.js:478:38)
    at MediaRepository.generateThumbnail (/usr/src/app/dist/repositories/media.repository.js:60:22)
    at PersonService.handleGeneratePersonThumbnail (/usr/src/app/dist/services/person.service.js:422:36)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /usr/src/app/dist/services/job.service.js:148:36
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7  - 07/03/2024, 12:35:40 PM   ERROR [Microservices:JobService] Object:
{
  "id": "37826645-0eca-4d44-a56b-3cb78aa45088"
}
mertalev commented 3 months ago

Thanks for the logs! Hmm, the ENOENT error makes sense, but I'll need to take another look at the NaN error.

luckyycode commented 2 months ago

any news on this? I am not sure if it is related, but it is the issue I found first when I was investigating the problem I just had. I just deployed Immich v1.107.2 with external library and funny thing, that I can see blurred previews, yet when I open any image it says error and in logs I see "ENOENT, no such file or directory, access "upload/thumbs/guid/whatever/whatever/filename in guid.jpeg"

I am running Immich in kubernetes container (not helm-based from repo), all paths are correct, yet I don't understand

paths in k8s: image

import paths in Immich: image

edit: I have also noticed that image previews are broken randomly. I got enough of space and there are no errors in logs, it just shows the broken preview thumbnail and when I open some image I get that "broken thumbnail" image in full preview. AND even if I can see the preview thumbnail in high quality and when I click on an image, I see "Error loading image" text, tho thumbnail loaded correctly

mertalev commented 2 months ago

@luckyycode This issue is about an edge case when cropping person thumbnails, not relevant to your issue. You can try generating thumbnails for all assets through the job panel. It will either fix your issue, or you'll get error logs describing why the thumbnail or preview couldn't be generated.

freitzzz commented 2 months ago

Hi, I had the same issue even after switching back to JPEG. The issue lied on the immich-microservices image that was removed from the compose file on newer versions, and I think it was using a different image version, despite being removed from the file. I passed the flag --remove-orphans the next time I launched immich and it started working.

heikomat commented 2 months ago

@freitzzz i had no orphaned images or containers around, but tried --remove-orphans anyway. Didn't change anything for me though, but thanks for trying to help :)

heikomat commented 1 month ago

I'm currently investigating this. After looking at the stacktrace from earlier, i added a log in getCrop here that looks like this:

    this.logger.log(
      `getting crop ${[
        dims.new.width, dims.old.width,
        dims.new.height, dims.old.height,
        widthScale, heightScale,
        x2, x1, y2, y1,
        halfWidth, halfHeight,
        middleX, middleY,
        targetHalfSize,
        newHalfSize,
        middleX - newHalfSize,
        middleY - newHalfSize,
        newHalfSize * 2,
      ].join(', ')}`,
    );

immich hasn't finished regenerating all thumbnails yet, but i do have logs of failed person thumbnails. Here is one example:

LOG [Microservices:PersonService] getting crop 4032, 0, 3024, 0, Infinity, Infinity, 0, 0, 0, 0, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN, NaN

According to this

Next step would be to find out where dims.old comes from and why it has no size

heikomat commented 1 month ago

Turns out i have a lot of entries in asset_faces that don't have an imageWidth (or any size/dimension for that matter).

select count(*) from asset_faces where "imageWidth" = 0;
 16125

select count(*) from asset_faces where "imageWidth" <> 0;
 71129

From the code i've seen i strongly assume that that's the reason why getCrop fails, which is the reason i have missing people-faces

@mertalev i'm not sure what the best way to continue is. I see a couple of possibilities:

  1. update getCrop so that it doesn't fail when dims.old is not correct
  2. just delete the broken entries from asset_faces (can i just safely do that, or would that break things)
  3. is there maybe a job that would fix the entries in asset_faces that i could run?
  4. write a migration to fix the broken database entries (if that is even possible)

Would be interesting to know how these entries ended up there. What do you think?

mertalev commented 1 month ago

That's pretty strange. There's not much we can do with a bounding box if we don't know the dimensions of the image it's associated with.

I'm working on a way to re-run face detection and merge the results. Maybe I can make it update the image dimensions if they're different.

If you know what the actual preview dimensions are, you can update the DB yourself. Or if you at least know that images taken with X camera end up with certain preview dimensions, you can join on the exif table and only update those face entries.

heikomat commented 1 month ago

@mertalev Thank's for the quick reply!

As you mention the bounding-box values, they are also 0 for these entries: Screenshot 2024-08-23 at 21 39 50

Am i correct to assume that these entries are therefore damaged beyond repair?

Updating the values by hand sound tedious and error-prone. I would be fine with deleting them and just re-running face-detection, if it safe to assume that deleting these entries doesn't cause any other issues or inconsistencies.

i know the person-table references the asset_faces table, so i would also have to delete entries in there. But as someone who is not deep in the codebase, i can't assess whether or not other tables need to also be updated, and how complex such a cleanup would get.

mertalev commented 1 month ago

I would be fine with deleting them and just re-running face-detection

You can just re-run face detection on all assets and it will clean everything up.

heikomat commented 1 month ago

I would be fine with deleting them and just re-running face-detection

You can just re-run face detection on all assets and it will clean everything up.

That worked :)

I have to re-assign names to faces, but that's fine (for me). immich warned me that that would happen if i confirm to re-run face-detection on all assets.

But there are now no more missing face thumbnails :)

jrasm91 commented 1 month ago

I believe this issue has been resolved now. We did find a case with changing webp to jpeg and running the thumbnail job could overwrite thumbnails.