Bug after external HDD was disconnected

The bug

My Immich instance (1.108.0) is running on an Intel Nuc under Windows 10 and Docker Desktop 4.31.1. All the photos are located on an external hard drive via USB. In some cases the hdd can be disconnected while Immich is running.

When reconnecting the hdd, all newly uploaded images are working fine, but all the old ones in the library can't be found with the following error in the logs and the thumbnails are all blurry: Error: ENOENT: no such file or directory

I believe the root cause is that the mounts in Docker are broken due to the hdd disconnect. When I look in the mounted directory in Docker, I can for example see that my User library folder is there, but not my wife's if she hasn't uploaded anything since the hdd disconnect. After restarting Docker Desktop completely, I can see all my old images without any error in the Web UI, but the ones which where uploaded between the hdd disconnect and the Docker restart are broken now. In this case I just use a workaround and delete these images (it's usually not more than a few days worth of images), remove them from the recycle bin and reupload them from the app manually (as auto sync ignores deleted images). Looking at the docker mount, I can see all libraries again (including my wife's).

In my opinion it would be great to get a push notification that something went wrong during syncing the images, or just don't upload the images at all, showing an error message.

Is there anything I can do or does this have to be fixed by the devs? I didn't find any similar issues in the open issues.

Anyway, I will probably buy a bigger NVMe and put the Immich libraries on there.

The OS that Immich Server is running on

Win10 x64

Version of Immich Server

1.108.0

Version of Immich Mobile App

1.108.0 Android

Platform with the issue

[X] Server
[ ] Web
[ ] Mobile

Your docker-compose.yml content

version: "3.8"

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:afb290a0a0d0b2bd7537b62ebff1eb84d045c757c1c31ca2ca48c79536c0de82
    restart: always

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    env_file:
      - .env
    ports:
      - 5432:5432
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=Y:\REDACTED\REDACTED

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=REDACTED

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=REDACTED
DB_USERNAME=REDACTED
DB_DATABASE_NAME=REDACTED

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. Accidental disconnect external HDD containing Immich libraries
2. Auto-Sync images from the app
3. Reconnect HDD after noticing it a few days later
4. Look at the web UI and see that old files (before the disconnect) are missing (blurry thumbnails, photos can't be opened although ext. HDD is reconnected)
5. Restart Docker Desktop
6. Old images from before the disconnect are now working again, but new images from after the disconnect until restarting Docker Desktop are now broken and have to be reuploaded manually.

Relevant log output

2024-07-15 17:56:27 [Nest] 19  - 07/15/2024, 3:56:27 PM   ERROR [Api:ExceptionsHandler~kfuc7ccr] ENOENT: no such file or directory, access 'upload/thumbs/81959f46-d741-4239-a389-d2e321ea9d47/24/a4/24a45903-9c50-4641-835e-1b9a89b8bf4e-thumbnail.webp'
2024-07-15 17:56:27 Error: ENOENT: no such file or directory, access 'upload/thumbs/81959f46-d741-4239-a389-d2e321ea9d47/24/a4/24a45903-9c50-4641-835e-1b9a89b8bf4e-thumbnail.webp'
2024-07-15 17:56:27     at async access (node:internal/fs/promises:606:10)
2024-07-15 17:56:27     at async sendFile (/usr/src/app/dist/utils/file.js:51:9)
2024-07-15 17:56:27     at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:57:9)
2024-07-15 17:56:27 [Nest] 19  - 07/15/2024, 3:56:27 PM   ERROR [Api:LoggerRepository~6u7pjrqy] Unable to send file: Error
2024-07-15 17:56:27 Error: ENOENT: no such file or directory, access 'upload/thumbs/81959f46-d741-4239-a389-d2e321ea9d47/1c/e4/1ce4f5d9-a1e1-44af-afb3-465de97a68e1-thumbnail.webp'
2024-07-15 17:56:27     at async access (node:internal/fs/promises:606:10)
2024-07-15 17:56:27     at async sendFile (/usr/src/app/dist/utils/file.js:51:9)
2024-07-15 17:56:27     at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:57:9)

Additional information

No response

immich-app / immich