Running STORAGE TEMPLATE MIGRATION does not move files from upload to library

[gomme600]

The bug

Nothing happens when adding a template after initially keeping everything stored in uploads. The files aren't moved. I can see this in the logs for multiple files:

[Nest] 7 - 02/23/2024, 9:02:52 AM ERROR [StorageTemplateService] Asset e7bb92b3-ee49-4cb5-bc4c-7692a6b300cf missing exif info, skipping storage template migration

The OS that Immich Server is running on

Debian 12 (docker)

Version of Immich Server

1.95.1

Version of Immich Mobile App

0

Platform with the issue

• [X] Server
• [ ] Web
• [ ] Mobile

Your docker-compose.yml content

version: "3.8"

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - /tank/important/files/photos/immich:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    command: [ "start.sh", "microservices" ]
    volumes:
      - /tank/important/files/photos/immich:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - /srv/dev-disk-by-label-containers/databases/immich/modelcache:/cache
    env_file:
      - stack.env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5
    restart: always

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - /srv/dev-disk-by-label-containers/databases/immich/postgres:/var/lib/postgresql/data
    restart: always

Your .env content

IMMICH_VERSION=release
DB_PASSWORD=**
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis

Reproduction steps

1.Go to settings, storage template and turn on the storage template
2.Go to jobs and run storage template migration
3.Nothing happens (active 1, waiting 0)

Additional information

No response

[bo0tzz]

Try running extract metadata (missing) to see if that resolves it.

[gomme600]

Try running extract metadata (missing) to see if that resolves it.

I think that the metadata got lost somehow. I am trying again with the template active and a fresh install. I seem to be getting a big backlog during the upload and lots of broken thumbnails. I currently have 1 active, 166 waiting (rapidly increasing) for storage template migration.

[bestrocker221]

Same issue here with latest immich version. I tried also to create a new user and upload again. I uploaded 10 pictures, re-run everything, including metadata and template migration. Only 1 of the pictures get moved (or just copied) to the library. One user in immich (which has content) is not even considered. the user folder does not even get created in the library, all of its content resides in the upload folder. It is very inconsistent. With another user that has content uploaded on multiple days, only the folder for one day has been created in the library (in the last month and a half).

[mertalev]

Can you test with 1.96.0?

[bestrocker221]

Tested with 1.96. Left running the Storage template migration job last night, and its progress has not moved. (actually it cannot even be restarted because it thinks is "running") I still have one user that does not even have a folder in the library at all and one that does but only has few pictues imported correctly, all the rest is in the upload folder. I cannot figure out why.

Although I noticed the logging was improved which is great. I see now that Immich has a wrong path for many pictures, as he is trying to move un-existing elements from upload to library, thus resulting in failure, any way to fix this?

[alessandriniluca]

@bestrocker221 sorry to ping you personally, but did you solved it somehow? Otherwise I think I will stick without storage template for a bit more.

[bestrocker221]

@alessandriniluca not resolved yet. (v1.100) Same issues

[cameronwlewis]

v1.100, same issues.

[TriplEight]

v1.112 Repro:

1. Set up immich. Docker-compose is just different in upload location.
2. Apply a default (for v.1.111) storage template
3. Sync some pictures from phone.
4. Stop this instance.
5. Copy all created library with thumbs and transcoded videos to a new place.
6. Setup a new instance with a volume pointing to that new location.
7. Apply a default storage template.

Result: 0 photos. all the files are in Repair section. None of the Jobs can help it. Phone starts to upload the same photos again.

[alextran1502]

@TriplEight when you setup a new instance, it doesn't have the record of what has been uploaded from the mobile device, so the behavior is expected

[TriplEight]

@TriplEight when you setup a new instance, it doesn't have the record of what has been uploaded from the mobile device, so the behavior is expected

Thanks, but I had to solve the problem with thousands of files in Untracked files and Offline paths. And all the photos were shown super blurry. I'll describe how I solved it, in case someone finds themselves in a similar situation:

1. Ran rmlint in the library dir and removed all the duplicate files with +1 ending in name.
2. After this, in /admin/repair "select all" made the work that I expected from "refresh": updated the results
3. Then in /photos selected all photos and "refresh thumbnails" fixed most of the pictures display.
4. The remaining problems were solved by running "face detection" and "transcode videos" in /admin/jobs-status

Might as well reupload everything, but it gets tricky when you have multiple sources.