Update from 1.106.4 => 1.107.2 immich-server can't start

[SuberSeb]

The bug

After updating 1.106.4 => 1.107.2 immich-server container can't start anymore.

I'm running pgvectro postgres image from the official docker-compose, no any changes to the db was made recently. I've tried to update to 0.2.1 but no luck. Vectors extension is installed and updated.

Solutions from this thread didn't help https://github.com/immich-app/immich/issues/10767

The OS that Immich Server is running on

Unraid 6.12.10

Version of Immich Server

v1.107.2

Version of Immich Mobile App

v1.107.2

Platform with the issue

• [X] Server
• [ ] Web
• [ ] Mobile

Your docker-compose.yml content

version: "3.8"

networks:
  immich-network:
    name: immich-network
    ipam:
      config:
        - subnet: 172.20.80.0/24

services:
  immich-server:
    container_name: immich-server
    image: ghcr.io/immich-app/immich-server:release
    extends:
      file: ../addons/immich-gpu.yaml
      service: gpu
    env_file:
      - ../envs/immich.env
    networks:
      - immich-network
    ports:
      - 2283:3001
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ${MNT_CACHE_LOCATION}/encoded-video:/usr/src/app/upload/encoded-video
      - ${MNT_CACHE_LOCATION}/thumbs:/usr/src/app/upload/thumbs
      - ${MNT_CACHE_LOCATION}/profile:/usr/src/app/upload/profile
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped
    mem_limit: 6144m
    cpus: 8

  immich-machine-learning:
    container_name: immich-machinelearning
    image: ghcr.io/immich-app/immich-machine-learning:release-cuda
    extends:
      file: ../addons/immich-gpu.yaml
      service: gpu
    env_file:
      - ../envs/immich.env
    networks:
      - immich-network
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ${MNT_CACHE_LOCATION}/encoded-video:/usr/src/app/upload/encoded-video
      - ${MNT_CACHE_LOCATION}/thumbs:/usr/src/app/upload/thumbs
      - ${MNT_CACHE_LOCATION}/profile:/usr/src/app/upload/profile
      - ${MNT_CACHE_LOCATION}/machinelearning/cache:/cache
    restart: unless-stopped
    mem_limit: 6144m
    cpus: 8

  immich-postgres:
    container_name: immich-postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.1
    command:
      [
        "postgres",
        "-c",
        "shared_preload_libraries=vectors.so",
        "-c",
        'search_path="$$user", public, vectors',
        "-c",
        "logging_collector=on",
        "-c",
        "max_wal_size=2GB",
        "-c",
        "shared_buffers=512MB",
        "-c",
        "wal_compression=on",
      ]
    ulimits:
      nofile:
        soft: 80000
        hard: 80000
    environment:
      - POSTGRES_DB=postgres
      - POSTGRES_PASSWORD=passhere
      - POSTGRES_USER=postgres
      - POSTGRES_INITDB_ARGS='--data-checksums'
      - TZ=Europe/Moscow
    networks:
      - immich-network
    ports:
      - 5450:5432/tcp
    volumes:
      - /mnt/user/Appdata/immich/postgres:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    restart: unless-stopped

  immich-redis:
    container_name: immich-redis
    image: docker.io/redis:6.2-alpine
    environment:
      - TZ=Europe/Moscow
    networks:
      - immich-network
    healthcheck:
      test: redis-cli ping || exit 1
    restart: unless-stopped

Your .env content

# Postgres
DB_HOSTNAME=192.168.1.45
DB_PORT=5450
DB_USERNAME=immichuser
DB_PASSWORD=passhere
DB_DATABASE_NAME=immich

# Redis
REDIS_HOSTNAME=immich-redis
REDIS_PORT=6379
REDIS_DBINDEX=5

# Config
UPLOAD_LOCATION=/mnt/user/Photos/Immich/Upload
LOG_LEVEL=log
JWT_SECRET=sec
DISABLE_REVERSE_GEOCODING=false
REVERSE_GEOCODING_PRECISION=1

# Cache
MNT_CACHE_LOCATION=/mnt/user/Apps/immich

Reproduction steps

Just start the stack.

Relevant log output

[Nest] 7  - 07/04/2024, 4:31:15 PM     LOG [Microservices:EventRepository] Initialized websocket server Migration "AddFaceSearchRelation1718486162779" failed, error: type "vector" does not exist QueryFailedError: type "vector" does not exist at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async AddFaceSearchRelation1718486162779.up (/usr/src/app/dist/migrations/1718486162779-AddFaceSearchRelation.js:12:9) at async MigrationExecutor.executePendingMigrations (/usr/src/app/node_modules/typeorm/migration/MigrationExecutor.js:225:17) at async DataSource.runMigrations (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:265:35) at async DatabaseRepository.runMigrations (/usr/src/app/dist/repositories/database.repository.js:169:9) at async /usr/src/app/dist/services/database.service.js:134:17 at async /usr/src/app/dist/repositories/database.repository.js:177:23 { query: '\n' + '            CREATE TABLE face_search (\n' + '            "faceId"  uuid PRIMARY KEY REFERENCES asset_faces(id) ON DELETE CASCADE,\n' + '            embedding  vector(512) NOT NULL )', parameters: undefined,

Additional information

Tried

SET search_path TO "$user", public, vectors;
DROP INDEX face_index;
ALTER TABLE asset_faces ALTER COLUMN embedding SET DATA TYPE real[];
ALTER TABLE asset_faces ALTER COLUMN embedding SET DATA TYPE vector(512);

but not working still

[alextran1502]

After running the queries, did you restart the database?

[SuberSeb]

After running the queries, did you restart the database?

yy and the entire stack multiple times. I'm not sure what I'm doing wrong

[SuberSeb]

This is the whole script that I tried to apply last time:

SET search_path TO "$user", public, vectors; DROP INDEX face_index; ALTER TABLE asset_faces ALTER COLUMN embedding SET DATA TYPE real[];

DROP INDEX clip_index; ALTER TABLE smart_search ALTER COLUMN embedding SET DATA TYPE real[];

DROP EXTENSION vectors;

CREATE EXTENSION vectors

ALTER TABLE asset_faces ALTER COLUMN embedding SET DATA TYPE vector(512);

[alextran1502]

Can you capture the whole logs when the server starting up?

[SuberSeb]

Can you capture the whole logs when the server starting up?

Here it is https://pastebin.com/K9z2fLHg

[SuberSeb]

Following command is executed successfully under postgres user but not under immich-user (I have a separate user for database access): CREATE TABLE face_search ("faceId" uuid PRIMARY KEY REFERENCES asset_faces(id) ON DELETE CASCADE, embedding vector(512) NOT NULL);

Under immich-user I got an error with vector (from the above).

What I'm doing wrong? Maybe permissions issue?

[SuberSeb]

I changed user from the immich-user to the postgres just for testing reasons. And now immich is working ok. I'm not sure where the issue is but it's looks like permissions.

Anyway it's a bad practice to run immich connection to the db via postgres user.

@alextran1502 maybe u have some ideas?

[SuberSeb]

Changed back to the immich-user and it's still working. Seems like a particular migration permissions issue? I'm not sure.

[a-jackson]

I've been getting the same problem although in my .env DB_USERNAME is already postgres. Haven't had much chance to test anything out yet

[werwolfby]

Have the same issue, will try to switch to postgres user.

[WyrmLabs]

This appears to be very similar to what we have in this other thread: https://github.com/imagegenius/docker-immich/issues/395 Running a few of the commands there helped fix mine.

[a-jackson]

I restored a backup from before the upgrade and then upgraded again and it's worked this time.

[mishak87]

First let me say I like Immich and thank you for improving it! I don't like upgrading immich because mobile app refuses to work only to spend hour debugging ML features that I don't use so server can start.

Since my experience with building CloudNative PG image with vectors extensions and debugging several issues with that extensions from broken upgrades and access issues. I don't consider vectors extension mature product. My DB was untouched running Immich 1.101.0 cleanly installed vectors 0.2.1 on pg 16.2. But todays migration running as immich user did not work because of access rights to internals of vectors extension 😖

If you are still set on changing ML design please make all ML tables droppable and if they are missing on server start just recreate them. And/or anything using vectors.vector type. The extension cleanly installed 0.2.1 did not work with this migration under psql immich user.

Please use full column names when moving data between tables with same column names and dropping them after. Ommiting them leads to different and more confusing errors

https://github.com/immich-app/immich/blob/main/server/src/migrations/1718486162779-AddFaceSearchRelation.ts#L36-L42 SELECT id, embedding refers to face_search.embedding when run second time

      await queryRunner.query(`
        INSERT INTO face_search("faceId", embedding)
        SELECT id, embedding
        FROM asset_faces faces`);
    }

    await queryRunner.query(`ALTER TABLE asset_faces DROP COLUMN IF EXISTS embedding`);

[mishak87]

@werwolfby Running grant helped me despite the fact I had clean install of vectors 0.2.1 (because deb does not include last line of finalize.sql) https://github.com/tensorchord/pgvecto.rs/blob/152f94aceaf9480dd8012e9901721ecd140d7f10/src/sql/finalize.sql#L1011

psql immich # as postgres
immich=> GRANT USAGE ON SCHEMA vectors TO immich;
immich=> GRANT SELECT ON TABLE pg_vector_index_stat TO PUBLIC;