Restore procedure leads to non-updatable Immich installation

erikvanoosten commented 10 months ago

The bug

About a month ago I had to restore from backup. For this I followed the procedure that is outlined in the Immich documentation (see detailed steps below). Unfortunately, the restore process leaves the docker-compose installation in a weird state that prevents further updates.

The OS that Immich Server is running on

Ubuntu 22.04

Version of Immich Server

v1.93.2

Version of Immich Mobile App

NA

Platform with the issue

[X] Server
[ ] Web
[ ] Mobile

Your docker-compose.yml content

version: "3.8"

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

# See https://stackoverflow.com/a/73427527
# We are setting env variable COMPOSE_PROJECT_NAME=immich
# instead of the following line:
# name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:80cc8518800438c684a53ed829c621c94afd1087aaeb59b0d4343ed3e7bcf6c5
    restart: always

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

###################################################################################
# Database
###################################################################################

# NOTE: The following four database variables support Docker secrets by adding a *_FILE suffix to the variable name
# See the docker-compose documentation on secrets for additional details: https://docs.docker.com/compose/compose-file/compose-file-v3/#secrets
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=****
DB_DATABASE_NAME=immich

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich_redis

# REDIS_URL will be used to pass custom options to ioredis.
# Example for Sentinel
# {"sentinels":[{"host":"redis-sentinel-node-0","port":26379},{"host":"redis-sentinel-node-1","port":26379},{"host":"redis-sentinel-node-2","port":26379}],"name":"redis-sentinel"}
# REDIS_URL=ioredis://eyJzZW50aW5lbHMiOlt7Imhvc3QiOiJyZWRpcy1zZW50aW5lbDEiLCJwb3J0IjoyNjM3OX0seyJob3N0IjoicmVkaXMtc2VudGluZWwyIiwicG9ydCI6MjYzNzl9XSwibmFtZSI6Im15bWFzdGVyIn0=

# Optional Redis settings:

# Note: these parameters are not automatically passed to the Redis Container
# to do so, please edit the docker-compose.yml file as well. Redis is not configured
# via environment variables, only redis.conf or the command line

# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_USERNAME=
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION=/mnt/md2/srv/immich-data

###################################################################################
# Typesense
###################################################################################
TYPESENSE_API_KEY=****
# TYPESENSE_ENABLED=false
# TYPESENSE_URL uses base64 encoding for the nodes json.
# Example JSON that was used:
# [
#      { "host": "typesense-1.example.net", "port": "443", "protocol": "https" },
#      { "host": "typesense-2.example.net", "port": "443", "protocol": "https" },
#      { "host": "typesense-3.example.net", "port": "443", "protocol": "https" },
# ]
# TYPESENSE_URL=ha://WwogIHsgImhvc3QiOiAidHlwZXNlbnNlLTEuZXhhbXBsZS5uZXQiLCAicG9ydCI6ICI0NDMiLCAicHJvdG9jb2wiOiAiaHR0cHMiIH0sCiAgeyAiaG9zdCI6ICJ0eXBlc2Vuc2UtMi5leGFtcGxlLm5ldCIsICJwb3J0IjogIjQ0MyIsICJwcm90b2NvbCI6ICJodHRwcyIgfSwKICB7ICJob3N0IjogInR5cGVzZW5zZS0zLmV4YW1wbGUubmV0IiwgInBvcnQiOiAiNDQzIiwgInByb3RvY29sIjogImh0dHBzIiB9Cl0=

###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

# DISABLE_REVERSE_GEOCODING=false
# REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>demo@demo.de</i><br>Password: <i>demo</i>"
####################################################################################

PUBLIC_LOGIN_PAGE_MESSAGE=

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

####################################################################################
# Alternative API's External Address - Optional
#
# This is an advanced feature used to control the public server endpoint returned to clients during Well-known discovery.
# You should only use this if you want mobile apps to access the immich API over a custom URL. Do not include trailing slash.
# NOTE: At this time, the web app will not be affected by this setting and will continue to use the relative path: /api
# Examples: http://localhost:3001, http://immich-api.example.com, etc
####################################################################################

#IMMICH_API_URL_EXTERNAL=http://localhost:3001

###################################################################################
# Immich Version - Optional
#
# This allows all immich docker images to be pinned to a specific version. By default,
# the version is "release" but could be a specific version, like "v1.59.0".
###################################################################################

IMMICH_VERSION=v1.93.2

Reproduction steps

Install Immich v1.92.1 using docker-compose
Do a backup and restore as described on https://immich.app/docs/administration/backup-and-restore/, but with fixes as described on https://github.com/immich-app/immich/issues/5630#issuecomment-1872201525.
Update Immich to v1.93.2 (the update script I used is in this gist: https://gist.github.com/erikvanoosten/3991fff2ea17c1e40d062f234009ce00)

The update results in an error:

root:~/docker.immich# ./update.sh
Getting latest version of Immich...
Update from v1.92.1 to v1.93.2?
1) Update
2) Skip
#? 1
Updating Immich...
Pulling immich-machine-learning ... done
Pulling redis                   ... done
Pulling database                ... done
Pulling immich-microservices    ... done
Pulling immich-server           ... done
Creating immich_postgres         ... error
Creating immich_machine_learning ...
Creating immich_redis            ...

Creating immich_machine_learning ... error
2a7f5eaf8facad". You have to remove (or rename) that container to be able to reuse that name.

ERROR: for immich_machine_learning  Cannot create container for service immich-machine-learning: Conflict. The container name "/immich_machine_learning" is already in use by container "f3584a9ccf993a5ede7
Creating immich_redis            ... error

ERROR: for immich_redis  Cannot create container for service redis: Conflict. The container name "/immich_redis" is already in use by container "786ba9ae4a63a330dc8c894aa51bf2f02f808322f2f0784c6ec6668b17b8acd3". You have to remove (or rename) that container to be able to reuse that name.

ERROR: for database  Cannot create container for service database: Conflict. The container name "/immich_postgres" is already in use by container "8c0d14348de07f7941fd96663ee274022e4c2a7f5eaf8fbc8fc780d32c2dacad". You have to remove (or rename) that container to be able to reuse that name.

ERROR: for immich-machine-learning  Cannot create container for service immich-machine-learning: Conflict. The container name "/immich_machine_learning" is already in use by container "f3584f993a4c3fe84dcc5ede76028372158ab72ebab9cb7dcfa233a929691aac". You have to remove (or rename) that container to be able to reuse that name.

ERROR: for redis  Cannot create container for service redis: Conflict. The container name "/immich_redis" is already in use by container "786ba51ba07832f2f09ae46330dc8c8a632f4c94aa2f8082fec6668b17b8acd3". You have to remove (or rename) that container to be able to reuse that name.
ERROR: Encountered errors while bringing up the project.

The following workaround can be used to get the project working again:

In the docker-compose.yaml file remove all lines starting with container_name
docker-compose up -d
Restore the lines starting with container_name
docker-compose up -d

Additional information

I am using Ubuntu's docker.io package, not the docker community edition. However, from similar questions on Stackoverflow I understood that the docker community edition has the same problem.

dpkg -s docker.io | grep '^Version:'
Version: 24.0.5-0ubuntu1~22.04.1

erikvanoosten commented 10 months ago

Would it be possible to update the machine names in the .env file to the names that docker-compose assigns by default, and then permanently remove the container_name fields from the docker-compose file?

bo0tzz commented 10 months ago

You probably shouldn't blindly be pulling new versions of the compose file like that. The recommended way to upgrade when there are no changes to the docker-compose.yml is to just run docker compose pull && docker compose up -d. Can you try that method?

erikvanoosten commented 10 months ago

@bo0tzz I do not understand your comment. I am not blindly pulling the compose file and I am doing exactly what you are proposing.

bo0tzz commented 10 months ago

Apologies, for some reason I understood the update script you linked as downloading the new compose file, which it obviously doesn't...

Does the issue still happen when you don't do the restore procedure? What about when you don't use the COMPOSE_PROJECT_NAME env var?

erikvanoosten commented 10 months ago

No worries, apologies accepted. I can and will test your suggestion as soon as there is a new release of Immich.

To be honest, I have no idea what the COMPOSE_PROJECT_NAME env var does. I tried to follow the documented docker-compose changes (when the name property was added) and for some reason I forgot ended up using this env var instead.

erikvanoosten commented 10 months ago

Thanks @bo0tzz ! Your hunch was correct. After removing the COMPOSE_PROJECT_NAME env variable, I can update Immich (now at v1.93.3) without problems.

A plausible scenario is that when I performed the restore procedure, the env var was not set. Normally, setting it is taken care of from the update script, so it is not something I had in mind.

I guess I will continue using docker-compose without the COMPOSE_PROJECT_NAME env var, and also without the name parameter.

immich-app / immich