immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
51.73k stars 2.74k forks source link

[BUG] Concurrency not respected upon container rebuild #6285

Closed undaunt closed 4 months ago

undaunt commented 10 months ago

The bug

I modified several high CPU concurrency settings (faces and smart search) from 2 down to 1. An hour or two later, I then decided to set some CPU limiter settings in docker compose and recreated the container. Upon initial launch, both jobs were running with active of 2 even though the settings for concurrency still showed 1. After hitting save again on concurrency settings, they dropped to 1 active.

The OS that Immich Server is running on

Ubuntu 22.04

Version of Immich Server

1.92.1

Version of Immich Mobile App

N/A

Platform with the issue

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich-server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    restart: unless-stopped
    profiles: ["all","immich"]
    networks:
      - $PROXY_NETWORK
      - $APP_NETWORK
    security_opt:
      - no-new-privileges:true
    depends_on:
      - immich-redis
      - immich-db
    command: [ "start.sh", "immich" ]
    cpus: 1
    volumes:
      - ${UPLOAD_DIR}:/usr/src/app/upload
      - /mnt/media/photos/icloud:/icloud:ro
      - /mnt/users/user/Pictures/Photos:/sync:ro
      - $DOCKERDIR/immich/config/auth.yml:/root/.config/immich/auth.yml:ro
      - /etc/localtime:/etc/localtime:ro
    environment:
      - DB_HOSTNAME=${POSTGRES_IMMICH_HOST}
      - DB_DATABASE_NAME=${POSTGRES_IMMICH_DB}
      - DB_USERNAME=${POSTGRES_IMMICH_USER}
      - DB_PASSWORD=${POSTGRES_IMMICH_PASS}
      - REDIS_HOSTNAME=${IMMICH_REDIS_HOSTNAME}
    labels:
      - traefik.enable=true
      - traefik.http.routers.immich.entrypoints=https
      - traefik.http.routers.immich.rule=Host(`immich.$DOMAINNAME`)
      - traefik.http.routers.immich.middlewares=secure-headers@file,crowdsec@file
      - traefik.http.routers.immich.service=immich
      - traefik.http.services.immich.loadbalancer.server.port=3001

  immich-microservices:
    container_name: immich-microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    restart: unless-stopped
    profiles: ["all","immich"]
    networks:
      - $APP_NETWORK
    security_opt:
      - no-new-privileges:true
    depends_on:
      - immich-redis
      - immich-db
    cpus: 3
    devices:
      - /dev/dri:/dev/dri
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_DIR}:/usr/src/app/upload
      - /mnt/media/photos/icloud:/icloud:ro
      - /etc/localtime:/etc/localtime:ro
    environment:
      - DB_HOSTNAME=${POSTGRES_IMMICH_HOST}
      - DB_DATABASE_NAME=${POSTGRES_IMMICH_DB}
      - DB_USERNAME=${POSTGRES_IMMICH_USER}
      - DB_PASSWORD=${POSTGRES_IMMICH_PASS}
      - REDIS_HOSTNAME=${IMMICH_REDIS_HOSTNAME}

  immich-machine-learning:
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    container_name: immich-machine-learning
    volumes:
      - $DOCKERDIR/immich/models:/cache
    restart: unless-stopped
    profiles: ["all","immich"]
    networks:
      - $APP_NETWORK
    security_opt:
      - no-new-privileges:true

  immich-redis:
    image: redis:6.2-alpine@sha256:b6124ab2e45cc332e16398022a411d7e37181f21ff7874835e0180f56a09e82a
    container_name: immich-redis
    restart: unless-stopped
    profiles: ["all","immich"]
    networks:
      - $APP_NETWORK
    security_opt:
      - no-new-privileges:true
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping | grep PONG"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 3s
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

  immich-db:
    image: tensorchord/pgvecto-rs:pg14-v0.1.11@sha256:0335a1a22f8c5dd1b697f14f079934f5152eaaa216c09b61e293be285491f8ee
    container_name: immich-db
    restart: unless-stopped
    profiles: ["all","immich"]
    networks:
      - $APP_NETWORK
    security_opt:
      - no-new-privileges:true
    environment:
      - POSTGRES_PASSWORD=${POSTGRES_IMMICH_PASS}
      - POSTGRES_USER=${POSTGRES_IMMICH_USER}
      - POSTGRES_DB=${POSTGRES_IMMICH_DB}
    volumes:
      - $DOCKERDIR/immich/db:/var/lib/postgresql/data

Your .env content

I am interpolating from a parent .env in the compose folder but no env_file variable.

Reproduction steps

1. Set concurrency lower (or possibly higher than default)
2. Recreate the container and observe jobs running at default levels
3. Hit save on concurrency page
4. Observe job active levels change again.

Additional information

No response

bo0tzz commented 10 months ago

Can you set your log level to verbose, then reproduce the issue and share the logs here?

undaunt commented 10 months ago

Which container should I pull the logs from, server or microservices?

I noted that the issue only occurs when there is a job already in process when I restart the container. To simulate, I started reprocessing all images for smart search and it correctly only had 1 active at a time. Upon restart, it had 2 until such time that I hit 'save' again on the concurrency settings, where it again dropped to 1.

jrasm91 commented 4 months ago

I believe this has been fixed. Let me know if it is still an issue.