Jobs are not running automatically

alternativesurfer commented 2 weeks ago

The bug

I have to manually click the "missing" button from Admin/Jobs every few days for it to run machine learning, generate thumbnails, transcode videos, etc. All of them.

The OS that Immich Server is running on

Ubuntu 22.10

Version of Immich Server

v1.112.1

Version of Immich Mobile App

1.112.1 build.154

Platform with the issue

[X] Server
[ ] Web
[ ] Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    logging:
      driver: "journald"
      options:
        tag: "immich-server"
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - ${EXTERNAL_PATH}:/usr/src/app/external
      - ${THUMB_LOCATION}:/usr/src/app/upload/thumbs
      - ${ENCODED_VIDEO_LOCATION}:/usr/src/app/upload/encoded-video
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:d6c2911ac51b289db208767581a5d154544f2b2fe4914ea5056443f62dc6e900
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

GNU nano 6.2                                                                               .env
# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/media/photos/immich
EXTERNAL_PATH=/mnt/media/photos/historical
THUMB_LOCATION=/mnt/cache/thumbs
ENCODED_VIDEO_LOCATION=/mnt/cache/encoded-video

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=ISARANDOMPASSWORD

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

Upload image/video
Wait
No jobs are ever kicked off
The daily scheduled jobs also do not appear to run
If I manually click one of the "missing" buttons on the jobs page, all the new items that had not been processed will be picked up and run againt.

Relevant log output

No response

Additional information

No response

alextran1502 commented 2 weeks ago

Can you monitor the jobs page and the server logs, then upload a few photos and see the behavior and update here?

I am thinking it might be some weird issue with your redis deployment. So it might worth to bring down and bring up the whole stack

aihellos commented 2 weeks ago

same issue here. Redis server is external though

alternativesurfer commented 2 weeks ago

Added a photo
reviewing jobs....the following did not run: -- Generate Thumbnails -- Extract Metadata -- Library -- Face Detection -- Facial Recognition
I can then click "Refresh" on the library task, and it kicks off all the above

I tested this before and after restarting all my containers, same behavior.

immich-app / immich