Higher periodic CPU usage since 1.160.2

[ykrasik]

The bug

Hi,

Since upgrading to version 1.106.2 I noticed increased periodic CPU usage in both immich-server and immich-machine-learning. There are spikes of cpu usage in immich-machine-learning every 6-7 minutes and in immich-server every 12-15 minutes:

Before

After

After upgrading to 1.106.2 I immediately ran the duplicates detection & thumbnail generation jobs, so in case one of those jobs got the system into an invalid state which now causes these periodic spikes - I wouldn't be able to provide the CPU graph before running those jobs.

I'm running this on an Orange Pi 5+ (Rockchip RK3588). Also a while ago I tried to get hw acceleration to work - as far as I recall I managed to make it work for transcoding but not for ML, and left it in this unfinished state. AFAICT it's working, just not utilizing the hardware.

The logs for immich-server show this warning periodically:

[Nest] 17  - 06/14/2024, 7:55:53 AM    WARN [Api:DuplicateService~b4mce4tp] Unknown keys found: {
  "machineLearning": {
    "classification": {
      "enabled": true,
      "modelName": "google/vit-base-patch16-224"
    }
  }
}

My CLIP model is ViT-H-14-quickgelu__dfn5b and facial recognition is buffalo_l.

I use centralized postrgres (16.3) & redis (7.2.5) instances.

The OS that Immich Server is running on

Ubuntu 22.04.4 LTS

Version of Immich Server

v1.106.4

Version of Immich Mobile App

v1.106.1.build.142

Platform with the issue

• [X] Server
• [ ] Web
• [ ] Mobile

Your docker-compose.yml content

### docker-compose.yml
services:
  immich-server:
    container_name: immich-server
    image: ghcr.io/immich-app/immich-server:release
    restart: unless-stopped
    user: "$PUID:$PGID"
    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
      file: hwaccel.transcoding.yml
      service: rkmpp # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    ports:
      - 2283:3001
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ./reverse-geocoding:/usr/src/app/.reverse-geocoding-dump
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    labels:
      - homepage.group=Media
      - homepage.name=Immich
      - homepage.icon=immich
      - homepage.href=my_url
      - com.centurylinklabs.watchtower.monitor-only=true

  immich-machine-learning:
    container_name: immich-machine-learning
    image: ghcr.io/immich-app/immich-machine-learning:release-armnn
    restart: unless-stopped
    user: "$PUID:$PGID"
    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
      file: hwaccel.ml.yml
      service: armnn # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    # ports:
    #   - 3003:3003
    volumes:
      - ./model-cache:/cache
    env_file:
      - .env
    labels:
      - com.centurylinklabs.watchtower.monitor-only=true

networks:
  default:
    name: infra_default
    external: true

### hwaccel.ml.yml
# Configurations for hardware-accelerated machine learning

# If using Unraid or another platform that doesn't allow multiple Compose files,
# you can inline the config for a backend by copying its contents
# into the immich-machine-learning service in the docker-compose.yml file.

# See https://immich.app/docs/features/ml-hardware-acceleration for info on usage.

services:
  armnn:
    devices:
      - /dev/mali0:/dev/mali0
      # - /dev/rga:/dev/rga
      # - /dev/dri:/dev/dri
      # - /dev/dma_heap:/dev/dma_heap
      # - /dev/mpp_service:/dev/mpp_service
    volumes:
      - /lib/firmware/mali_csffw.bin:/lib/firmware/mali_csffw.bin:ro # Mali firmware for your chipset (not always required depending on the driver)
      # - /usr/lib/libmali-valhall-g610-g6p0-x11-wayland-gbm.so:/usr/lib/libmali.so:ro # Mali driver for your chipset (always required)
      - /usr/lib/aarch64-linux-gnu/libmali.so:/usr/lib/libmali.so:ro # Mali driver for your chipset (always required)
      # - /usr/lib/aarch64-linux-gnu/libdrm.so.2:/usr/lib/libdrm.so.2:ro
      - /usr/lib/aarch64-linux-gnu/libwayland-client.so.0:/usr/lib/libwayland-client.so.0:ro
      - /usr/lib/aarch64-linux-gnu/libwayland-server.so.0:/usr/lib/libwayland-server.so.0:ro
      - /usr/lib/aarch64-linux-gnu/libX11.so.6:/usr/lib/libX11.so.6:ro
      - /usr/lib/aarch64-linux-gnu/libX11-xcb.so.1:/usr/lib/libX11-xcb.so.1:ro
      - /usr/lib/aarch64-linux-gnu/libxcb.so.1:/usr/lib/libxcb.so.1:ro
      - /usr/lib/aarch64-linux-gnu/libxcb-dri2.so.0:/usr/lib/libxcb-dri2.so.0:ro
      - /usr/lib/aarch64-linux-gnu/libXau.so.6:/usr/lib/libXau.so.6:ro
      - /usr/lib/aarch64-linux-gnu/libXdmcp.so.6:/usr/lib/libXdmcp.so.6:ro
      # - /usr/lib/aarch64-linux-gnu/libbsd.so.0:/usr/lib/libbsd.so.0:ro

  cpu:

  cuda:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu
                - compute
                - video

  openvino:
    device_cgroup_rules:
      - "c 189:* rmw"
    devices:
      - /dev/dri:/dev/dri
    volumes:
      - /dev/bus/usb:/dev/bus/usb

  openvino-wsl:
    devices:
      - /dev/dri:/dev/dri
      - /dev/dxg:/dev/dxg
    volumes:
      - /dev/bus/usb:/dev/bus/usb
      - /usr/lib/wsl:/usr/lib/wsl

### hwaccel.transcoding.yml
# Configurations for hardware-accelerated transcoding

# If using Unraid or another platform that doesn't allow multiple Compose files,
# you can inline the config for a backend by copying its contents
# into the immich-microservices service in the docker-compose.yml file.

# See https://immich.app/docs/features/hardware-transcoding for more info on using hardware transcoding.

services:
  cpu:

  nvenc:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu
                - compute
                - video

  quicksync:
    devices:
      - /dev/dri:/dev/dri

  rkmpp:
    security_opt: # enables full access to /sys and /proc, still far better than privileged: true
      - systempaths=unconfined
      - apparmor=unconfined
    group_add:
      - video
    devices:
      - /dev/rga:/dev/rga
      - /dev/dri:/dev/dri
      - /dev/dma_heap:/dev/dma_heap
      - /dev/mpp_service:/dev/mpp_service
    volumes:
      - /usr/bin/ffmpeg:/usr/bin/ffmpeg_mpp
      - /lib/aarch64-linux-gnu:/lib/ffmpeg-mpp
      - /lib/aarch64-linux-gnu/libblas.so.3:/lib/ffmpeg-mpp/libblas.so.3:ro # symlink is resolved by mounting
      - /lib/aarch64-linux-gnu/liblapack.so.3:/lib/ffmpeg-mpp/liblapack.so.3:ro # symlink is resolved by mounting
      - /lib/aarch64-linux-gnu/pulseaudio/libpulsecommon-15.99.so:/lib/ffmpeg-mpp/libpulsecommon-15.99.so:ro

  vaapi:
    devices:
      - /dev/dri:/dev/dri

  vaapi-wsl: # use this for VAAPI if you're running Immich in WSL2
    devices:
      - /dev/dri:/dev/dri
    volumes:
      - /usr/lib/wsl:/usr/lib/wsl
    environment:
      - LD_LIBRARY_PATH=/usr/lib/wsl/lib
      - LIBVA_DRIVER_NAME=d3d12

Your .env content

PUID=1000
PGID=1000
TZ=Europe/Warsaw

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

###################################################################################
# Ports
###################################################################################
# PORT=3000  # immich-web
# SERVER_PORT=3001  # immich-server
# MICROSERVICES_PORT=3002  # immich-microservices
# MACHINE_LEARNING_PORT=3003  # immich-machine-learning

###################################################################################
# Database
###################################################################################

DB_HOSTNAME=postgres
DB_DATABASE_NAME=immich
DB_USERNAME=immich
DB_PASSWORD=##########

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=redis
REDIS_DBINDEX=2

# Optional Redis settings:

# Note: these parameters are not automatically passed to the Redis Container
# to do so, please edit the docker-compose.yml file as well. Redis is not configured
# via environment variables, only redis.conf or the command line

# REDIS_PORT=6379
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION=/home/ykrasik/photos

###################################################################################
# Log message level - [simple|verbose]
###################################################################################

# LOG_LEVEL=log
LOG_LEVEL=debug

###################################################################################
# Typesense
###################################################################################
# TYPESENSE_ENABLED=false
# TYPESENSE_API_KEY=v37%ikAn^M&jaa
# TYPESENSE_HOST: typesense
# TYPESENSE_PORT: 8108
# TYPESENSE_PROTOCOL: http

###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

# DISABLE_REVERSE_GEOCODING=false
REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>demo@demo.de</i><br>Password: <i>demo</i>"
####################################################################################

# PUBLIC_LOGIN_PAGE_MESSAGE="Photos"

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

# IMMICH_WEB_URL=http://immich-web:3004
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
# PUBLIC_IMMICH_SERVER_URL=http://immich-server:3001

####################################################################################
# Alternative API's External Address - Optional
#
# This is an advanced feature used to control the public server endpoint returned to clients during Well-known discovery.
# You should only use this if you want mobile apps to access the immich API over a custom URL. Do not include trailing slash.
# NOTE: At this time, the web app will not be affected by this setting and will continue to use the relative path: /api
# Examples: http://localhost:3001, http://immich-api.example.com, etc
####################################################################################

#IMMICH_API_URL_EXTERNAL=http://localhost:3001

###################################################################################
# Immich Version - Optional
#
# This allows all immich docker images to be pinned to a specific version. By default,
# the version is "release" but could be a specific version, like "v1.59.0".
###################################################################################

#IMMICH_VERSION=

Reproduction steps

1.
2.
3.
...

Relevant log output

# immich-server:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:ApiService] Feature Flags: {
  "smartSearch": true,
  "facialRecognition": true,
  "duplicateDetection": true,
  "map": true,
  "reverseGeocoding": true,
  "sidecar": true,
  "search": true,
  "trash": true,
  "oauth": true,
  "oauthAutoLaunch": true,
  "passwordLogin": true,
  "configFile": false,
  "email": false
}
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:NestFactory] Starting Nest application...
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:InstanceLoader] BullModule dependencies initialized
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:InstanceLoader] OpenTelemetryModule dependencies initialized
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:InstanceLoader] TypeOrmModule dependencies initialized
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:InstanceLoader] OpenTelemetryCoreModule dependencies initialized
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:InstanceLoader] ConfigHostModule dependencies initialized
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:InstanceLoader] DiscoveryModule dependencies initialized
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:InstanceLoader] ClsModule dependencies initialized
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:InstanceLoader] ConfigModule dependencies initialized
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:InstanceLoader] EventEmitterModule dependencies initialized
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:InstanceLoader] ScheduleModule dependencies initialized
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:InstanceLoader] BullModule dependencies initialized
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:InstanceLoader] BullModule dependencies initialized
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:InstanceLoader] TypeOrmCoreModule dependencies initialized
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:InstanceLoader] TypeOrmModule dependencies initialized
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:InstanceLoader] ApiModule dependencies initialized
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] APIKeyController {/api/api-keys}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/api-keys, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/api-keys, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/api-keys/:id, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/api-keys/:id, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/api-keys/:id, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] ActivityController {/api/activities}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/activities, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/activities/statistics, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/activities, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/activities/:id, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] AlbumController {/api/albums}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/albums/count, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/albums, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/albums, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/albums/:id, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/albums/:id, PATCH} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/albums/:id, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/albums/:id/assets, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/albums/:id/assets, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/albums/:id/users, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/albums/:id/user/:userId, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/albums/:id/user/:userId, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] AppController {/api}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/.well-known/immich, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/custom.css, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] AssetController {/api/assets}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets/memory-lane, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets/random, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets/device/:deviceId, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets/statistics, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets/jobs, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets/stack/parent, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets/:id, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets/:id, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] AssetMediaController {/api/assets}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets/:id/original, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets/:id/original, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets/:id/thumbnail, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets/:id/video/playback, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets/exist, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/assets/bulk-upload-check, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] AuditController {/api/audit}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/audit/deletes, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] AuthController {/api/auth}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/auth/login, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/auth/admin-sign-up, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/auth/validateToken, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/auth/change-password, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/auth/logout, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] DownloadController {/api/download}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/download/info, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/download/archive, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] DuplicateController {/api/duplicates}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/duplicates, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] FaceController {/api/faces}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/faces, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/faces/:id, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] JobController {/api/jobs}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/jobs, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/jobs/:id, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] LibraryController {/api/libraries}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/libraries, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/libraries, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/libraries/:id, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/libraries/:id, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/libraries/:id/validate, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/libraries/:id, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/libraries/:id/statistics, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/libraries/:id/scan, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/libraries/:id/removeOffline, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] MapController {/api/map}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/map/markers, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/map/style.json, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] MemoryController {/api/memories}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/memories, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/memories, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/memories/:id, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/memories/:id, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/memories/:id, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/memories/:id/assets, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/memories/:id/assets, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] NotificationController {/api/notifications}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/notifications/test-email, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] OAuthController {/api/oauth}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/oauth/mobile-redirect, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/oauth/authorize, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/oauth/callback, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/oauth/link, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/oauth/unlink, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] PartnerController {/api/partners}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/partners, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/partners/:id, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/partners/:id, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/partners/:id, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] PersonController {/api/people}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/people, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/people, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/people, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/people/:id, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/people/:id, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/people/:id/statistics, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/people/:id/thumbnail, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/people/:id/assets, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/people/:id/reassign, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/people/:id/merge, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] ReportController {/api/reports}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/reports, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/reports/checksum, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/reports/fix, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] SearchController {/api/search}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/search/metadata, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/search/smart, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/search/explore, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/search/person, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/search/places, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/search/cities, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/search/suggestions, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] ServerInfoController {/api/server-info}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/server-info/storage, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/server-info/ping, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/server-info/version, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/server-info/features, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/server-info/theme, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/server-info/config, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/server-info/statistics, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/server-info/media-types, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] SessionController {/api/sessions}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/sessions, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/sessions, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/sessions/:id, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] SharedLinkController {/api/shared-links}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links/me, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links/:id, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links/:id, PATCH} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links/:id, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links/:id/assets, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/shared-links/:id/assets, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] SyncController {/api/sync}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/sync/full-sync, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/sync/delta-sync, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] SystemConfigController {/api/system-config}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/system-config, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/system-config/defaults, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/system-config, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/system-config/storage-template-options, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] SystemMetadataController {/api/system-metadata}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/system-metadata/admin-onboarding, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/system-metadata/admin-onboarding, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/system-metadata/reverse-geocoding-state, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] TagController {/api/tags}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/tags, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/tags, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/tags/:id, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/tags/:id, PATCH} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/tags/:id, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/tags/:id/assets, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/tags/:id/assets, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/tags/:id/assets, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] TimelineController {/api/timeline}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/timeline/buckets, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/timeline/bucket, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] TrashController {/api/trash}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/trash/empty, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/trash/restore, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/trash/restore/assets, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] UserAdminController {/api/admin/users}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id/preferences, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id/preferences, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/admin/users/:id/restore, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RoutesResolver] UserController {/api/users}:
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/users, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/users/me, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/users/me, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/users/me/preferences, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/users/me/preferences, PUT} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/users/:id, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/users/profile-image, POST} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/users/profile-image, DELETE} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:RouterExplorer] Mapped {/api/users/:id/profile-image, GET} route
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:NestApplication] Nest application successfully started
[Nest] 17  - 06/13/2024, 11:55:17 PM     LOG [Api:Bootstrap] Immich Server is listening on http://[::1]:3001 [v1.106.4] [PRODUCTION] 
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:MapRepository] Geodata import completed
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:MetadataService] Initialized local reverse geocoder
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:NestFactory] Starting Nest application...
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:InstanceLoader] BullModule dependencies initialized
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:InstanceLoader] OpenTelemetryModule dependencies initialized
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:InstanceLoader] TypeOrmModule dependencies initialized
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:InstanceLoader] OpenTelemetryCoreModule dependencies initialized
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:InstanceLoader] ConfigHostModule dependencies initialized
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:InstanceLoader] DiscoveryModule dependencies initialized
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:InstanceLoader] ClsModule dependencies initialized
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:InstanceLoader] ConfigModule dependencies initialized
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:InstanceLoader] EventEmitterModule dependencies initialized
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:InstanceLoader] BullModule dependencies initialized
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:InstanceLoader] BullModule dependencies initialized
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:InstanceLoader] TypeOrmCoreModule dependencies initialized
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:InstanceLoader] TypeOrmModule dependencies initialized
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:InstanceLoader] MicroservicesModule dependencies initialized
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:NestApplication] Nest application successfully started
[Nest] 7  - 06/13/2024, 11:56:01 PM     LOG [Microservices:Bootstrap] Immich Microservices is running [v1.106.4] [PRODUCTION] 
[Nest] 17  - 06/13/2024, 11:59:20 PM    WARN [Api:DuplicateService~8sxm87nw] Unknown keys found: {
  "machineLearning": {
    "classification": {
      "enabled": true,
      "modelName": "google/vit-base-patch16-224"
    }
  }
}
[Nest] 17  - 06/13/2024, 11:59:21 PM    WARN [Api:DuplicateService~7hq96zvj] Unknown keys found: {
  "machineLearning": {
    "classification": {
      "enabled": true,
      "modelName": "google/vit-base-patch16-224"
    }
  }
}
[Nest] 17  - 06/13/2024, 11:59:21 PM     LOG [Api:EventRepository] Websocket Connect:    4qy8kzKYzoDQTFzKAAAB
[Nest] 7  - 06/14/2024, 12:00:00 AM    WARN [Microservices:DuplicateService] Unknown keys found: {
  "machineLearning": {
    "classification": {
      "enabled": true,
      "modelName": "google/vit-base-patch16-224"
    }
  }
}
[Nest] 7  - 06/14/2024, 12:00:00 AM    WARN [Microservices:DuplicateService] Unknown keys found: {
  "machineLearning": {
    "classification": {
      "enabled": true,
      "modelName": "google/vit-base-patch16-224"
    }
  }
}
[Nest] 17  - 06/14/2024, 12:34:16 AM     LOG [Api:EventRepository] Websocket Disconnect: 4qy8kzKYzoDQTFzKAAAB
[Nest] 17  - 06/14/2024, 12:38:40 AM     LOG [Api:EventRepository] Websocket Connect:    4AuG7XTXRvubM70lAAAD
[Nest] 17  - 06/14/2024, 12:39:25 AM     LOG [Api:EventRepository] Websocket Disconnect: 4AuG7XTXRvubM70lAAAD
[Nest] 17  - 06/14/2024, 12:43:50 AM     LOG [Api:EventRepository] Websocket Connect:    iBIadejLbff1BiElAAAF
[Nest] 17  - 06/14/2024, 12:44:35 AM     LOG [Api:EventRepository] Websocket Disconnect: iBIadejLbff1BiElAAAF
[Nest] 17  - 06/14/2024, 12:49:00 AM     LOG [Api:EventRepository] Websocket Connect:    ayGI5U4V-jbNl6JHAAAH
[Nest] 17  - 06/14/2024, 12:49:45 AM     LOG [Api:EventRepository] Websocket Disconnect: ayGI5U4V-jbNl6JHAAAH
[Nest] 17  - 06/14/2024, 7:45:01 AM    WARN [Api:DuplicateService~pg75hoxp] Unknown keys found: {
  "machineLearning": {
    "classification": {
      "enabled": true,
      "modelName": "google/vit-base-patch16-224"
    }
  }
}
[Nest] 17  - 06/14/2024, 7:45:01 AM     LOG [Api:EventRepository] Websocket Connect:    ovV459YLs4rO-_eOAAAJ
[Nest] 17  - 06/14/2024, 7:45:01 AM    WARN [Api:DuplicateService~kovcv9a3] Unknown keys found: {
  "machineLearning": {
    "classification": {
      "enabled": true,
      "modelName": "google/vit-base-patch16-224"
    }
  }
}
[Nest] 17  - 06/14/2024, 7:47:36 AM     LOG [Api:EventRepository] Websocket Connect:    ragIIwWJnUO-dml6AAAL
[Nest] 17  - 06/14/2024, 7:47:36 AM    WARN [Api:DuplicateService~rmn8a1dn] Unknown keys found: {
  "machineLearning": {
    "classification": {
      "enabled": true,
      "modelName": "google/vit-base-patch16-224"
    }
  }
}
[Nest] 17  - 06/14/2024, 7:47:36 AM    WARN [Api:DuplicateService~ek30fv7e] Unknown keys found: {
  "machineLearning": {
    "classification": {
      "enabled": true,
      "modelName": "google/vit-base-patch16-224"
    }
  }
}
[Nest] 17  - 06/14/2024, 7:49:47 AM     LOG [Api:EventRepository] Websocket Disconnect: ragIIwWJnUO-dml6AAAL
[Nest] 17  - 06/14/2024, 7:54:53 AM    WARN [Api:DuplicateService~j3vz1ujs] Unknown keys found: {
  "machineLearning": {
    "classification": {
      "enabled": true,
      "modelName": "google/vit-base-patch16-224"
    }
  }
}
[Nest] 17  - 06/14/2024, 7:54:53 AM    WARN [Api:DuplicateService~1b7ecwxj] Unknown keys found: {
  "machineLearning": {
    "classification": {
      "enabled": true,
      "modelName": "google/vit-base-patch16-224"
    }
  }
}
[Nest] 17  - 06/14/2024, 7:55:53 AM    WARN [Api:DuplicateService~b4mce4tp] Unknown keys found: {
  "machineLearning": {
    "classification": {
      "enabled": true,
      "modelName": "google/vit-base-patch16-224"
    }
  }
}

# immich-machine-learning:
[06/13/24 23:55:09] INFO     Starting gunicorn 22.0.0                           
[06/13/24 23:55:09] INFO     Listening at: http://[::]:3003 (9)                 
[06/13/24 23:55:09] INFO     Using worker: app.config.CustomUvicornWorker       
[06/13/24 23:55:09] INFO     Booting worker with pid: 10                        
[06/13/24 23:55:15] WARNING  Matplotlib created a temporary cache directory at  
                             /tmp/matplotlib-3bfrjprp because the default path  
                             (/.config/matplotlib) is not a writable directory; 
                             it is highly recommended to set the MPLCONFIGDIR   
                             environment variable to a writable directory, in   
                             particular to speed up the import of Matplotlib and
                             to better support multiprocessing.                 
[06/13/24 23:55:16] INFO     Started server process [10]                        
[06/13/24 23:55:16] INFO     Waiting for application startup.                   
[06/13/24 23:55:16] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                     
[06/13/24 23:55:16] INFO     Initialized request thread pool with 8 threads.    
[06/13/24 23:55:16] INFO     Application startup complete.

Additional information

I'm on 1.106.4 now, but the issues started after upgrading to 1.106.2

[alextran1502]

There seems to be a bug about the settings key that doesn't get removed correctly, so you are seeing

[Nest] 17  - 06/14/2024, 7:55:53 AM    WARN [Api:DuplicateService~b4mce4tp] Unknown keys found: {
  "machineLearning": {
    "classification": {
      "enabled": true,
      "modelName": "google/vit-base-patch16-224"
    }
  }
}

Can you go to the machine-learning settings on the administration page, reset them to default, save them, and then reconfigure them to your current settings?

[ykrasik]

Hi,

Thanks for answering. That did remove the warnings from the logs, but the CPU spikes are still there.

[titro]

unfortunately i can confirm the behaviour for the server and ml docker container since 1.160.1 :-( No obvious errors in the logs

server:

ml:

[mertalev]

Could it be related to the health checks that were added recently? I'm thinking maybe the constant requests are creating a need to do more frequent garbage collection or something.

[mertalev]

To confirm, you can add this to both the server and ML services.

healthcheck:
  disable: true

[titro]

That did it :-) Thanks

[ykrasik]

I can also confirm: