1.113 Facial recognition not working

The bug

I was using the 1.09 version before upgrading in which I had not turned machine learning on. My machine learning server had an issue with openBLAS not being allowed multiple threads, so I used the "opt-security" priviliged setting for that in the docker compose.

I went to jobs and clicked run all. Looking at the logs file of the ML container I get the following output continiously

2024-09-02 11:11:00.881613426 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 683
2024-09-02 11:11:02.180080903 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:02.322959747 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {4,512} for output 683
2024-09-02 11:11:03.180117114 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:03.280537676 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:04.344415389 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:05.226606656 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 683
2024-09-02 11:11:05.799214362 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:07.552269746 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {5,512} for output 683
2024-09-02 11:11:07.792963316 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 683
2024-09-02 11:11:09.754034963 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {4,512} for output 683
2024-09-02 11:11:10.764368678 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {6,512} for output 683
2024-09-02 11:11:10.813287722 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 683
2024-09-02 11:11:11.940279203 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:11.978081636 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 683
2024-09-02 11:11:13.303886994 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:14.150040267 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {5,512} for output 683
2024-09-02 11:11:15.509054157 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:16.031306946 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:16.672024857 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 683
2024-09-02 11:11:17.850123084 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {5,512} for output 683
2024-09-02 11:11:18.116726508 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {5,512} for output 683
2024-09-02 11:11:19.743661668 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {6,512} for output 683
2024-09-02 11:11:19.826529388 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {5,512} for output 683
2024-09-02 11:11:20.974636550 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 683
2024-09-02 11:11:21.047272423 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 683
2024-09-02 11:11:21.847070020 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:25.230369980 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:25.251567163 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 683
2024-09-02 11:11:26.378359890 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 683
2024-09-02 11:11:26.404328824 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:28.123263289 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {6,512} for output 683
2024-09-02 11:11:28.317390753 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {5,512} for output 683
2024-09-02 11:11:29.521775237 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {4,512} for output 683
2024-09-02 11:11:29.701053056 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {4,512} for output 683
2024-09-02 11:11:30.530752974 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:30.899817085 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {4,512} for output 683
2024-09-02 11:11:31.367168670 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:31.831852909 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:32.165905916 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:32.731809280 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:33.091296833 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 683
2024-09-02 11:11:33.880740462 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 683
2024-09-02 11:11:36.364525111 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:36.545587389 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 683
2024-09-02 11:11:37.462788580 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 683
2024-09-02 11:11:38.570564694 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:38.641351629 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:39.647968790 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {2,512} for output 683
2024-09-02 11:11:39.920819785 [W:onnxruntime:, execution_frame.cc:870 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 683

I know enough about ML to know what the problem is, but not why it is happening since I assume everybody uses this default model. I didnt change any of the default settings regarding ML.

The OS that Immich Server is running on

Ubuntu 19.04.6 LTS

Version of Immich Server

v1.113

Version of Immich Mobile App

non applicable

Platform with the issue

[X] Server
[X] Web
[ ] Mobile

Your docker-compose.yml content

#
  2 # WARNING: Make sure to use the docker-compose.yml of the current release:
  3 #
  4 # https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
  5 #
  6 # The compose file on main may not be compatible with the latest release.
  7 #
  8
  9 name: immich
 10
 11 services:
 12   immich-server:
 13     container_name: immich_server
 14     privileged: true
 15     image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
 16     # extends:
 17     #   file: hwaccel.transcoding.yml
 18     #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
 19     volumes:
 20       - ${UPLOAD_LOCATION}:/usr/src/app/upload
 21       - /etc/localtime:/etc/localtime:ro
 22     env_file:
 23       - .env
 24     ports:
 25       - 8161:3001
 26     depends_on:
 27       - redis
 28       - database
 29     restart: always
 30
 31   immich-machine-learning:
 32     container_name: immich_machine_learning
 33     security_opt:
 34       - seccomp:unconfined
 35     # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
 36     # Example tag: ${IMMICH_VERSION:-release}-cuda
 37     image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
 38     # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
 39     #   file: hwaccel.ml.yml
 40     #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
 41     volumes:
 42       - model-cache:/cache
 43     env_file:
 44       - .env
 45     restart: always
 46
 47   redis:
 48     container_name: immich_redis
 49     image: docker.io/redis:6.2-alpine@sha256:328fe6a5822256d065debb36617a8169dbfbd77b797c525288e465f56c1d392b
 50     healthcheck:
 51       test: redis-cli ping || exit 1
 52     restart: always
 53
 54   database:
 55     container_name: immich_postgres
 56     image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
 57     security_opt:
 58       - seccomp:unconfined
 59     environment:
 60       POSTGRES_PASSWORD: ${DB_PASSWORD}
 61       POSTGRES_USER: ${DB_USERNAME}
 62       POSTGRES_DB: ${DB_DATABASE_NAME}
 63       POSTGRES_INITDB_ARGS: '--data-checksums'
 64     volumes:
 65       - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
 66     healthcheck:
 67       test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum    "; [ "$$Chksum" = '0' ] || exit 1
 68     command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
 69     restart: always
 70
 71 volumes:
 72   model-cache:


### Your .env content

```Shell
1 # You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables
 2
 3 # The location where your uploaded files are stored
 4 UPLOAD_LOCATION=./library
 5 # The location where your database files are stored
 6 DB_DATA_LOCATION=./postgres
 7
 8 # To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
 9 # TZ=Etc/UTC
10
11 # The Immich version to use. You can pin this to a specific version like "v1.71.0"
12 IMMICH_VERSION=release
13
14 # Connection secret for postgres. You should change it to a random password
15 DB_PASSWORD=**
16
17 # The values below this line do not need to be changed
18 ###################################################################################
19 DB_USERNAME=postgres
20 DB_DATABASE_NAME=immich



### Reproduction steps

1. Go to jobs
2. Turn on face detection
3. Check logs, 
4. Numbers keep decreasing in the jobs tab, but no results are being generated. 
...

### Relevant log output

_No response_

### Additional information

_No response_

immich-app / immich