Photo searches stop working after upgrading to v1.108.0

[plord12]

The bug

Simple photo searches for, say, "cat" now return no results.

The immich_machine_learning container reports :

[07/10/24 22:01:02] INFO     Loading textual model 'ViT-B-32__openai' to memory 
arm_release_ver of this libmali is 'g6p0-01eac0', rk_so_ver is '7'.
[07/10/24 22:01:02] INFO     Loading ANN model                                  
                             /cache/clip/ViT-B-32__openai/textual/model.armnn   
                             ...                                                
Warning: WARNING: Layer of type Constant is not supported on requested backend GpuAcc for input data type Float32 and output data type Signed64 (reason: Unsupported DataType), falling back to the next backend.
Warning: WARNING: Layer of type Constant is not supported on requested backend GpuAcc for input data type Float32 and output data type Signed64 (reason: Unsupported DataType), falling back to the next backend.
Warning: WARNING: Layer of type Cast is not supported on requested backend GpuAcc for input data type Boolean and output data type Signed64 (reason: in validate_arguments src/gpu/cl/kernels/ClCastKernel.cpp:59: ITensor data type S64 not supported by this kernel), falling back to the next backend.
Warning: ERROR: Layer of type Cast is not supported on any preferred backend [GpuAcc ]
Warning: WARNING: Layer of type ElementwiseBinary is not supported on requested backend GpuAcc for input data type Signed64 and output data type Signed64 (reason: in validate_arguments src/gpu/cl/kernels/ClMulKernel.cpp:63: ITensor data type S64 not supported by this kernel), falling back to the next backend.
Warning: ERROR: Layer of type ElementwiseBinary is not supported on any preferred backend [GpuAcc ]
Warning: WARNING: Layer of type ElementwiseBinary is not supported on requested backend GpuAcc for input data type Signed64 and output data type Signed64 (reason: in validate_arguments_with_arithmetic_rules src/gpu/cl/kernels/ClElementwiseKernel.cpp:160: ITensor data type S64 not supported by this kernel), falling back to the next backend.
Warning: ERROR: Layer of type ElementwiseBinary is not supported on any preferred backend [GpuAcc ]
Warning: WARNING: Layer of type Gather is not supported on requested backend GpuAcc for input data type Float32 and output data type Float32 (reason: in validate_arguments src/core/CL/kernels/CLGatherKernel.cpp:58: ITensor data type S64 not supported by this kernel), falling back to the next backend.
Warning: ERROR: Layer of type Gather is not supported on any preferred backend [GpuAcc ]
terminate called after throwing an instance of 'armnn::InvalidArgumentException'
  what():  Failed to assign a backend to each layer

The OS that Immich Server is running on

Debian

Version of Immich Server

v1.108.0

Version of Immich Mobile App

-

Platform with the issue

• [ ] Server
• [X] Web
• [ ] Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
      file: hwaccel.transcoding.yml
      service: rkmpp # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-armnn
    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
      file: hwaccel.ml.yml
      service: armnn # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:328fe6a5822256d065debb36617a8169dbfbd77b797c525288e465f56c1d392b
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=Etc/UTC

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

#IMMICH_LOG_LEVEL=debug

Reproduction steps

1. Upgrade to v1.108.0
2. Try a search

Relevant log output

No response

Additional information

No response

[alextran1502]

@mertalev is this the same issue that Fynn is facing?

[mertalev]

Yes, it's because some of the recently uploaded ARMNN models don't work. Before this, it would just use the ONNX model on CPU, but now that the ARMNN model exists and the service has libann, it tries to use this model unsuccessfully.

[plord12]

Thanks.

Are there some models that work with ARMNN ?

I've disabled hardware acceleration in docker-compose.yml and I'm back working again.

[mertalev]

The visual search models (used when processing smart search jobs) work along with the facial recognition models. Unfortunately, they're gimped by the fact that their counterparts (textual search and face detection) don't work.

I've been looking into adding full ARMNN model coverage without having a device myself, so it's a little trial and error. I'll remove the textual and detection models for now so you can still use the models that do work.

[mertalev]

You can delete the model cache volume and re-enable ARMNN. Smart search jobs will be accelerated and part of the face detection job will be as well.

[plord12]

Many thanks

[ThomasConrad]

When can we expect ARMNN facial detection? I would love this feature.

[jdicioccio]

I'm having a very similar issue but with a different model. This is with 1.117.0

[10/05/24 01:06:28] INFO     Downloading textual model
                             'ViT-B-16-SigLIP-384__webli'. This may take a
                             while.
Fetching 11 files: 100%|██████████| 11/11 [00:20<00:00,  1.83s/it]
[10/05/24 01:06:49] INFO     Loading textual model 'ViT-B-16-SigLIP-384__webli'
                             to memory
arm_release_ver: g13p0-01eac0, rk_so_ver: 10
[10/05/24 01:06:49] INFO     Loading ANN model
                             /cache/clip/ViT-B-16-SigLIP-384__webli/textual/mode
                             l.armnn ...
Warning: WARNING: Layer of type Cast is not supported on requested backend GpuAcc for input data type Signed32 and output data type Signed64 (reason: in validate_arguments src/gpu/cl/kernels/ClCastKernel.cpp:59: ITensor data type S64 not supported by this kernel), falling back to the next backend.
Warning: ERROR: Layer of type Cast is not supported on any preferred backend [GpuAcc ]
Warning: WARNING: Layer of type Gather is not supported on requested backend GpuAcc for input data type Float32 and output data type Float32 (reason: in validate_arguments src/core/CL/kernels/CLGatherKernel.cpp:58: ITensor data type S64 not supported by this kernel), falling back to the next backend.
Warning: ERROR: Layer of type Gather is not supported on any preferred backend [GpuAcc ]
Warning: WARNING: Layer of type Transpose is not supported on requested backend GpuAcc for input data type Float32 and output data type Float32 (reason: in validate_arguments src/gpu/cl/kernels/ClPermuteKernel.cpp:60: Permutation up to 4-D src tensor is supported), falling back to the next backend.