CLI Upload Error

[curi0usJack]

The bug

Uploading a large folder (170G) using immich cli via docker throws the following error. This happens repeatedly and predicatably. It hangs for several minutes before erroring.

The OS that Immich Server is running on

Arch 6.7.8

Version of Immich Server

1.98.1

Version of Immich Mobile App

n/a

Platform with the issue

• [ ] Server
• [X] Web
• [ ] Mobile

Your docker-compose.yml content

version: '3.8'

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    command: ['start.sh', 'microservices']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5
    restart: always

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD=redacted

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. Install immmich using portainer stack. Note that I copy the .env file to /data/compose in order to get the stack to start properly, otherwise it just threw an error that the .env file could not be found.
2. On my immich server running minimal Arch linux, I made a local copy of my photos to upload (170Gb worth).
3.Also on the immich server, I ran the docker command in the screenshot to perform a bulk upload. It seems to hang here then dies with the error shown. 
4. The server is still responsive during this time.
...

Additional information

Node version 20.11.0

[alextran1502]

This is a very old version of the CLI. Please use this guide instead https://immich.app/docs/features/command-line-interface

[curi0usJack]

I'm using the exact docker command that's listed in the article that you gave, which is what I used in the first place to try the bulk upload. Not sure why you closed this?

[jrasm91]

We only recently added back the docker version of the CLI, so I think he might have just assume it was the v1 version (oppose the the current), which it looks like it is.

This looks like a timeout issue, possibly within the fetch client itself.

[bo0tzz]

What does the server-info cli command return? Are you using a reverse proxy?

[jrasm91]

The error is from node:internal/timers

[curi0usJack]

Thank you for reopening the issue.

server-info:

Architecturally, immich is running in portainer that is on a HP Z2 Mini G3 workstation with plenty of spec. I'm using an NFS mount to my Synology which is where UPLOAD_LOCATION is (Apologies, the correct value is actually /mnt/immich). I have a backup of photos local to the host which is what I'm trying to upload in to immich.

immich cli -> local immich instance (same host) -> 1gb switch same LAN -> Synology NFS mount. No firewalling/revproxy in the way

I just tried rerunning the same command, it processed an additional 700M of items then froze.

Install node locally and try again with that command?

[alextran1502]

@curi0usJack Ah my bad, sorry about that 😅

[curi0usJack]

@alextran1502 all good. It's probably some dumb issue on my end anyways. Appreciate the help. :-)

[curi0usJack]

I installed node/immich-cli. It's definitely not a problem with docker. Seems to be hanging on something related to Canon Powershot formats.

Edit: Yikes. This seems to bork the docker container running immich also. Immich became unresponsive after encountering the error, so I tried to stop the stack in portainer. No go...

Edit2: Ok, tracking it down further, something is locking the mount to Synology. Can't umount. Unsure where the issue is, but that is likely explaining the inability to stop the stack.

[curi0usJack]

Problem seems to be related to one or more of the following.

1) The Immich application itself is having a hard time processing the files, eventually leading to a crash of some sort. 2) Some kind of docker problem under the hood. 3) An issue with NFS/Synology. When the timeout error occurs, docker.socket is still open, causing a total hang of the application. Synology is working fine and I can access the directories from other machines, but not from the immich host machine. The mount to /mnt/immich (which is where the library is), is hung entirely, probably causing the application to go unresponsive.

After an upload of a few hundred MB, the timeout occurs and the following processes go zombie/defunct:

For reference: this is a 4 core system with 32G ram and 1 TB ssd with gigabit wired connection to backend storage. Should be fine resource wise.

[curi0usJack]

hmm so this is the ffmpeg command that is hung. Is it just taking a while to process a large video, and this is causing an upstream timeout with immich?

/usr/bin/ffmpeg -ss 00:00:00 -sws_flags accurate_rnd+bitexact+full_chroma_int -i upload/upload/1be8d687-edee-4615-8a05-0abb2072d12d/83/04/83048d03-5f5a-4d9b-a968-eb382163aeab.AVI -y -frames:v 1 -v verbose -vf format=yuv420p upload/thumbs/1be8d687-edee-4615-8a05-0abb2072d12d/35/0f/350fec24-dda9-4ed5-965c-d5fc45d40543.webp

That AVI file is 7.8mb in size and plays fine. ffmpeg command works properly.

[curi0usJack]

Instead of running through the upload process, is there a way I can just point immich to a directory of photos and say "index these" instead of having to upload each one through the api?

[curi0usJack]

Going with an external library. Probably what I should have done to start with. -_-

[jrasm91]

Either way you may need to limit the usage of the microservices container or reevaluate your mount decisions if it is running into problems that impact the CLI upload process since that implies the whole server is being impacted.