search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
51.97k stars 2.75k forks source link

[BUG] Server crashes after, opning `../admin/repair` #6767

Closed Torstein-Eide closed 6 months ago

Torstein-Eide commented 9 months ago

The bug

The server crashes after admins try to open ../admin/repair.

Docker log:

<--- Last few GCs --->
[7:0x5c862320000]    16399 ms: Mark-Compact 250.8 (258.6) -> 249.9 (258.9) MB, 104.03 / 0.00 ms  (average mu = 0.308, current mu = 0.182) allocation failure; scavenge might not succeed
[7:0x5c862320000]    16589 ms: Mark-Compact 251.0 (258.9) -> 250.1 (259.1) MB, 184.65 / 0.00 ms  (average mu = 0.156, current mu = 0.028) allocation failure; scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----
 1: 0xca5430 node::Abort() [immich_server]
 2: 0xb7807d  [immich_server]
 3: 0xeca0b0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_server]
 4: 0xeca397 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_server]
 5: 0x10dc0e5  [immich_server]
 6: 0x10f3f68 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_server]
 7: 0x10ca081 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_server]
 8: 0x10cb215 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_server]
 9: 0x10a8866 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_server]
10: 0x15035f6 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_server]
11: 0x193cef6  [immich_server]
[Nest] 7  - 01/30/2024, 3:06:06 PM     LOG [SystemConfigService] LogLevel=log (set via system config)
[Nest] 7  - 01/30/2024, 3:06:07 PM     LOG [AppService] Feature Flags: {
  "clipEncode": true,
  "facialRecognition": true,
  "map": true,
  "reverseGeocoding": true,
  "sidecar": true,
  "search": true,
  "trash": true,
  "oauth": false,
  "oauthAutoLaunch": false,
  "passwordLogin": true,
  "configFile": false
}

The OS that Immich Server is running on

Docker, Ubuntu 20.04

Version of Immich Server

1.9.3

Version of Immich Mobile App

1.9.3

Platform with the issue

Your docker-compose.yml content

version: "3.8"

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - $USER1_path:/user1:ro
      - $USER2_path:/user2:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    deploy:
      resources:
        reservations:
          memory: 2G

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
      file: hwaccel.yml
      service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - $USER1_path:/user1:ro
      - $USER2_path:/user2:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always
    deploy:
      resources:
        reservations:
          memory: 2G

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    deploy:
      resources:
        reservations:
          cpus: '0.001'
          memory: 50M

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:b6124ab2e45cc332e16398022a411d7e37181f21ff7874835e0180f56a09e82a
    restart: always
    deploy:
      resources:
        reservations:
          cpus: '0.001'
          memory: 50M

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11@sha256:0335a1a22f8c5dd1b697f14f079934f5152eaaa216c09b61e293be285491f8ee
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always
    deploy:
      resources:
        reservations:
          cpus: '0.001'
          memory: 50M

volumes:
  pgdata:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/volum1/@docker/immich/upload_files

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
DB_PASSWORD="Some random password"

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1. open ../admin/repair
2. get error popup `Request failed with status code 503 - 503 - 
undefined`
3. see server is restarting.
...

Additional information

No response

aviv926 commented 9 months ago

Have you tried allocating more RAM? It seems that the system is unable to render the page as required because there is not enough RAM

JavaScript heap out of memory

Torstein-Eide commented 9 months ago

It was running unlimited.

currenly it looks like this, with 100MB in use:

$docker compose stats
CONTAINER ID   NAME                      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O           BLOCK I/O   PIDS
af665bd080b4   immich_microservices      2.10%     250.7MiB / 2GiB     12.24%    540MB / 2.57GB    0B / 0B     24
ecfa9c5d2b9f   immich_machine_learning   50.54%    200.8MiB / 2GiB     9.80%     8.43GB / 138MB    0B / 0B     7
da70fd4cb0b6   immich_server             0.17%     117.7MiB / 2GiB     5.75%     54.9MB / 240MB    0B / 0B     12
9725ad84f7cf   immich_postgres           0.01%     424MiB / 2GiB       20.71%    3.15GB / 2.05GB   0B / 0B     20
75f96f41e1ad   immich_redis              0.28%     45MiB / 2GiB        2.20%     584MB / 188MB     0B / 0B     5
aviv926 commented 9 months ago

It was running unlimited.

currently it looks like this, with 100MB in use:

$docker compose stats
CONTAINER ID   NAME                      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O           BLOCK I/O   PIDS
af665bd080b4   immich_microservices      2.10%     250.7MiB / 2GiB     12.24%    540MB / 2.57GB    0B / 0B     24
ecfa9c5d2b9f   immich_machine_learning   50.54%    200.8MiB / 2GiB     9.80%     8.43GB / 138MB    0B / 0B     7
da70fd4cb0b6   immich_server             0.17%     117.7MiB / 2GiB     5.75%     54.9MB / 240MB    0B / 0B     12
9725ad84f7cf   immich_postgres           0.01%     424MiB / 2GiB       20.71%    3.15GB / 2.05GB   0B / 0B     20
75f96f41e1ad   immich_redis              0.28%     45MiB / 2GiB        2.20%     584MB / 188MB     0B / 0B     5

Try to check the compose stats when you try enter to the repair page

diyoyo commented 9 months ago

I have a Javascript Heap OoM as well, but for a different problem: immich won't update the new Library files as the file count is already very big.

Currently, I found this Sof issue: https://stackoverflow.com/questions/38558989/node-js-heap-out-of-memory where they suggest the use of the following: NODE_OPTIONS="--max-old-space-size=4096"

I'm not sure I can just paste it right away in the docker-compose file, but I'll try. If you have better suggestion to use that env variable, let me know. Of course, in your case, having 2G as your upper limit may be the problem...

Also, I'm wondering if we could play with 'shm_size'. It was in my docker-compose.yml file when I had tried photoview, but I'm not sure whether it's related to the tech stack choices or if this applies to any docker container that needs more memory...

alextran1502 commented 6 months ago

4510