search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
31.9k stars 1.51k forks source link

iOS app synchronisation issues #8023

Open pauleseifert opened 1 month ago

pauleseifert commented 1 month ago

The bug

I have experienced synchronisation issues between the mobile app on IOS and the server for quite a while. Symptoms are:

  1. Only parts of the database are synced and shown in the app. Sync includes files on the phone and external library files if they are in albums. Rest of the immich database's files and external library files are missing.
  2. Files retain the "crossed out cloud" symbol even when files are synced to the server.
  3. Frequent mobile database rebuilds when opening the app.
  4. I have the same spinning wheel as in #7498 but without hash calculation errors.
  5. Memories don't show; mobile app has an error (see attached log file)
  6. App error "RangeError (index): Invalid value: Valid value range is empty: 0 " (see attached log file)
  7. Occasionally, I get a Server error with a JS out of memory (see below), but that might be unrelated. The host system should have plenty of memory.

The file upload works as expected, and so does the web app. Reinstalling the mobile app doesn't help, nor does restarting the stack. Same on another (i)Phone. The domain is protected by Cloudflare, but local deployment sees the same behaviour.

Happy to help with more logs or for suggestions to fix if my setup is the problem.

The OS that Immich Server is running on

Docker on QTS 5.1.5

Version of Immich Server

1.98.2

Version of Immich Mobile App

1.98.2 build 144

Platform with the issue

Your docker-compose.yml content

version: '3.9'
name: immich
networks:
  cloudflare_backend:
    external: true
services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "immich"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - xxx:/usr/src/app/upload/library
      - xxx:/mnt/media/xxx:ro
      - /etc/localtime:/etc/localtime:ro
    dns:
      - xxx
    networks:
      backend:
      cloudflare_backend:
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
    restart: always
    ports:
      - 2283:3001

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "microservices"]
    networks:
      backend:
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - xxx:/usr/src/app/upload/library
      - xxx:/mnt/media/xxx:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
    restart: always
    devices:
      - /dev/dri:/dev/dri 
      - /dev/dri/card0:/dev/dri/card0

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    restart: always
    networks:
      backend:
    env_file:
      - stack.env

  redis:
    container_name: immich_redis
    image: redis:alpine
    restart: always
    networks:
      backend:
    env_file:
      - stack.env

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    networks:
      backend:
    env_file:
      - stack.env
volumes:
  pgdata:
  model-cache:

Your .env content

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=xxx
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
UPLOAD_LOCATION=xxx
LOG_LEVEL=debug
DISABLE_REVERSE_GEOCODING=false
REVERSE_GEOCODING_PRECISION=3
PUBLIC_LOGIN_PAGE_MESSAGE="xxx"
IMMICH_VERSION="1.98.2"

Reproduction steps

opening the mobile app.

Additional information

App log:

Immich_log_2024-03-17T14:45:27.092139.csv

Error message from the server container:

[Nest] 7  - 03/17/2024, 2:09:19 PM     LOG [CommunicationRepository] Websocket Disconnect: gL1VexRjzCE0wcTOAAAJ

<--- Last few GCs --->

 0[7:0x2ab08320000]   641447 ms: Mark-Compact (reduce) 2048.0 (2083.9) -> 2047.2 (2084.2) MB, 1863.39 / 0.00 ms  (+ 81.2 ms in 15 steps since start of marking, biggest step 14.9 ms, walltime since start of marking 1982 ms) (average mu = 0.319, current mu = [7:0x2ab08320000]   644122 ms: Mark-Compact (reduce) 2048.2 (2084.2) -> 2047.4 (2084.4) MB, 1830.37 / 0.00 ms  (+ 48.8 ms in 15 steps since start of marking, biggest step 11.8 ms, walltime since start of marking 1918 ms) (average mu = 0.308, current mu = 

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

----- Native stack trace -----

 1: 0xca5580 node::Abort() [immich_server]

 2: 0xb781f9  [immich_server]

 3: 0xeca4d0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_server]

 4: 0xeca7b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_server]

 5: 0x10dc505  [immich_server]

 6: 0x10f4388 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_server]

 7: 0x10ca4a1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_server]

 8: 0x10cb635 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_server]

 9: 0x10a8c86 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_server]

10: 0x1503a16 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_server]

11: 0x7ff941e19ef6 

[Nest] 7  - 03/17/2024, 2:10:40 PM     LOG [CommunicationRepository] Initialized websocket server

[Nest] 7  - 03/17/2024, 2:10:40 PM     LOG [SystemConfigService] LogLevel=debug (set via LOG_LEVEL)

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [AppService] Feature Flags: {

  "smartSearch": true,

  "facialRecognition": true,

  "map": true,

  "reverseGeocoding": true,

  "sidecar": true,

  "search": true,

  "trash": true,

  "oauth": false,

  "oauthAutoLaunch": false,

  "passwordLogin": true,

  "configFile": false

}

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [NestFactory] Starting Nest application...

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [InstanceLoader] BullModule dependencies initialized

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [InstanceLoader] ConfigModule dependencies initialized

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [InstanceLoader] BullModule dependencies initialized

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [InstanceLoader] BullModule dependencies initialized

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [InstanceLoader] InfraModule dependencies initialized

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [InstanceLoader] DomainModule dependencies initialized

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [InstanceLoader] AppModule dependencies initialized

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] ActivityController {/api/activity}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/activity, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/activity/statistics, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/activity, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/activity/:id, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] AssetsController {/api/assets}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/assets, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] AssetController {/api/asset}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/upload, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/file/:id, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/thumbnail/:id, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/curated-objects, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/curated-locations, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/search-terms, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/exist, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/bulk-upload-check, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] AssetController {/api/asset}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/map-marker, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/memory-lane, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/random, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/device/:deviceId, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/statistics, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/time-buckets, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/time-bucket, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/jobs, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/stack/parent, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/:id, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/asset/:id, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] AppController {/api}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/.well-known/immich, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/custom.css, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] AlbumController {/api/album}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/album/count, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/album, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/album, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/album/:id, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/album/:id, PATCH} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/album/:id, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/album/:id/assets, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/album/:id/assets, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/album/:id/users, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/album/:id/user/:userId, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] APIKeyController {/api/api-key}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/api-key, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/api-key, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/api-key/:id, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/api-key/:id, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/api-key/:id, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] AuditController {/api/audit}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/audit/deletes, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/audit/file-report, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/audit/file-report/checksum, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/audit/file-report/fix, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] AuthController {/api/auth}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/auth/login, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/auth/admin-sign-up, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/auth/devices, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/auth/devices, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/auth/devices/:id, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/auth/validateToken, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/auth/change-password, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/auth/logout, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] DownloadController {/api/download}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/download/info, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/download/archive, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/download/asset/:id, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] FaceController {/api/face}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/face, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/face/:id, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] JobController {/api/jobs}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/jobs, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/jobs/:id, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] LibraryController {/api/library}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/library, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/library, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/library/:id, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/library/:id, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/library/:id/validate, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/library/:id, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/library/:id/statistics, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/library/:id/scan, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/library/:id/removeOffline, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] OAuthController {/api/oauth}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/oauth/mobile-redirect, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/oauth/authorize, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/oauth/callback, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/oauth/link, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/oauth/unlink, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] PartnerController {/api/partner}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/partner, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/partner/:id, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/partner/:id, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/partner/:id, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] SearchController {/api/search}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/search, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/search/metadata, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/search/smart, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/search/explore, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/search/person, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/search/places, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/search/suggestions, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] ServerInfoController {/api/server-info}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/server-info, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/server-info/ping, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/server-info/version, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/server-info/features, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/server-info/theme, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/server-info/config, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/server-info/statistics, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/server-info/media-types, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/server-info/admin-onboarding, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] SharedLinkController {/api/shared-link}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/shared-link, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/shared-link/me, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/shared-link, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id, PATCH} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id/assets, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id/assets, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] SystemConfigController {/api/system-config}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/system-config, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/system-config/defaults, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/system-config, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/system-config/storage-template-options, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/system-config/map/style.json, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] TagController {/api/tag}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/tag, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/tag, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/tag/:id, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/tag/:id, PATCH} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/tag/:id, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/tag/:id/assets, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/tag/:id/assets, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/tag/:id/assets, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] TrashController {/api/trash}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/trash/empty, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/trash/restore, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/trash/restore/assets, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] UserController {/api/user}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/user, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/user/info/:id, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/user/me, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/user, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/user/profile-image, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/user/:id, DELETE} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/user/:id/restore, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/user, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/user/profile-image, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/user/profile-image/:id, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RoutesResolver] PersonController {/api/person}:

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/person, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/person, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/person, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/person/:id, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/person/:id, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/person/:id/statistics, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/person/:id/thumbnail, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/person/:id/assets, GET} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/person/:id/reassign, PUT} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [RouterExplorer] Mapped {/api/person/:id/merge, POST} route

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [NestApplication] Nest application successfully started

[Nest] 7  - 03/17/2024, 2:10:41 PM     LOG [ImmichServer] Immich Server is listening on http://[::1]:3001 [v1.98.2] [PRODUCTION] 

[Nest] 7  - 03/17/2024, 2:10:44 PM     LOG [CommunicationRepository] Websocket Connect:    v5Y0jI7JT2_F29aFAAAB
alextran1502 commented 1 month ago

We have fixed some of the issues on the latest update, please try to log out and log back in, it should clear the local database and resync with the server

Ref: #7752

pauleseifert commented 1 month ago

I tried that a couple of times. #7752 should be included in the latest 1.98.2 but the problem persist, at least in my case.

alextran1502 commented 1 month ago

@pauleseifert when you log out and log back in do you access over local IP or CF? Can you try over local IP if you haven't?

pauleseifert commented 1 month ago

@alextran1502 I tried both before with the same issues. Tried it yet again on the newest version of 1.99.0 without any success. Any ideas on what to try?

alextran1502 commented 1 month ago

The symptom looks like the mobile app cannot get all the asset information from the server, leads to missing info for other tasks.

What is your phone version and how many assets do you have on the server? How much Ram do you have on the server?

pauleseifert commented 1 month ago

App version is also 1.99.0, same as the server version. iPhone 13@17.4. I have about 52k assets and the server has 16gb of RAM.

pauleseifert commented 1 month ago

@alextran1502 I did a couple of reinstalls and I think the app itself is fine. Database synchronisation works, pictures from the server are slowly fetched (I put some docker cpu limitations because QTS seems to be sometimes unstable under full load) until at some point I run into socket timeouts and the JS out of memory issue. Below are the latest logs. Memory of the host is not depleted when it occurs (with at least 8GB left).

Do you have any ideas why I get this out of memory problem? If not, I wouldn't fully cross out that this is caused by QTS. I migrate to proper server hardware in the next couple of weeks so maybe this solves the problem. 

[Nest] 7  - 03/28/2024, 9:20:55 AM   ERROR [ExceptionsHandler] Connection terminated due to connection timeout

Error: Connection terminated due to connection timeout

    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)

    at Object.onceWrapper (node:events:632:28)

    at Connection.emit (node:events:518:28)

    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)

    at Socket.emit (node:events:518:28)

    at TCP.<anonymous> (node:net:337:12)

<--- Last few GCs --->

[7:0x40f8e320000]   445614 ms: Mark-Compact (reduce) 2046.9 (2083.6) -> 2046.6 (2084.4) MB, 6896.27 / 0.01 ms  (+ 83.5 ms in 13 steps since start of marking, biggest step 61.5 ms, walltime since start of marking 7018 ms) (average mu = 0.203, current mu = [7:0x40f8e320000]   456268 ms: Mark-Compact (reduce) 2047.7 (2084.4) -> 2046.9 (2084.6) MB, 7758.25 / 0.00 ms  (+ 63.7 ms in 15 steps since start of marking, biggest step 20.0 ms, walltime since start of marking 7979 ms) (average mu = 0.235, current mu = 

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

----- Native stack trace -----

 1: 0xca5580 node::Abort() [immich_server]

 2: 0xb781f9  [immich_server]

 3: 0xeca4d0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_server]

 4: 0xeca7b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_server]

 5: 0x10dc505  [immich_server]

 6: 0x10f4388 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_server]

 7: 0x10ca4a1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_server]

 8: 0x10cb635 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_server]

 9: 0x10a8c86 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [immich_server]

10: 0x1503a16 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [immich_server]

11: 0x7f8f98e59ef6
Dankni95 commented 4 days ago

I am also experiencing this on android, i posted this on discord:

Im getting this error that crashes immich_server when building the timeline.

04/23/2024
11:06:53 PM
<--- Last few GCs --->
04/23/2024
11:06:53 PM
04/23/2024
11:06:53 PM
[6:0x5d52e330000]   917559 ms: Scavenge (reduce) 2044.8 (2081.5) -> 2044.1 (2081.8) MB, 7.28 / 0.00 ms  (average mu = 0.338, current mu = 0.325) allocation failure; 
04/23/2024
11:06:53 PM
[6:0x5d52e330000]   917618 ms: Scavenge (reduce) 2044.9 (2081.8) -> 2044.2 (2081.8) MB, 5.81 / 0.00 ms  (average mu = 0.338, current mu = 0.325) allocation failure; 
04/23/2024
11:06:53 PM
[6:0x5d52e330000]   917681 ms: Scavenge (reduce) 2045.0 (2081.8) -> 2044.4 (2082.0) MB, 4.54 / 0.00 ms  (average mu = 0.338, current mu = 0.325) allocation failure; 
04/23/2024
11:06:53 PM
04/23/2024
11:06:53 PM
04/23/2024
11:06:53 PM
<--- JS stacktrace --->
04/23/2024
11:06:53 PM
04/23/2024
11:06:53 PM
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
04/23/2024
11:06:53 PM
----- Native stack trace -----
04/23/2024
11:06:53 PM
04/23/2024
11:06:53 PM
 1: 0xb84bd6 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [immich_server]
04/23/2024
11:06:53 PM
 2: 0xefeb90 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_server]
04/23/2024
11:06:53 PM
 3: 0xefee77 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_server]
04/23/2024
11:06:53 PM
 4: 0x1110885  [immich_server]
04/23/2024
11:06:53 PM
 5: 0x1110e14 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_server]
04/23/2024
11:06:53 PM
 6: 0x1127d04 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_server]
04/23/2024
11:06:53 PM
 7: 0x112851c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_server]
04/23/2024
11:06:53 PM
 8: 0x112a67a v8::internal::Heap::HandleGCRequest() [immich_server]
04/23/2024
11:06:53 PM
 9: 0x1095ce7 v8::internal::StackGuard::HandleInterrupts() [immich_server]
04/23/2024
11:06:53 PM
10: 0x1537542 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_server]
04/23/2024
11:06:53 PM
11: 0x7fc429e99ef6

Server crashes before reaching 2.3 gb of ram, when i added- NODE_OPTIONS="--max-old-space-size=8192" it now reaches 6gb (i guess my max) on my 8gb system.

With the env variable i got this error:

Nest] 7  - 04/22/2024, 5:51:27 PM   ERROR [ExceptionsHandler] Invalid string length
04/22/2024
07:51:27 PM
RangeError: Invalid string length
04/22/2024
07:51:27 PM
    at JSON.stringify (<anonymous>)
04/22/2024
07:51:27 PM
    at stringify (/usr/src/app/node_modules/express/lib/response.js:1159:12)
04/22/2024
07:51:27 PM
    at ServerResponse.json (/usr/src/app/node_modules/express/lib/response.js:272:14)
04/22/2024
07:51:27 PM
    at ExpressAdapter.reply (/usr/src/app/node_modules/@nestjs/platform-express/adapters/express-adapter.js:62:62)
04/22/2024
07:51:27 PM
    at RouterResponseController.apply (/usr/src/app/node_modules/@nestjs/core/router/router-response-controller.js:15:36)
04/22/2024
07:51:27 PM
    at /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:176:48
04/22/2024
07:51:27 PM
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
04/22/2024
07:51:27 PM
    at async /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:47:13
04/22/2024
07:51:27 PM
    at async /usr/src/app/node_modules/@nestjs/core/router/router-proxy.js:9:17

This is only for 18k photos for this user, total 30k. Is 6gb allocated not enough for 18k images?

This happens only on android, the timeline builds on web.

I'm running in docker v1.102.0, remote through cloudflare.

Help much appreciated.

pauleseifert commented 4 days ago

Interesting. I tested it without cloudflare and the problem persists. Do you use QTS as a host system as well? I haven't found a solution yet but are working on migrating to Debian based truenas scale and a lot more ram and hope that the problem magically disappears. However, I had plenty of space left on my machine before (16gb).

I had hoped for #8755 to fix the issue, but this didn't happen.

Dankni95 commented 4 days ago

The problem persist even when running locally. Im running it on ubuntu server with docker lastest version and android app latest , no NAS.

Hopefully more ram can help, but with my low asset count and high ram usage(6gb), something is broken. If devs need more logs or any help, im open to it.