search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
53.02k stars 2.81k forks source link

Postgres timeout #14069

Closed jonas-jasas closed 1 week ago

jonas-jasas commented 2 weeks ago

The bug

Immich server timeouts PG on slow drives. Need a parameter to increase timeouts.

The OS that Immich Server is running on

Kubernetes 1.27.4

Version of Immich Server

v1.120.1

Version of Immich Mobile App

v1.120.1

Platform with the issue

Your docker-compose.yml content

## This chart relies on the common library chart from bjw-s
## You can find it at https://github.com/bjw-s/helm-charts/tree/main/charts/library/common
## Refer there for more detail about the supported values

# These entries are shared between all the Immich components

env:
  REDIS_HOSTNAME: '{{ printf "%s-redis-master" .Release.Name }}'
  DB_HOSTNAME: "{{ .Release.Name }}-postgresql"
  DB_USERNAME: "{{ .Values.postgresql.global.postgresql.auth.username }}"
  DB_DATABASE_NAME: "{{ .Values.postgresql.global.postgresql.auth.database }}"
  # -- You should provide your own secret outside of this helm-chart and use `postgresql.global.postgresql.auth.existingSecret` to provide credentials to the postgresql instance
  DB_PASSWORD: "{{ .Values.postgresql.global.postgresql.auth.password }}"
  IMMICH_MACHINE_LEARNING_URL: '{{ printf "http://%s-machine-learning:3003" .Release.Name }}'

image:
  tag: v1.120.1
  # HELM chart version is in helm.yaml

immich:
  metrics:
    # Enabling this will create the service monitors needed to monitor immich with the prometheus operator
    enabled: false
  persistence:
    # Main data store for all photos shared between different components.
    library:
      # Automatically creating the library volume is not supported by this chart
      # You have to specify an existing PVC to use
      existingClaim: immich

# Dependencies

postgresql:
  enabled: true
  image:
    repository: tensorchord/pgvecto-rs
    tag: pg16-v0.2.0-rootless

  resources:
    limits: {}

  global:
    persistence:
      existingClaim: immich

    postgresql:
      auth:
        username: immich
        database: immich
        password: immich
  primary:
    containerSecurityContext:
      readOnlyRootFilesystem: false

    initdb:
      scripts:
        create-extensions.sql: |
          CREATE EXTENSION cube;
          CREATE EXTENSION earthdistance;
          CREATE EXTENSION vectors;

redis:
  enabled: true
  architecture: standalone
  auth:
    enabled: false

# Immich components

server:
  enabled: true
  image:
    repository: ghcr.io/immich-app/immich-server
    pullPolicy: IfNotPresent

machine-learning:
  enabled: true
  image:
    repository: ghcr.io/immich-app/immich-machine-learning
    pullPolicy: IfNotPresent
  env:
    TRANSFORMERS_CACHE: /cache
  persistence:
    cache:
      enabled: true
      size: 10Gi
      # Optional: Set this to pvc to avoid downloading the ML models every start.
      type: emptyDir
      accessMode: ReadWriteMany
      storageClass: raid

Your .env content

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: immich
spec:
  interval: 5m
  url: https://immich-app.github.io/immich-charts

---

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: immich
spec:
  interval: 10m
  releaseName: immich
  chart:
    spec:
      chart: immich
      version: '0.8.4'
      sourceRef:
        kind: HelmRepository
        name: immich
  valuesFrom:
    - kind: ConfigMap
      name: helm

Reproduction steps

  1. Run Immich server on slow drive

Relevant log output

[Nest] 17  - 11/11/2024, 9:58:09 AM     LOG [Api:Bootstrap] Immich Server is listening on http://[::1]:2283 [v1.120.1] [production]                                                                                                                                                                                        
[Nest] 7  - 11/11/2024, 9:59:31 AM     LOG [Microservices:MapRepository] 10000 geodata records imported                                                                                                                                                                                                                    
[Nest] 7  - 11/11/2024, 10:00:51 AM     LOG [Microservices:MapRepository] 20000 geodata records imported                                                                                                                                                                                                                   
[Nest] 7  - 11/11/2024, 10:01:49 AM     LOG [Microservices:MapRepository] 30000 geodata records imported                                                                                                                                                                                                                   
[Nest] 7  - 11/11/2024, 10:02:45 AM     LOG [Microservices:MapRepository] 40000 geodata records imported                                                                                                                                                                                                                   
[Nest] 7  - 11/11/2024, 10:03:50 AM     LOG [Microservices:MapRepository] 50000 geodata records imported                                                                                                                                                                                                                   
[Nest] 7  - 11/11/2024, 10:04:54 AM     LOG [Microservices:MapRepository] 60000 geodata records imported                                                                                                                                                                                                                   
[Nest] 7  - 11/11/2024, 10:06:12 AM     LOG [Microservices:MapRepository] 70000 geodata records imported                                                                                                                                                                                                                   
[Nest] 7  - 11/11/2024, 10:08:36 AM     LOG [Microservices:MapRepository] 80000 geodata records imported                                                                                                                                                                                                                   
[Nest] 7  - 11/11/2024, 10:10:32 AM     LOG [Microservices:MapRepository] 90000 geodata records imported                                                                                                                                                                                                                   
[Nest] 7  - 11/11/2024, 10:11:55 AM     LOG [Microservices:MapRepository] 100000 geodata records imported                                                                                                                                                                                                                  
[Nest] 7  - 11/11/2024, 10:14:45 AM     LOG [Microservices:MapRepository] 110000 geodata records imported                                                                                                                                                                                                                  
[Nest] 7  - 11/11/2024, 10:18:42 AM     LOG [Microservices:MapRepository] 120000 geodata records imported                                                                                                                                                                                                                  
[Nest] 17  - 11/11/2024, 10:22:47 AM   ERROR [Api:GlobalExceptionFilter~w9h7xry1] Unknown error: Error: Connection terminated due to connection timeout                                                                                                                                                                    
Error: Connection terminated due to connection timeout                                                                                                                                                                                                                                                                     
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:131:73)                                                                                                                                                                                                                                          
    at Object.onceWrapper (node:events:632:28)                                                                                                                                                                                                                                                                             
    at Connection.emit (node:events:518:28)                                                                                                                                                                                                                                                                                
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:62:12)                                                                                                                                                                                                                                           
    at Socket.emit (node:events:518:28)                                                                                                                                                                                                                                                                                    
    at TCP.<anonymous> (node:net:343:12)                                                                                                                                                                                                                                                                                   
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)                                                                                                                                                                                                                                                           
[Nest] 17  - 11/11/2024, 10:22:47 AM   ERROR [Api:GlobalExceptionFilter~qqebz4p2] Unknown error: Error: Connection terminated due to connection timeout                                                                                                                                                                    
Error: Connection terminated due to connection timeout                                                                                                                                                                                                                                                                     
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:131:73)                                                                                                                                                                                                                                          
    at Object.onceWrapper (node:events:632:28)                                                                                                                                                                                                                                                                             
    at Connection.emit (node:events:518:28)                                                                                                                                                                                                                                                                                
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:62:12)                                                                                                                                                                                                                                           
    at Socket.emit (node:events:518:28)                                                                                                                                                                                                                                                                                    
    at TCP.<anonymous> (node:net:343:12)                                                                                                                                                                                                                                                                                   
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)                                                                                                                                                                                                                                                           
[Nest] 17  - 11/11/2024, 10:22:47 AM   ERROR [Api:GlobalExceptionFilter~z13qnlfz] Unknown error: Error: Connection terminated due to connection timeout                                                                                                                                                                    
Error: Connection terminated due to connection timeout                                                                                                                                                                                                                                                                     
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:131:73)                                                                                                                                                                                                                                          
    at Object.onceWrapper (node:events:632:28)                                                                                                                                                                                                                                                                             
    at Connection.emit (node:events:518:28)                                                                                                                                                                                                                                                                                
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:62:12)                                                                                                                                                                                                                                           
    at Socket.emit (node:events:518:28)                                                                                                                                                                                                                                                                                    
    at TCP.<anonymous> (node:net:343:12)                                                                                                                                                                                                                                                                                   
    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)

Additional information

No response

bo0tzz commented 2 weeks ago

The geodata import process shouldn't ever take this long. What sort of drives are you using?

jonas-jasas commented 2 weeks ago

I am using Intel i7 with the 13 various old drives on BTRFS RAID 10. Speed is not great, not terrible. Still timeout should be configurable. There was no issues with the previous versions. I am not sure since when it started to happen but I guess with the 1.117 was everything fine.

gagan97 commented 1 week ago

I'm also getting this issue, since my pg docker was taking more then 2GB RAM, I tried to limit and now i'm getting connection timeouts

mmomjian commented 1 week ago

I'm also getting this issue, since my pg docker was taking more then 2GB RAM, I tried to limit and now i'm getting connection timeouts

What did you limit it to? Postgres recommends a minimum of 2 GB.

mmomjian commented 1 week ago

I am converting this to a discussion as I don't view this as an Immich bug at current. A postgres timeout change would be a reasonable feature request, but overall an SSD is strongly recommended for the database.