search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
53.51k stars 2.83k forks source link

Mysterious pgvecto.rs errors in 1.95.1 #7374

Closed Foritus closed 9 months ago

Foritus commented 9 months ago

The bug

Unsure if this is benign or indicative of an issue, but since upgrading I see this in the logs whenever I uploaded a new photo:

[Nest] 7  - 02/23/2024, 3:37:13 PM     LOG [CommunicationRepository] Websocket Disconnect: o4bKtLj1UpWOqNGQAAAJ
[Nest] 7  - 02/23/2024, 3:37:23 PM     LOG [CommunicationRepository] Websocket Connect:    NRdAnple4fZGcWNGAAAL
[Nest] 7  - 02/23/2024, 3:37:29 PM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset 277037e8-a12a-4e9b-a29c-594fd59130b8
[Nest] 7  - 02/23/2024, 3:37:29 PM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 277037e8-a12a-4e9b-a29c-594fd59130b8

[Nest] 7  - 02/23/2024, 3:37:29 PM   ERROR [JobService] Unable to run job handler (smartSearch/smart-search): QueryFailedError: pgvecto.rs: The index is not existing in the background worker.
ADVICE: Drop or rebuild the index.
[Nest] 7  - 02/23/2024, 3:37:29 PM   ERROR [JobService] QueryFailedError: pgvecto.rs: The index is not existing in the background worker.
ADVICE: Drop or rebuild the index.
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
    at async SearchRepository.upsertEmbedding (/usr/src/app/dist/infra/repositories/search.repository.js:125:9)
    at async SearchRepository.upsert (/usr/src/app/dist/infra/repositories/search.repository.js:122:9)
    at async SmartInfoService.handleEncodeClip (/usr/src/app/dist/domain/smart-info/smart-info.service.js:75:9)
    at async /usr/src/app/dist/domain/job/job.service.js:137:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
[Nest] 7  - 02/23/2024, 3:37:29 PM   ERROR [JobService] Object:
{
  "id": "277037e8-a12a-4e9b-a29c-594fd59130b8",
  "source": "upload"
}

The photo appears in my feed ok but I'm guessing maybe some search feature is broken? (I don't really use search so haven't noticed)

I've tried restart my PG server, no change.

The OS that Immich Server is running on

Debian 12 kubernetes worker node

Version of Immich Server

v1.95.1

Version of Immich Mobile App

v1.95.1 build.122

Platform with the issue

Your docker-compose.yml content

Unused (I self host on my local k8s cluster), here's my deployment yaml if you like boilerplate:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: photos
  namespace: photos
spec:
  replicas: 1
  selector:
    matchLabels:
      app: photos
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: photos
    spec:
      imagePullSecrets:
      - name: redacted
      securityContext:
        runAsNonRoot: true
        runAsUser: 2000
        runAsGroup: 3000
        fsGroup: 3000
      containers:
        - name: immich-server
          image: ((immich-server))
          command: ["/bin/sh", "start-server.sh"]
          envFrom:
            - configMapRef:
                name: photos-config
          volumeMounts:
            - mountPath: /usr/src/app/upload
              name: photos
          resources:
            requests:
              cpu: "250m"
              memory: "768Mi"
          livenessProbe:
            exec:
              command:
              - /bin/bash
              - -c
              - ./liveness.sh
            initialDelaySeconds: 10
            periodSeconds: 30
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 3

        - name: immich-microservices
          image: ((immich-microservices))
          command: ["/bin/sh", "start-microservices.sh"]
          envFrom:
            - configMapRef:
                name: photos-config
          volumeMounts:
            - mountPath: /usr/src/app/upload
              name: photos
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"

        - name: immich-machine-learning
          image: ((immich-machine-learning))
          envFrom:
            - configMapRef:
                name: photos-config
          volumeMounts:
            - mountPath: /cache
              name: ml-models
            - mountPath: /usr/src/app/upload
              name: photos
          resources:
            requests:
              cpu: "250m"
              memory: "4Gi"

      volumes:
        - name: ml-models
          persistentVolumeClaim:
            claimName: photos-ml-model-storage
        - name: photos
          persistentVolumeClaim:
            claimName: photos-storage

Your .env content

apiVersion: v1
kind: ConfigMap
metadata:
  name: photos-config
  namespace: photos
  labels:
    app: photos
data:
  ###################################################################################
  # Database
  ###################################################################################
  DB_HOSTNAME: "redacted"
  DB_USERNAME: "redacted"
  DB_PASSWORD: "redacted"
  DB_DATABASE_NAME: "redacted"

  # Optional Database settings:
  # DB_PORT=5432

  ###################################################################################
  # Redis
  ###################################################################################

  REDIS_HOSTNAME: "redacted"

  ###################################################################################
  # Upload File Location
  #
  # This is the location where uploaded files are stored.
  ###################################################################################

  UPLOAD_LOCATION: "/usr/src/app/upload/"

  ###################################################################################
  # Reverse Geocoding
  #
  # Reverse geocoding is done locally which has a small impact on memory usage
  # This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
  # This ranges from 0-3 with 3 being the most precise
  # 3 - Cities > 500 population: ~200MB RAM
  # 2 - Cities > 1000 population: ~150MB RAM
  # 1 - Cities > 5000 population: ~80MB RAM
  # 0 - Cities > 15000 population: ~40MB RAM
  ####################################################################################

  # DISABLE_REVERSE_GEOCODING=false
  # REVERSE_GEOCODING_PRECISION=3

  ####################################################################################
  # WEB - Optional
  #
  # Custom message on the login page, should be written in HTML form.
  # For example:
  # PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>demo@demo.de</i><br>Password: <i>demo</i>"
  ####################################################################################

  PUBLIC_LOGIN_PAGE_MESSAGE: ""

  ####################################################################################
  # Alternative Service Addresses - Optional
  #
  # This is an advanced feature for users who may be running their immich services on different hosts.
  # It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
  # Note: immich-microservices is bound to 3002, but no references are made
  ####################################################################################

  # Note: These are k8s-pod-DNS addresses, not LAN-routable
  IMMICH_WEB_URL: "http://photos:3000"
  IMMICH_SERVER_URL: "http://photos:3001"
  IMMICH_MACHINE_LEARNING_URL: "http://photos:3003"

  ####################################################################################
  # Alternative API's External Address - Optional
  #
  # This is an advanced feature used to control the public server endpoint returned to clients during Well-known discovery.
  # You should only use this if you want mobile apps to access the immich API over a custom URL. Do not include trailing slash.
  # NOTE: At this time, the web app will not be affected by this setting and will continue to use the relative path: /api
  # Examples: http://localhost:3001, http://immich-api.example.com, etc
  ####################################################################################

  IMMICH_API_URL_EXTERNAL: "redacted"

Reproduction steps

1. Upgrade from Immich v1.94.1 to v1.95.1 (Including upgrading the postgresql pgvecto.rs extension to 0.200)
2. Upload a new photo while watching the immich service logs
3. Observe weird errors due to the new pgvecto.rs stuff
...

Additional information

No response

danieldietzler commented 9 months ago

Did you follow the migration steps from the release notes?

Foritus commented 9 months ago

Yes (assuming you mean the DB config migration stuff?). I can try it again assuming its safe to run it again?

Foritus commented 9 months ago

There is a note on the db migration steps that says "This is normal. The server will do some magic and start to work." but the error keeps being logged each time I upload a new photo.