immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
46.08k stars 2.29k forks source link

[BUG] immich_microservices crash in k3s with v1.89.0 #5439

Closed yodatak closed 2 months ago

yodatak commented 10 months ago

The bug

With generation of thumbnails immich_microservices restart all the 2/5 minutes i try to disable geodata and it don't fix anything

My log

[Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [NestFactory] Starting Nest application...
2023-12-02T00:58:33.140206127+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +35ms
2023-12-02T00:58:33.140230396+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-12-02T00:58:33.140803237+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms
2023-12-02T00:58:33.140847863+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
2023-12-02T00:58:33.140955646+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
2023-12-02T00:58:33.144826992+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [InstanceLoader] ConfigModule dependencies initialized +4ms
2023-12-02T00:58:33.144853173+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-12-02T00:58:33.145027698+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
2023-12-02T00:58:33.271895996+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +127ms
2023-12-02T00:58:33.271979237+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM 
    LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
2023-12-02T00:58:33.274191071+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [InstanceLoader] InfraModule dependencies initialized +3ms
2023-12-02T00:58:33.286911670+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [InstanceLoader] DomainModule dependencies initialized +12ms
2023-12-02T00:58:33.287060735+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
2023-12-02T00:58:33.323392812+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [SearchService] Running bootstrap
2023-12-02T00:58:33.379858585+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [TypesenseRepository] Schema up to date: assets/assets-v10
2023-12-02T00:58:33.393335998+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [TypesenseRepository] Schema up to date: albums/albums-v2
2023-12-02T00:58:33.395475400+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [TypesenseRepository] Schema up to date: faces/faces-v1
2023-12-02T00:58:33.398248083+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [TypesenseRepository] Alias mapping: [{"collection_name":"albums-v2","name":"albums"}]
2023-12-02T00:58:33.398327250+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [TypesenseRepository] Collections needing migration: {"assets":true,"albums":false,"faces":true}
2023-12-02T00:58:33.422193782+01:00 [Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [NestApplication] Nest application successfully started +24ms
[Nest] 8  - 12/02/2023, 12:58:33 AM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:10004 [v1.89.0] [PRODUCTION] 
2023-12-02T00:58:39.238398325+01:00 [Nest] 8  - 12/02/2023, 12:58:39 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 6b88ff0a-72a3-4d65-a036-72ce2dd4f55c
2023-12-02T00:58:39.238414139+01:00 [Nest] 8  - 12/02/2023, 12:58:39 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset e146b317-0954-4f2e-b445-532118ea5995
2023-12-02T00:58:39.238416723+01:00 [Nest] 8  - 12/02/2023, 12:58:39 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset d9869d43-c4f2-4001-a2a6-e6d5482119df
2023-12-02T00:58:39.951044616+01:00 [Nest] 8  - 12/02/2023, 12:58:39 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 85699f13-c741-4075-b9db-3a7567491e1f
2023-12-02T00:58:44.990297684+01:00 [Nest] 8  - 12/02/2023, 12:58:44 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 4aa75333-cfa3-4360-b0d4-14fb7d325dd3
2023-12-02T00:58:44.991038248+01:00 [Nest] 8  - 12/02/2023, 12:58:44 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 13dd5fbb-7fa8-40e0-99bc-c0919e2c0430
2023-12-02T00:58:48.193724149+01:00 [Nest] 8  - 12/02/2023, 12:58:48 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 779f82f8-a22f-4a6f-a8ab-e1008df8d701
2023-12-02T00:58:48.749272269+01:00 [Nest] 8  - 12/02/2023, 12:58:48 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 78bd192a-df5f-4645-b99f-f
2228e29f0c1
2023-12-02T00:58:52.536384030+01:00 [Nest] 8  - 12/02/2023, 12:58:52 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 6342495f-81f3-4f4d-b91e-5f508414b01e
2023-12-02T00:58:55.502835875+01:00 [Nest] 8  - 12/02/2023, 12:58:55 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset b689cdcf-e6af-4159-9f72-aa3678ffcab4
2023-12-02T00:58:56.086038852+01:00 [Nest] 8  - 12/02/2023, 12:58:56 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 8f9c2cfd-1fb5-404c-beec-85855d86a955
2023-12-02T00:59:00.999630330+01:00 [Nest] 8  - 12/02/2023, 12:59:00 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 18a7e1b6-8d9b-4c56-b3b7-4e9c2b28b1ab
2023-12-02T00:59:02.131970353+01:00 [Nest] 8  - 12/02/2023, 12:59:02 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 6ea00a2e-68ce-4773-a14d-f9428b9e678c
2023-12-02T00:59:03.086966639+01:00 [Nest] 8  - 12/02/2023, 12:59:03 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset a3955b7a-b0e1-461f-80ef-664757768c22
2023-12-02T00:59:07.299434876+01:00 [Nest] 8  - 12/02/2023, 12:59:07 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset a26c7073-66e8-454e-aaaf-7e66f506c120
2023-12-02T00:59:08.658031483+01:00 [Nest] 8  - 12/02/2023, 12:59:08 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 7b25ab87-d663-4b80-bf9c-08327ed870cb
2023-12-02T00:59:15.255744440+01:00 [Nest] 8  - 12/02/2023, 12:59:15 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset c7c
900fd-457e-44b5-9ea0-8a54445a25d8
2023-12-02T00:59:18.269116416+01:00 [Nest] 8  - 12/02/2023, 12:59:18 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset e3e317e9-093a-4cdf-955b-3cecee72b428
2023-12-02T00:59:20.195532757+01:00 [Nest] 8  - 12/02/2023, 12:59:20 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 70ae818d-6a02-425a-97c7-d94939c8fcbd
2023-12-02T00:59:21.547554297+01:00 [Nest] 8  - 12/02/2023, 12:59:21 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset ac41049d-4731-4925-9550-4493094fb75c
2023-12-02T00:59:25.912006465+01:00 [Nest] 8  - 12/02/2023, 12:59:25 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset a054c2b6-8219-49a0-8923-bb0bffe19907
2023-12-02T00:59:30.430527578+01:00 [Nest] 8  - 12/02/2023, 12:59:30 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset d2e3d249-ef96-43c3-acba-95cdb0e117e2
2023-12-02T00:59:31.766361871+01:00 [Nest] 8  - 12/02/2023, 12:59:31 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset fcb4dcf1-7850-498c-985d-9ca1a0a5b95c
2023-12-02T00:59:35.420247206+01:00 [Nest] 8  - 12/02/2023, 12:59:35 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 3a432135-4c1e-49c6-88da-b5082756828e
2023-12-02T00:59:38.184168416+01:00 [Nest] 8  - 12/02/2023, 12:59:38 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset d9d2b021-2ad2-49e2-92f3-f5bb0e609ec8
2023-12-02T00:59:43.250772535+01:00 [Nest] 8  - 12/02/2023, 12:59:43 AM     LOG [MediaService] Successfully generated WEBP image t
humbnail for asset d0ec4629-336a-4cd5-b50e-a9797f079b59
2023-12-02T00:59:45.095521051+01:00 [Nest] 8  - 12/02/2023, 12:59:45 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset e738e56f-d552-467c-926f-0aaf13cbbf7c
2023-12-02T00:59:49.289729351+01:00 [Nest] 8  - 12/02/2023, 12:59:49 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 8b5eb00f-4555-4bfd-8b81-bfd036d6fff7
2023-12-02T00:59:52.231646741+01:00 [Nest] 8  - 12/02/2023, 12:59:52 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset c89cebb2-743f-4d49-bcf4-352cc9d45aed
2023-12-02T00:59:57.348400549+01:00 [Nest] 8  - 12/02/2023, 12:59:57 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 6daff61d-7bdb-43f2-8f6e-38a7de55462e
2023-12-02T00:59:59.964982049+01:00 [Nest] 8  - 12/02/2023, 12:59:59 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset b4c2daa1-4f69-4d9a-bfd2-dc7333d6abeb
2023-12-02T01:00:04.777166214+01:00 [Nest] 8  - 12/02/2023, 1:00:04 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset e51101aa-cbb6-479c-a6d8-eb105fde80be
2023-12-02T01:00:07.437448473+01:00 [Nest] 8  - 12/02/2023, 1:00:07 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 099a97ef-e3c6-408b-8df2-a38681395407
2023-12-02T01:00:14.489595244+01:00 [Nest] 8  - 12/02/2023, 1:00:14 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 20d1703c-4f7b-411c-9cb1-64a87e14b1b4
2023-12-02T01:00:17.351817074+01:00 [Nest] 8  - 12/02/2023, 1:00:17 AM     LOG [MediaService] Successfully gene
rated WEBP image thumbnail for asset fa4e1812-638b-466f-aab8-a28ee8fc3203
2023-12-02T01:00:22.789337784+01:00 [Nest] 8  - 12/02/2023, 1:00:22 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset d74ccf97-1e4e-410e-b0d9-aae79cb6ce05
2023-12-02T01:00:22.789369047+01:00 [Nest] 8  - 12/02/2023, 1:00:22 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset ef8a4f0f-9e5e-41b6-be9b-1b79bc1247f2
2023-12-02T01:00:22.790929292+01:00 [Nest] 8  - 12/02/2023, 1:00:22 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset db6e2d25-eeee-4e27-ab75-fd5e66070862
2023-12-02T01:00:26.558333774+01:00 [Nest] 8  - 12/02/2023, 1:00:26 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 8722de65-102d-4c55-8802-321907082925
2023-12-02T01:00:26.559311495+01:00 [Nest] 8  - 12/02/2023, 1:00:26 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset f90e4e92-6a80-49ac-9f12-4ad37f485ddd
2023-12-02T01:00:39.677543311+01:00 [Nest] 8  - 12/02/2023, 1:00:39 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 8cd6e5a1-b466-4abf-84ad-1a1db0dd658c
2023-12-02T01:00:39.678001118+01:00 [Nest] 8  - 12/02/2023, 1:00:39 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 38fe3ef8-7d74-4867-a7e7-cdc887f704b4
2023-12-02T01:00:51.770765512+01:00 [Nest] 8  - 12/02/2023, 1:00:51 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-webp-thumbnail): Error: Connection terminated due to connection timeout
2023-12-02T01:00:51.770791473+01:00 [Nest] 8  - 12/02/2023, 1:00:51 AM   ERROR 
[JobService] Error: Connection terminated due to connection timeout
2023-12-02T01:00:51.770794916+01:00     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
2023-12-02T01:00:51.770797316+01:00     at Object.onceWrapper (node:events:628:28)
2023-12-02T01:00:51.770799549+01:00     at Connection.emit (node:events:514:28)
2023-12-02T01:00:51.770802594+01:00     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
2023-12-02T01:00:51.770804753+01:00     at Socket.emit (node:events:514:28)
2023-12-02T01:00:51.770807282+01:00     at TCP.<anonymous> (node:net:337:12)
2023-12-02T01:00:51.770816083+01:00 [Nest] 8  - 12/02/2023, 1:00:51 AM   ERROR [JobService] Object:
2023-12-02T01:00:51.770818586+01:00 {
2023-12-02T01:00:51.770820882+01:00   "id": "38fe3ef8-7d74-4867-a7e7-cdc887f704b4"
2023-12-02T01:00:51.770823185+01:00 }
2023-12-02T01:00:51.770825118+01:00 
2023-12-02T01:00:51.771319168+01:00 [Nest] 8  - 12/02/2023, 1:00:51 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-webp-thumbnail): Error: Connection terminated due to connection timeout
2023-12-02T01:00:51.771325050+01:00 [Nest] 8  - 12/02/2023, 1:00:51 AM   ERROR [JobService] Error: Connection terminated due to connection timeout
2023-12-02T01:00:51.771682876+01:00     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
2023-12-02T01:00:51.771699052+01:00     at Object.onceWrapper (node:events:628:28)
2023-12-02T01:00:51.771702574+01:00     at Connection.emit (node:events:514:28)
2023-12-02T01:00:51.771705662+01:00     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
2023-12-02T01:00:51.771708482+01:00     at Socket.emit (node:events:514:28)
2023-12-02T01:00:51.771711127+01:00     at TCP.<anonymous> (node:net:337:12)
2023-12-02T01:00:51.771733802+01:00 [Nest] 8  - 12/0
2/2023, 1:00:51 AM   ERROR [JobService] Object:
2023-12-02T01:00:51.771744067+01:00 {
2023-12-02T01:00:51.771747150+01:00   "id": "8cd6e5a1-b466-4abf-84ad-1a1db0dd658c"
2023-12-02T01:00:51.771749071+01:00 }
2023-12-02T01:00:51.771751057+01:00 
2023-12-02T01:00:55.821022660+01:00 [Nest] 8  - 12/02/2023, 1:00:55 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 28f19ed2-0800-4fbc-97c3-cf66bba6053a
2023-12-02T01:01:09.978382350+01:00 [Nest] 8  - 12/02/2023, 1:01:09 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset 4d590bb8-7f81-44d2-9a10-1f0a5311f860

2023-12-02T01:01:17.771478652+01:00 <--- Last few GCs --->
2023-12-02T01:01:17.771481117+01:00 
2023-12-02T01:01:17.771483577+01:00 [8:0x53ad2310000]   165952 ms: Scavenge (reduce) 2045.4 (2082.8) -> 2044.5 (2082.8) MB, 3.65 / 0.00 ms  (average mu = 0.334, current mu = 0.349) allocation failure; 
2023-12-02T01:01:17.771487985+01:00 [8:0x53ad2310000]   166031 ms: Scavenge (reduce) 2045.4 (2082.8) -> 2044.6 (2082.8) MB, 17.18 / 0.00 ms  (average mu = 0.334, current mu = 0.349) allocation failure; 
2023-12-02T01:01:17.771492816+01:00 [8:0x53ad2310000]   166095 ms: Scavenge (reduce) 2045.5 (2082.8) -> 2044.7 (2083.1) MB, 3.33 / 0.00 ms  (average mu = 0.334, current mu = 0.349) allocation failure; 
2023-12-02T01:01:17.771495130+01:00 
2023-12-02T01:01:17.771497047+01:00 
<--- JS stacktrace --->
2023-12-02T01:01:17.771500994+01:00 
2023-12-02T01:01:17.771503327+01:00 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2023-12-02T01:01:17.772018679+01:00  1: 0xc99970 node::Abort() [immich_microservices]
2023-12-02T01:01:17.772454800+01:00  2: 0xb6ffcb  [immich_microservices]
2023-12-02T01:01:17.772919137+01:00  3: 0xebe9f0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
2023-12-02T01:01:17.773381921+01:00  4: 0xebecd7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [immich_microservices]
2023-12-02T01:01:17.773906910+01:00  5: 0x10d0785  [immich_microservices]
2023-12-02T01:01:17.774442251+01:00  6: 0x10d0d14 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices]
2023-12-02T01:01:17.774972657+01:00  7: 0x10e7c04 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [immich_microservices]
2023-12-02T01:01:17.775512923+01:00  8: 0x10e841c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices]
2023-12-02T01:01:17.776048819+01:00  9: 0x10ea57a v8::internal::Heap::HandleGCRequest() [immich_microservices]
2023-12-02T01:01:17.776564728+01:00 10: 0x10559e7 v8::internal::StackGuard::HandleInterrupts() [immich_microservices]
2023-12-02T01:01:17.777168337+01:00 11: 0x14f7402 v8::internal::Runtime_StackGuardWithGap(int, unsigned long*, v8::internal::Isolate*) [immich_microservices]
12: 0x7f6e41d99ef6 
2023-12-02T01:01:18.012393920+01:00 Aborted (core dumped)

The OS that Immich Server is running on

Fedora k3s

Version of Immich Server

v1.89.0

Version of Immich Mobile App

non used

Platform with the issue

Your docker-compose.yml content

The pod definition 

apiVersion: v1
kind: Pod
metadata:
  annotations:
    rollme: 4MYIi
  creationTimestamp: '2023-12-01T23:52:26Z'
  generateName: immich-microservices-b59d867bd-
  labels:
    app: immich-9.0.2
    app.kubernetes.io/instance: immich
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: immich
    app.kubernetes.io/version: v1.88.2
    helm-revision: '4'
    helm.sh/chart: immich-9.0.2
    pod-template-hash: b59d867bd
    pod.name: microservices
    release: immich
  managedFields:
    - apiVersion: v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:rollme: {}
          f:generateName: {}
          f:labels:
            .: {}
            f:app: {}
            f:app.kubernetes.io/instance: {}
            f:app.kubernetes.io/managed-by: {}
            f:app.kubernetes.io/name: {}
            f:app.kubernetes.io/version: {}
            f:helm-revision: {}
            f:helm.sh/chart: {}
            f:pod-template-hash: {}
            f:pod.name: {}
            f:release: {}
          f:ownerReferences:
            .: {}
            k:{"uid":"f190709f-048b-467d-9d96-34392d059285"}: {}
        f:spec:
          f:automountServiceAccountToken: {}
          f:containers:
            k:{"name":"immich"}:
              .: {}
              f:args: {}
              f:command: {}
              f:env:
                .: {}
                k:{"name":"S6_READ_ONLY_ROOT"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"TZ"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"UMASK"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"UMASK_SET"}:
                  .: {}
                  f:name: {}
                  f:value: {}
              f:envFrom: {}
              f:image: {}
              f:imagePullPolicy: {}
              f:livenessProbe:
                .: {}
                f:failureThreshold: {}
                f:initialDelaySeconds: {}
                f:periodSeconds: {}
                f:successThreshold: {}
                f:tcpSocket:
                  .: {}
                  f:port: {}
                f:timeoutSeconds: {}
              f:name: {}
              f:ports:
                .: {}
                k:{"containerPort":10004,"protocol":"TCP"}:
                  .: {}
                  f:containerPort: {}
                  f:name: {}
                  f:protocol: {}
              f:readinessProbe:
                .: {}
                f:failureThreshold: {}
                f:initialDelaySeconds: {}
                f:periodSeconds: {}
                f:successThreshold: {}
                f:tcpSocket:
                  .: {}
                  f:port: {}
                f:timeoutSeconds: {}
              f:resources:
                .: {}
                f:limits:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
                f:requests:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
              f:securityContext:
                .: {}
                f:allowPrivilegeEscalation: {}
                f:capabilities:
                  .: {}
                  f:add: {}
                f:privileged: {}
                f:readOnlyRootFilesystem: {}
                f:runAsGroup: {}
                f:runAsNonRoot: {}
                f:runAsUser: {}
                f:seccompProfile:
                  .: {}
                  f:type: {}
              f:startupProbe:
                .: {}
                f:failureThreshold: {}
                f:initialDelaySeconds: {}
                f:periodSeconds: {}
                f:successThreshold: {}
                f:tcpSocket:
                  .: {}
                  f:port: {}
                f:timeoutSeconds: {}
              f:terminationMessagePath: {}
              f:terminationMessagePolicy: {}
              f:volumeMounts:
                .: {}
                k:{"mountPath":"/dev/shm"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/microcache"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/mnt/XXXX/Photos Videos"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/shared"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/tmp"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/usr/src/app/upload/encoded-video"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/usr/src/app/upload/library"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/usr/src/app/upload/profile"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/usr/src/app/upload/thumbs"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/usr/src/app/upload/upload"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/var/logs"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/var/run"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
          f:dnsConfig:
            .: {}
            f:options: {}
          f:dnsPolicy: {}
          f:enableServiceLinks: {}
          f:initContainers:
            .: {}
            k:{"name":"immich-init-wait-server"}:
              .: {}
              f:command: {}
              f:env:
                .: {}
                k:{"name":"S6_READ_ONLY_ROOT"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"TZ"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"UMASK"}:
                  .: {}
                  f:name: {}
                  f:value: {}
                k:{"name":"UMASK_SET"}:
                  .: {}
                  f:name: {}
                  f:value: {}
              f:image: {}
              f:imagePullPolicy: {}
              f:name: {}
              f:resources:
                .: {}
                f:limits:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
                f:requests:
                  .: {}
                  f:cpu: {}
                  f:memory: {}
              f:securityContext:
                .: {}
                f:allowPrivilegeEscalation: {}
                f:capabilities:
                  .: {}
                  f:add: {}
                f:privileged: {}
                f:readOnlyRootFilesystem: {}
                f:runAsGroup: {}
                f:runAsNonRoot: {}
                f:runAsUser: {}
                f:seccompProfile:
                  .: {}
                  f:type: {}
              f:terminationMessagePath: {}
              f:terminationMessagePolicy: {}
              f:volumeMounts:
                .: {}
                k:{"mountPath":"/dev/shm"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/mnt/XXXX/Photos Videos"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/shared"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/tmp"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/var/logs"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
                k:{"mountPath":"/var/run"}:
                  .: {}
                  f:mountPath: {}
                  f:name: {}
          f:nodeSelector: {}
          f:restartPolicy: {}
          f:runtimeClassName: {}
          f:schedulerName: {}
          f:securityContext:
            .: {}
            f:fsGroup: {}
            f:fsGroupChangePolicy: {}
            f:supplementalGroups: {}
          f:serviceAccount: {}
          f:serviceAccountName: {}
          f:shareProcessNamespace: {}
          f:terminationGracePeriodSeconds: {}
          f:topologySpreadConstraints:
            .: {}
            k:{"topologyKey":"kubernetes.io/hostname","whenUnsatisfiable":"ScheduleAnyway"}:
              .: {}
              f:labelSelector: {}
              f:maxSkew: {}
              f:nodeAffinityPolicy: {}
              f:nodeTaintsPolicy: {}
              f:topologyKey: {}
              f:whenUnsatisfiable: {}
            k:{"topologyKey":"truecharts.org/rack","whenUnsatisfiable":"ScheduleAnyway"}:
              .: {}
              f:labelSelector: {}
              f:maxSkew: {}
              f:nodeAffinityPolicy: {}
              f:nodeTaintsPolicy: {}
              f:topologyKey: {}
              f:whenUnsatisfiable: {}
          f:volumes:
            .: {}
            k:{"name":"devshm"}:
              .: {}
              f:emptyDir:
                .: {}
                f:medium: {}
              f:name: {}
            k:{"name":"library"}:
              .: {}
              f:hostPath:
                .: {}
                f:path: {}
                f:type: {}
              f:name: {}
            k:{"name":"microcache"}:
              .: {}
              f:emptyDir: {}
              f:name: {}
            k:{"name":"originals-photos"}:
              .: {}
              f:hostPath:
                .: {}
                f:path: {}
                f:type: {}
              f:name: {}
            k:{"name":"profile"}:
              .: {}
              f:hostPath:
                .: {}
                f:path: {}
                f:type: {}
              f:name: {}
            k:{"name":"shared"}:
              .: {}
              f:emptyDir: {}
              f:name: {}
            k:{"name":"thumbs"}:
              .: {}
              f:hostPath:
                .: {}
                f:path: {}
                f:type: {}
              f:name: {}
            k:{"name":"tmp"}:
              .: {}
              f:emptyDir: {}
              f:name: {}
            k:{"name":"uploads"}:
              .: {}
              f:hostPath:
                .: {}
                f:path: {}
                f:type: {}
              f:name: {}
            k:{"name":"varlogs"}:
              .: {}
              f:emptyDir: {}
              f:name: {}
            k:{"name":"varrun"}:
              .: {}
              f:emptyDir:
                .: {}
                f:medium: {}
              f:name: {}
            k:{"name":"video"}:
              .: {}
              f:hostPath:
                .: {}
                f:path: {}
                f:type: {}
              f:name: {}
      manager: k3s
      operation: Update
      time: '2023-12-01T23:52:26Z'
    - apiVersion: v1
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:conditions:
            k:{"type":"ContainersReady"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
            k:{"type":"Initialized"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
            k:{"type":"Ready"}:
              .: {}
              f:lastProbeTime: {}
              f:lastTransitionTime: {}
              f:status: {}
              f:type: {}
          f:containerStatuses: {}
          f:hostIP: {}
          f:initContainerStatuses: {}
          f:phase: {}
          f:podIP: {}
          f:podIPs:
            .: {}
            k:{"ip":"10.42.0.195"}:
              .: {}
              f:ip: {}
          f:startTime: {}
      manager: k3s
      operation: Update
      subresource: status
      time: '2023-12-02T00:05:14Z'
  name: immich-microservices-b59d867bd-7b4g2
  namespace: photos
  ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: immich-microservices-b59d867bd
      uid: f190709f-048b-467d-9d96-34392d059285
  resourceVersion: '115354950'
  uid: 1c07785d-b70c-4870-9dd5-b7dc072f8e27
spec:
  automountServiceAccountToken: false
  containers:
    - args:
        - '-c'
        - /usr/src/app/start-microservices.sh
      command:
        - /bin/sh
      env:
        - name: TZ
          value: Europe/Paris
        - name: UMASK
          value: '0022'
        - name: UMASK_SET
          value: '0022'
        - name: S6_READ_ONLY_ROOT
          value: '1'
      envFrom:
        - secretRef:
            name: immich-secret
        - secretRef:
            name: immich-deps-secret
        - configMapRef:
            name: immich-common-config
        - configMapRef:
            name: immich-micro-config
      image: >-
        altran1502/immich-server:v1.89.0@sha256:76484876f1b057d623925402ddf9ef3b79d27812fc4a1cb9b596719bf99b363d
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 5
        initialDelaySeconds: 10
        periodSeconds: 10
        successThreshold: 1
        tcpSocket:
          port: 10004
        timeoutSeconds: 5
      name: immich
      ports:
        - containerPort: 10004
          name: microservices
          protocol: TCP
      readinessProbe:
        failureThreshold: 5
        initialDelaySeconds: 10
        periodSeconds: 10
        successThreshold: 2
        tcpSocket:
          port: 10004
        timeoutSeconds: 5
      resources:
        limits:
          cpu: '4'
          memory: 8Gi
        requests:
          cpu: 10m
          memory: 50Mi
      securityContext:
        allowPrivilegeEscalation: true
        capabilities:
          add:
            - ALL
        privileged: true
        readOnlyRootFilesystem: false
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
      startupProbe:
        failureThreshold: 60
        initialDelaySeconds: 10
        periodSeconds: 5
        successThreshold: 1
        tcpSocket:
          port: 10004
        timeoutSeconds: 2
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /dev/shm
          name: devshm
        - mountPath: /usr/src/app/upload/library
          name: library
        - mountPath: /microcache
          name: microcache
        - mountPath: /mnt/XXXX/Photos Videos
          name: originals-photos
        - mountPath: /usr/src/app/upload/profile
          name: profile
        - mountPath: /shared
          name: shared
        - mountPath: /usr/src/app/upload/thumbs
          name: thumbs
        - mountPath: /tmp
          name: tmp
        - mountPath: /usr/src/app/upload/upload
          name: uploads
        - mountPath: /var/logs
          name: varlogs
        - mountPath: /var/run
          name: varrun
        - mountPath: /usr/src/app/upload/encoded-video
          name: video
  dnsConfig:
    options:
      - name: ndots
        value: '1'
  dnsPolicy: ClusterFirst
  enableServiceLinks: false
  initContainers:
    - command:
        - /bin/ash
        - '-c'
        - >
          echo "Pinging [http://immich:10323/api/server-info/ping] until it is
          ready..."

          until wget --spider --quiet --timeout=3 --tries 1
          "http://immich:10323/api/server-info/ping"; do
            echo "Waiting for [http://immich:10323/api/server-info/ping] to be ready..."
            sleep 2
          done

          echo "URL [http://immich:10323/api/server-info/ping] is ready!"
      env:
        - name: TZ
          value: Europe/Paris
        - name: UMASK
          value: '0022'
        - name: UMASK_SET
          value: '0022'
        - name: S6_READ_ONLY_ROOT
          value: '1'
      image: >-
        tccr.io/truecharts/alpine:v3.18.4@sha256:17cd77e25d3fa829d168caec4db7bb5b52ceeb935d8ca0d1180de6f615553dc4
      imagePullPolicy: IfNotPresent
      name: immich-init-wait-server
      resources:
        limits:
          cpu: '4'
          memory: 8Gi
        requests:
          cpu: 10m
          memory: 50Mi
      securityContext:
        allowPrivilegeEscalation: true
        capabilities:
          add:
            - ALL
        privileged: true
        readOnlyRootFilesystem: false
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
        seccompProfile:
          type: RuntimeDefault
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /dev/shm
          name: devshm
        - mountPath: /mnt/XXXX/Photos Videos
          name: originals-photos
        - mountPath: /shared
          name: shared
        - mountPath: /tmp
          name: tmp
        - mountPath: /var/logs
          name: varlogs
        - mountPath: /var/run
          name: varrun
  nodeName: anarchy.home
  nodeSelector:
    kubernetes.io/arch: amd64
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  runtimeClassName: nvidia
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 100
    fsGroupChangePolicy: OnRootMismatch
    supplementalGroups:
      - 568
      - 1000
      - 0
      - 568
  serviceAccount: default
  serviceAccountName: default
  shareProcessNamespace: false
  terminationGracePeriodSeconds: 60
  tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
  topologySpreadConstraints:
    - labelSelector:
        matchLabels:
          app.kubernetes.io/instance: immich
          app.kubernetes.io/name: immich
          pod.name: immich-microservices
      maxSkew: 1
      nodeAffinityPolicy: Honor
      nodeTaintsPolicy: Honor
      topologyKey: truecharts.org/rack
      whenUnsatisfiable: ScheduleAnyway
    - labelSelector:
        matchLabels:
          app.kubernetes.io/instance: immich
          app.kubernetes.io/name: immich
          pod.name: immich-microservices
      maxSkew: 1
      nodeAffinityPolicy: Honor
      nodeTaintsPolicy: Honor
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: ScheduleAnyway
  volumes:
    - emptyDir:
        medium: Memory
      name: devshm
    - hostPath:
        path: /mnt/XXXX/immich/library
        type: Directory
      name: library
    - emptyDir: {}
      name: microcache
    - hostPath:
        path: /mnt/XXXX/Photos Videos
        type: Directory
      name: originals-photos
    - hostPath:
        path: /mnt/XXXX/immich/profile
        type: Directory
      name: profile
    - emptyDir: {}
      name: shared
    - hostPath:
        path: /opt/immich/thumbs
        type: Directory
      name: thumbs
    - emptyDir: {}
      name: tmp
    - hostPath:
        path: /mnt/XXXXX/immich/upload
        type: Directory
      name: uploads
    - emptyDir: {}
      name: varlogs
    - emptyDir:
        medium: Memory
      name: varrun
    - hostPath:
        path: /mnt/XXXX/immich/encoded-video
        type: Directory
      name: video
status:
  conditions:
    - lastProbeTime: null
      lastTransitionTime: '2023-12-01T23:52:32Z'
      status: 'True'
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: '2023-12-02T00:05:14Z'
      status: 'True'
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: '2023-12-02T00:05:14Z'
      status: 'True'
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: '2023-12-01T23:52:26Z'
      status: 'True'
      type: PodScheduled
  containerStatuses:
    - containerID: >-
        containerd://671e9afb08a34885e634fa457058a5f05cb5e9124c9e9f37afcc16df544589a0
      image: sha256:2055a885b23bde5bcbe6b36652a02508c5d8edaa9f920168bca672f823f9e309
      imageID: >-
        docker.io/altran1502/immich-server@sha256:76484876f1b057d623925402ddf9ef3b79d27812fc4a1cb9b596719bf99b363d
      lastState:
        terminated:
          containerID: >-
            containerd://c02d9f1a28b22709822af76e5742d01b36dc776e6ccf7038b7497c11f1df3095
          exitCode: 134
          finishedAt: '2023-12-02T00:04:12Z'
          reason: Error
          startedAt: '2023-12-02T00:01:45Z'
      name: immich
      ready: true
      restartCount: 4
      started: true
      state:
        running:
          startedAt: '2023-12-02T00:05:00Z'
  hostIP: 192.168.X.XX
  initContainerStatuses:
    - containerID: >-
        containerd://655c419760c5b0fd2133ec0aee455834ad24f88590160c0a5ed363539e0c50c2
      image: sha256:d8ec0b5ed4e2e85ac9e6a66e7ad9e93d5c2a8f04facfca224941756344b63f72
      imageID: >-
        tccr.io/truecharts/alpine@sha256:17cd77e25d3fa829d168caec4db7bb5b52ceeb935d8ca0d1180de6f615553dc4
      lastState: {}
      name: immich-init-wait-server
      ready: true
      restartCount: 0
      state:
        terminated:
          containerID: >-
            containerd://655c419760c5b0fd2133ec0aee455834ad24f88590160c0a5ed363539e0c50c2
          exitCode: 0
          finishedAt: '2023-12-01T23:52:30Z'
          reason: Completed
          startedAt: '2023-12-01T23:52:30Z'
  phase: Running
  podIP: 10.42.0.195
  podIPs:
    - ip: 10.42.0.195
  qosClass: Burstable
  startTime: '2023-12-01T23:52:26Z'

### Your .env content

```Shell
My flux file

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: immich
  namespace: XXXX
spec:
  chart:
    spec:
      chart: immich
      interval: 5m
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: truecharts
        namespace: flux-system
      version: 9.0.2
  test:
    enable: true
    ignoreFailures: false
  interval: 10m
  values:
    TZ: Europe/Paris

    operator:
      register: true
      verify:
        enabled: false
        failOnError: false

    containerOptions:
      NVIDIA_CAPS:
        - all

    podOptions:
      runtimeClassName: "nvidia"

    # scaleGPU:
    # - gpu:
    #     nvidia.com/gpu: "1"

    image:
      repository: altran1502/immich-server
      tag: v1.89.0@sha256:76484876f1b057d623925402ddf9ef3b79d27812fc4a1cb9b596719bf99b363d
      pullPolicy: IfNotPresent
    mlImage:
      repository: altran1502/immich-machine-learning
      tag: v1.89.0@sha256:7a3217a8e4425e49984cd54ff42dc32bf45775e7c5701299c3e6fbc255cd1c54
      pullPolicy: IfNotPresent
    typesenseImage:
      repository: typesense/typesense
      tag: 0.25.1@sha256:3724f68156097114d07790e1c8bfaaca13b12eaecd602776d30d9933029064c7
      pullPolicy: IfNotPresent

    immich:
      enable_ml: true
      enable_typesense: true
      public_login_page_message: Immich
      log_level: log

    persistence:
      library:
        enabled: true
        mountPath: /usr/src/app/upload/library
        hostPath: /mnt/XXXX/immich/library
        hostPathType: Directory
        type: hostPath
      uploads:
        enabled: true
        mountPath: /usr/src/app/upload/upload
        hostPath: /mnt/XXXX/immich/upload
        hostPathType: Directory
        type: hostPath
      thumbs:
        enabled: true
        mountPath: /usr/src/app/upload/thumbs
        hostPath: /opt/immich/thumbs
        hostPathType: Directory
        type: hostPath
      profile:
        enabled: true
        mountPath: /usr/src/app/upload/profile
        hostPath: /mnt/XXXXX/immich/profile
        hostPathType: Directory
        type: hostPath
      video:
        enabled: true
        mountPath: /usr/src/app/upload/encoded-video
        hostPath: /mnt/XXXXX/immich/encoded-video
        hostPathType: Directory
        type: hostPath
      originals-photos:
        enabled: true
        hostPath: "XXXX"
        hostPathType: Directory
        mountPath: "XXXX"
        targetSelectAll: true
        type: hostPath

    ingress:
      main:
        enabled: true
        ingressClassName: traefik
        enableFixedMiddlewares: false
        hosts:
          - host: XXXXX
            paths:
              - path: /
                pathType: Prefix
        tls:
          - hosts:
              - XXXXX
            secretName: XXXX-tls-secret

    workload:
      main:
        podSpec:
          containers:
            main:
              env:
                NVIDIA_DRIVER_CAPABILITIES: all
                NVIDIA_VISIBLE_DEVICES: all

    securityContext:
      container:
        readOnlyRootFilesystem: false
        runAsNonRoot: false
        runAsUser: 1000
        runAsGroup: 1000
        allowPrivilegeEscalation: true
        privileged: true
        capabilities:
          add: 
             - ALL
          drop: []
          # When set to false, it will automatically
          # add CHOWN, SETUID, SETGID, FOWNER, DAC_OVERRIDE
          # capabilities ONLY when container runs as ROOT
          #disableS6Caps: false
        # -- PUID for all containers
        # Can be overruled per container
        PUID: 1000
        # -- UMASK for all containers
        # Can be overruled per container
        UMASK: "0022"
      pod:
        fsGroup: 100
        fsGroupChangePolicy: OnRootMismatch
        supplementalGroups: 
          - 568
          - 1000
          - 0

### Reproduction steps

```bash
1. run thumbernail genration with big library 
2. it crash
3.
...

Additional information

No response

bo0tzz commented 10 months ago

I don't think this is due to the helm chart you're using, but just in case, can you try with another approach such as the official chart? Truecharts is not really meant to be used on anything other than TrueNas Scale.

yodatak commented 9 months ago

I got the issue too :/ seem like maybe an issue with typesence how could i help to debug it ?

yodatak commented 9 months ago

maybe its because of bad asset like this

[Nest] 8  - 12/07/2023, 1:20:13 AM   ERROR [JobService] Error: ffprobe exited with code 1
2023-12-07T01:20:13.445608551+01:00 ffprobe version 6.0-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
2023-12-07T01:20:13.445624362+01:00   configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
  libavutil      58.  2.100 / 58.  2.100
2023-12-07T01:20:13.445629549+01:00   libavcodec     60.  3.100 / 60.  3.100
2023-12-07T01:20:13.445632236+01:00   libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
2023-12-07T01:20:13.445638127+01:00   libavfilter     9.  3.100 /  9.  3.100
2023-12-07T01:20:13.445640674+01:00   libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x22428190180] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x22428190180] moov atom not found
2023-12-07T01:20:13.445654639+01:00 /mnt/XXXX/rpd-tmp-soabixfp/q1OFr.mp4: Invalid data found when processing input

    at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
    at ChildProcess.emit (node:events:514:28)
2023-12-07T01:20:13.445666299+01:00     at ChildProcess._handle.onexit (node:internal/child_process:294:12)
[Nest] 8  - 12/07/2023, 1:20:13 AM   ERROR [JobService] Object:
{
  "id": "24f75584-cf4b-4efc-b9ee-0377d40b64d0"
}

2023-12-07T01:20:13.445902746+01:00 [Nest] 8  - 12/07/2023, 1:20:13 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: ffprobe exited with code 1
2023-12-07T01:20:13.445912833+01:00 ffprobe version 6.0-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
2023-12-07T01:20:13.445920499+01:00   configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
2023-12-07T01:20:13.445930965+01:00   libavutil      58.  2.100 / 58.  2.100
2023-12-07T01:20:13.445993317+01:00   libavcodec     60.  3.100 / 60.  3.100
2023-12-07T01:20:13.445997245+01:00   libavformat    60.  3.100 / 60.  3.100
2023-12-07T01:20:13.446000135+01:00   libavdevice    60.  1.100 / 60.  1.100
2023-12-07T01:20:13.446003119+01:00   libavfilter     9.  3.100 /  9.  3.100
2023-12-07T01:20:13.446005582+01:00   libswscale      7.  1.100 /  7.  1.100
2023-12-07T01:20:13.446008132+01:00   libswresample   4. 10.100 /  4. 10.100
2023-12-07T01:20:13.446011031+01:00   libpostproc    57.  1.100 / 57.  1.100
2023-12-07T01:20:13.446013466+01:00 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x5485a190180] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5485a190180] moov atom not found
2023-12-07T01:20:13.446022581+01:00 /mnt/XXXX/rpd-tmp-soabixfp/MJPBG.mp4: Invalid data found when processing input
2023-12-07T01:20:13.446025810+01:00 
[Nest] 8  - 12/07/2023, 1:20:13 AM   ERROR [JobService] Error: ffprobe exited with code 1
ffprobe version 6.0-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
2023-12-07T01:20:13.446042878+01:00   built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
2023-12-07T01:20:13.446048847+01:00   libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
2023-12-07T01:20:13.446059579+01:00   libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
2023-12-07T01:20:13.446064002+01:00   libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5485a190180] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
2023-12-07T01:20:13.446073431+01:00 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x5485a190180] moov atom not found
/mnt/XXXX/rpd-tmp-soabixfp/MJPBG.mp4: Invalid data found when processing input
2023-12-07T01:20:13.446078350+01:00 
    at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
2023-12-07T01:20:13.446083709+01:00     at ChildProcess.emit (node:events:514:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12)
[Nest] 8  - 12/07/2023, 1:20:13 AM   ERROR [JobService] Object:
2023-12-07T01:20:13.446095132+01:00 {
  "id": "3d96642c-3c5a-4ea3-95b9-167abfe5e140"
}

[Nest] 8  - 12/07/2023, 1:20:14 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: ffprobe exited with code 1
2023-12-07T01:20:14.427169330+01:00 ffprobe version 6.0-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
2023-12-07T01:20:14.427173586+01:00   built with gcc 12 (Debian 12.2.0-14)
2023-12-07T01:20:14.427182399+01:00   configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
2023-12-07T01:20:14.427185346+01:00   libavutil      58.  2.100 / 58.  2.100
2023-12-07T01:20:14.427187855+01:00   libavcodec     60.  3.100 / 60.  3.100
2023-12-07T01:20:14.427190374+01:00   libavformat    60.  3.100 / 60.  3.100
2023-12-07T01:20:14.427192994+01:00   libavdevice    60.  1.100 / 60.  1.100
2023-12-07T01:20:14.427195525+01:00   libavfilter     9.  3.100 /  9.  3.100
2023-12-07T01:20:14.427197969+01:00   libswscale      7.  1.100 /  7.  1.100
2023-12-07T01:20:14.427200255+01:00   libswresample   4. 10.100 /  4. 10.100
2023-12-07T01:20:14.427203290+01:00   libpostproc    57.  1.100 / 57.  1.100
2023-12-07T01:20:14.427206537+01:00 [mov,mp4,m4a,3gp,
3g2,mj2 @ 0x4f0e4190180] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
2023-12-07T01:20:14.427209150+01:00 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x4f0e4190180] moov atom not found
2023-12-07T01:20:14.427212612+01:00 /mnt/XXXX/rpd-tmp-soabixfp/gA0YT.mp4: Invalid data found when processing input
2023-12-07T01:20:14.427215207+01:00 
2023-12-07T01:20:14.427218217+01:00 [Nest] 8  - 12/07/2023, 1:20:14 AM   ERROR [JobService] Error: ffprobe exited with code 1
2023-12-07T01:20:14.427223419+01:00 ffprobe version 6.0-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
2023-12-07T01:20:14.427226252+01:00   built with gcc 12 (Debian 12.2.0-14)
2023-12-07T01:20:14.427228971+01:00   configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
2023-12-07T01:20:14.427238655+01:00   libavutil      58.  2.100 / 58.  2.100
2023-12-07T01:20:14.427241296+01:00   libavcodec     60.  3.100 / 60.  3.100
2023-12-07T01:20:14.427243760+01:00   libavformat    60.  3.100 / 60.  3.100
2023-12-07T01:20:14.427246156+01:00   libavdevice    60.  1.100 / 60.  1.100
2023-12-07T01:20:14.427248541+01:00   libavfi
lter     9.  3.100 /  9.  3.100
2023-12-07T01:20:14.427250935+01:00   libswscale      7.  1.100 /  7.  1.100
2023-12-07T01:20:14.427253725+01:00   libswresample   4. 10.100 /  4. 10.100
2023-12-07T01:20:14.427256504+01:00   libpostproc    57.  1.100 / 57.  1.100
2023-12-07T01:20:14.427259200+01:00 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x4f0e4190180] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
2023-12-07T01:20:14.427261658+01:00 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x4f0e4190180] moov atom not found
2023-12-07T01:20:14.427264790+01:00 /mnt/XXXX/rpd-tmp-soabixfp/gA0YT.mp4: Invalid data found when processing input
2023-12-07T01:20:14.427267146+01:00 
2023-12-07T01:20:14.427269933+01:00     at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
2023-12-07T01:20:14.427272802+01:00     at ChildProcess.emit (node:events:514:28)
2023-12-07T01:20:14.427275563+01:00     at ChildProcess._handle.onexit (node:internal/child_process:294:12)
2023-12-07T01:20:14.427322046+01:00 [Nest] 8  - 12/07/2023, 1:20:14 AM   ERROR [JobService] Object:
2023-12-07T01:20:14.427326593+01:00 {
2023-12-07T01:20:14.427329406+01:00   "id": "cc3533f3-2dd6-4bca-b6c0-a24545fa140d"
2023-12-07T01:20:14.427332118+01:00 }
2023-12-07T01:20:14.427334851+01:00 
[Nest] 8  - 12/07/2023, 1:20:24 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:24.907438089+01:00 [Nest] 8  - 12/07/2023, 1:20:24 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:24.907449640+01:00 [Nest] 8  - 12/07/2023, 1:20:24 AM   ERROR [JobService] Object:
2023-12-07T01:20:24.907453387+01:00 {
2023-12-07T01:20:24.907457034+01:00   "id": "4d705f98-f1e6-4c4b-ae79-80d7d414009b"
2023-12-07T01:20:24.907460485+01:00 }

[Nest] 8  - 12/07/2023, 1:20:24 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
[Nest] 8  - 12/07/2023, 1:20:24 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:24.908150062+01:00 [Nest] 8  - 12/07/2023, 1:20:24 AM   ERROR [JobService] Object:
{
2023-12-07T01:20:24.908161547+01:00   "id": "e3244edd-09fd-43a2-a2b8-28a5f89cc4b6"
2023-12-07T01:20:24.908164187+01:00 }
2023-12-07T01:20:24.908166470+01:00 
2023-12-07T01:20:24.908196195+01:00 [Nest] 8  - 12/07/2023, 1:20:24 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:24.908232916+01:00 [Nest] 8  - 12/07/2023, 1:20:24 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:24.908275477+01:00 [Nest] 8  - 12/07/2023, 1:20:24 AM   ERROR [JobService] Object:
2023-12-07T01:20:24.908283279+01:00 {
2023-12-07T01:20:24.908287108+01:00   "id": "1371ace3-b282-4b5c-9628-1b1e2ba3fe60"
2023-12-07T01:20:24.908290356+01:00 }
2023-12-07T01:20:24.908293053+01:00 
[Nest] 8  - 12/07/2023, 1:20:27 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:27.449111818+01:00 [Nest] 8  - 12/07/2023, 1:20:27 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:27.449150253+01:00 [Nest] 8  - 12/07/2023, 1:20:27 AM   ERROR [JobService] Object:
2023-12-07T01:20:27.449160581+01:00 {
2023-12-07T01:20:27.449164826+01:00   "id": "eca003db-e961-4e17-bb1a-d6b877f4d077"
}
2023-12-07T01:20:27.449171845+01:00 
[Nest] 8  - 12/07/2023, 1:20:28 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:28.556595864+01:00 [Nest] 8  - 12/07/2023, 1:20:28 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:28.556606060+01:00 [Nest] 8  - 12/07/2023, 1:20:28 AM   ERROR [JobService] Object:
2023-12-07T01:20:28.556609704+01:00 {
2023-12-07T01:20:28.556613307+01:00   "id": "f3de91b4-683a-4ffd-8866-ae94f384fcdd"
2023-12-07T01:20:28.556615882+01:00 }
2023-12-07T01:20:28.556618249+01:00 
[Nest] 8  - 12/07/2023, 1:20:28 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:28.557195844+01:00 [Nest] 8  - 12/07/2023, 1:20:28 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:28.557199484+01:00 [Nest] 8  - 12/07/2023, 1:20:28 AM   ERROR [JobService] Object:
2023-12-07T01:20:28.557202449+01:00 {
2023-12-07T01:20:28.557205444+01:00   "id": "d128dcda-cb65-4657-aa08-8edc23106c92"
2023-12-07T01:20:28.557208067+01:00 }
2023-12-07T01:20:28.557210462+01:00 
2023-12-07T01:20:28.557269054+01:00 [Nest] 8  - 12/07/2023, 1:20:28 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:28.557274032+01:00 [Nest] 8  - 12/07/2023, 1:20:28 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:28.557302300+01:00 [Nest] 8  - 12/07/2023, 1:20:28 AM   ERROR [JobService] Object:
2023-12-07T01:20:28.557306036+01:00 {
2023-12-07T01:20:28.557308866+01:00   "id": "121bd39e-a533-4904-bc6e-952ccf2dbf20"
2023-12-07T01:20:28.557311636+01:00 }
2023-12-07T01:20:28.557321404+01:00 
[Nest] 8  - 12/07/2023, 1:20:29 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: ENOENT: no such file or directory, open 'upload/thumbs/8d9dc17b-e5b6-43de-8ca1-0c5e77e220e9/32/de/32defc5e-eb70-4e44-8b82-351292c01c15.jpeg'
2023-12-07T01:20:29.211682708+01:00 [Nest] 8  - 12/07/2023, 1:20:29 AM   ERROR [JobService] Error: ENOENT: no such file or directory, open 'upload/thumbs/8d9dc17b-e5b6-43de-8ca1-0c5e77e220e9/32/de/32defc5e-eb70-4e44-8b82-351292c01c15.jpeg'
[Nest] 8  - 12/07/2023, 1:20:29 AM   ERROR [JobService] Object:
2023-12-07T01:20:29.211700182+01:00 {
2023-12-07T01:20:29.211705678+01:00   "id": "32defc5e-eb70-4e44-8b82-351292c01c15"
}
2023-12-07T01:20:29.211710571+01:00 
[Nest] 8  - 12/07/2023, 1:20:29 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
[Nest] 8  - 12/07/2023, 1:20:29 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:29.551684124+01:00 [Nest] 8  - 12/07/2023, 1:20:29 AM   ERROR [JobService] Object:
{
  "id": "afde8cc6-ec3b-43f6-a7ca-88f6907735af"
}
2023-12-07T01:20:29.551696106+01:00 
[Nest] 8  - 12/07/2023, 1:20:31 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:31.210023457+01:00 [Nest] 8  - 12/07/2023, 1:20:31 AM   ERROR [JobService] Error: Input file contains unsupported image format
[Nest] 8  - 12/07/2023, 1:20:31 AM   ERROR [JobService] Object:
2023-12-07T01:20:31.210072101+01:00 {
  "id": "cc93ee60-93db-4605-821f-9202af3a70ac"
}
2023-12-07T01:20:31.210079648+01:00 
2023-12-07T01:20:31.210386908+01:00 [Nest] 8  - 12/07/2023, 1:20:31 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:31.210395554+01:00 [Nest] 8  - 12/07/2023, 1:20:31 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:31.210418025+01:00 [Nest] 8  - 12/07/2023, 1:20:31 AM   ERROR [JobService] Object:
2023-12-07T01:20:31.210421587+01:00 {
2023-12-07T01:20:31.210424270+01:00   "id": "945628b8-5268-4039-a422-466f3a6a1ae3"
2023-12-07T01:20:31.210426767+01:00 }
2023-12-07T01:20:31.210429148+01:00 
2023-12-07T01:20:31.210667931+01:00 [Nest] 8  - 12/07/2023, 1:20:31 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:31.210712347+01:00 [Nest] 8  - 12/07/2023, 1:20:31 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:31.210722899+01:00 [Nest] 8  - 12/07/2023, 1:20:31 AM   ERROR [JobService] Object:
2023-12-07T01:20:31.210725610+01:00 {
2023-12-07T01:20:31.210728319+01:00   "id": "6d9518b3-723c-4a27-bd53-6a7089ad91d0"
2023-12-07T01:20:31.210730747+01:00 }
2023-12-07T01:20:31.210732998+01:00 
[Nest] 8  - 12/07/2023, 1:20:33 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:33.246186619+01:00 [Nest] 8  - 12/07/2023, 1:20:33 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:33.246230145+01:00 [Nest] 8  - 12/07/2023, 1:20:33 AM   ERROR [JobService] Object:
2023-12-07T01:20:33.246239909+01:00 {
2023-12-07T01:20:33.246244159+01:00   "id": "942d3736-b03d-42c3-9ac9-ca61824a68f9"
2023-12-07T01:20:33.246249255+01:00 }
2023-12-07T01:20:33.246251775+01:00 
[Nest] 8  - 12/07/2023, 1:20:34 AM   ERROR [JobService] Unable to run job handler (objectTagging/classify-image): Error: Request for image classification failed with status 500: Internal Server Error
2023-12-07T01:20:34.736497196+01:00 [Nest] 8  - 12/07/2023, 1:20:34 AM   ERROR [JobService] Error: Request for image classification failed with status 500: Internal Server Error
    at MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:18:19)
2023-12-07T01:20:34.736505512+01:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-12-07T01:20:34.736508407+01:00     at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:55:22)
2023-12-07T01:20:34.736511293+01:00     at async /usr/src/app/dist/domain/job/job.service.js:108:37
2023-12-07T01:20:34.736514709+01:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)
2023-12-07T01:20:34.736533233+01:00 [Nest] 8  - 12/07/2023, 1:20:34 AM   ERROR [JobService] Object:
2023-12-07T01:20:34.736537005+01:00 {
2023-12-07T01:20:34.736540225+01:00   "id": "1c4922cd-9b80-4535-9cd6-e97c603332c0"
}
2023-12-07T01:20:34.736545961+01:00 
[Nest] 8  - 12/07/2023, 1:20:34 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
[Nest] 8  - 12/07/2023, 1:20:34 AM   ERROR [JobService] Error: Input file contains unsupported image format
[Nest] 8  - 12/07/2023, 1:20:34 AM   ERROR [JobService] Object:
{
  "id": "13b7073a-fba5-4d4d-b326-afda79568091"
2023-12-07T01:20:34.888181575+01:00 }

[Nest] 8  - 12/07/2023, 1:20:36 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:36.699397525+01:00 [Nest] 8  - 12/07/2023, 1:20:36 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:36.699408447+01:00 [Nest] 8  - 12/07/2023, 1:20:36 AM   ERROR [JobService] Object:
2023-12-07T01:20:36.699411872+01:00 {
2023-12-07T01:20:36.699414934+01:00   "id": "46c06e3a-a4b6-4b04-ab2d-4a22f82194f1"
2023-12-07T01:20:36.699417559+01:00 }
2023-12-07T01:20:36.699419852+01:00 
2023-12-07T01:20:36.699735081+01:00 [Nest] 8  - 12/07/2023, 1:20:36 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:36.699745249+01:00 [Nest] 8  - 12/07/2023, 1:20:36 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:36.699786017+01:00 [Nest] 8  - 12/07/2023, 1:20:36 AM   ERROR [JobService] Object:
2023-12-07T01:20:36.699790271+01:00 {
2023-12-07T01:20:36.699793105+01:00   "id": "fd768686-687e-466d-bf28-2a3735339c1b"
2023-12-07T01:20:36.699795729+01:00 }
2023-12-07T01:20:36.699798071+01:00 
[Nest] 8  - 12/07/2023, 1:20:38 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:38.441459786+01:00 [Nest] 8  - 12/07/2023, 1:20:38 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:38.441463960+01:00 [Nest] 8  - 12/07/2023, 1:20:38 AM   ERROR [JobService] Object:
2023-12-07T01:20:38.441466685+01:00 {
2023-12-07T01:20:38.441472714+01:00   "id": "32f7cbbe-a95a-40d4-8cca-dd86f7e0f664"
2023-12-07T01:20:38.441475390+01:00 }
2023-12-07T01:20:38.441477783+01:00 
[Nest] 8  - 12/07/2023, 1:20:41 AM   ERROR [JobService] Unable to run job handler (recognizeFaces/recognize-faces): Error: Request for facial recognition failed with status 500: Internal Server Error
[Nest] 8  - 12/07/2023, 1:20:41 AM   ERROR [JobService] Error: Request for facial recognition failed with status 500: Internal Server Error
2023-12-07T01:20:41.281239728+01:00     at MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:18:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-12-07T01:20:41.281254518+01:00     at async PersonService.handleRecognizeFaces (/usr/src/app/dist/domain/person/person.service.js:183:23)
2023-12-07T01:20:41.281257517+01:00     at async /usr/src/app/dist/domain/job/job.service.js:108:37
2023-12-07T01:20:41.281259903+01:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:387:28)
2023-12-07T01:20:41.281262358+01:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:574:24)
[Nest] 8  - 12/07/2023, 1:20:41 AM   ERROR [JobService] Object:
2023-12-07T01:20:41.281272523+01:00 {
2023-12-07T01:20:41.281275179+01:00   "id": "179a14e8-737c-4965-80d3-92e82f9e86b6"
2023-12-07T01:20:41.281277632+01:00 }
2023-12-07T01:20:41.281279929+01:00 
[Nest] 8  - 12/07/2023, 1:20:44 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:44.249823558+01:00 [Nest] 8  - 12/07/2023, 1:20:44 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:44.249828734+01:00 [Nest] 8  - 12/07/2023, 1:20:44 AM   ERROR [JobService] Object:
2023-12-07T01:20:44.249831786+01:00 {
2023-12-07T01:20:44.249834464+01:00   "id": "e93e8315-b9b2-4ebe-96f3-6639fe0967f9"
2023-12-07T01:20:44.249837031+01:00 }
2023-12-07T01:20:44.249839142+01:00 
2023-12-07T01:20:44.249842121+01:00 [Nest] 8  - 12/07/2023, 1:20:44 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:44.249844671+01:00 [Nest] 8  - 12/07/2023, 1:20:44 AM   ERROR [JobService] Error: Input file contains unsupported image format
[Nest] 8  - 12/07/2023, 1:20:44 AM   ERROR [JobService] Object:
2023-12-07T01:20:44.249849961+01:00 {
2023-12-07T01:20:44.249852540+01:00   "id": "8bc9eb52-f14d-48ca-9bad-6743b8c6e03f"
}
2023-12-07T01:20:44.249857374+01:00 
[Nest] 8  - 12/07/2023, 1:20:46 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:46.771537153+01:00 [Nest] 8  - 12/07/2023, 1:20:46 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:46.771542624+01:00 [Nest] 8  - 12/07/2023, 1:20:46 AM   ERROR [JobService] Object:
2023-12-07T01:20:46.771546154+01:00 {
2023-12-07T01:20:46.771553757+01:00   "id": "c6435c78-e2ed-41bf-b5a8-d3dc566724f4"
2023-12-07T01:20:46.771557374+01:00 }
2023-12-07T01:20:46.771560287+01:00 
[Nest] 8  - 12/07/2023, 1:20:47 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:47.497787134+01:00 [Nest] 8  - 12/07/2023, 1:20:47 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:47.497791891+01:00 [Nest] 8  - 12/07/2023, 1:20:47 AM   ERROR [JobService] Object:
2023-12-07T01:20:47.497794657+01:00 {
2023-12-07T01:20:47.497801383+01:00   "id": "af698c6c-6217-4ac7-adfa-3954f01eda70"
2023-12-07T01:20:47.497804314+01:00 }
2023-12-07T01:20:47.497806870+01:00 
[Nest] 8  - 12/07/2023, 1:20:54 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:20:54.120126931+01:00 [Nest] 8  - 12/07/2023, 1:20:54 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:54.120129702+01:00 [Nest] 8  - 12/07/2023, 1:20:54 AM   ERROR [JobService] Object:
2023-12-07T01:20:54.120131506+01:00 {
  "id": "bca65953-a778-4e42-9f01-5ecf755d4590"
}
2023-12-07T01:20:54.120140544+01:00 
[Nest] 8  - 12/07/2023, 1:20:54 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
[Nest] 8  - 12/07/2023, 1:20:54 AM   ERROR [JobService] Error: Input file contains unsupported image format
2023-12-07T01:20:54.120469132+01:00 [Nest] 8  - 12/07/2023, 1:20:54 AM   ERROR [JobService] Object:
2023-12-07T01:20:54.120473665+01:00 {
  "id": "84fdc30a-d3a4-428c-8a38-04019637d38f"
}

[Nest] 8  - 12/07/2023, 1:20:54 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
[Nest] 8  - 12/07/2023, 1:20:54 AM   ERROR [JobService] Error: Input file contains unsupported image format
[Nest] 8  - 12/07/2023, 1:20:54 AM   ERROR [JobService] Object:
2023-12-07T01:20:54.120728170+01:00 {
2023-12-07T01:20:54.120731639+01:00   "id": "839cba2a-bcc7-473d-940c-c1a7439d3fb3"
}
2023-12-07T01:20:54.120736931+01:00 
[Nest] 8  - 12/07/2023, 1:21:00 AM   ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
2023-12-07T01:21:00.935995601+01:00 [Nest] 8  - 12/07/2023, 1:21:00 AM   ERROR [JobService] Error: Input file contains unsupported image format
[Nest] 8  - 12/07/2023, 1:21:00 AM   ERROR [JobService] Object:
2023-12-07T01:21:00.936330151+01:00 {
yodatak commented 9 months ago

I can provides bad asset for reproduce the error if wanted

grishy commented 9 months ago

I have a same issue in docker and 1.91.4

Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: Input file contains unsupported image format
alextran1502 commented 9 months ago

@grishy Do you know the original file type?

grishy commented 9 months ago

@alextran1502 I found that this is not related to version/config (k8s or docker). It happens when you have a photo that was corrupted during upload. It can be any format. I have it happen with JPEG or HEIC from my iPhone. I imported photos from Google Photos and this time it was broken. In the UI it looks like this.

image

If you download a photo from the UI, you won't be able to open it locally.

image
grishy commented 9 months ago

@alextran1502 do we have an API to get and remove all the broken photos? :) I want to reload this from a google backup.

alextran1502 commented 9 months ago

@grishy we don't unfortunately

grishy commented 9 months ago

@yodatak @alextran1502

Small snippet that will show all broken images.

  1. Generate all thumbnails via Job;
  2. Start script, before change base_addr and set env. API_KEY;
import requests
import os

baseaddr = "https://immich.lab.local"
url = f"{baseaddr}/api/asset"

headers = {
    "Accept": "application/json",
    "x-api-key": os.environ["API_KEY"],
}

response = requests.request("GET", url, headers=headers, verify=False).json()
without_thumbhash = [x for x in response if x["thumbhash"] is None]

for asset in without_thumbhash:
    print(f"id='{asset['id']}' fileCreatedAt={asset['fileCreatedAt']} name='{asset['originalFileName']}'")

print(f"Total: {len(without_thumbhash)}")

After that you can remove them by API or manually.

grishy commented 9 months ago

upd: Deleting broken images and re-uploading them eliminated the error. version 1.91.4

mfuellbier commented 9 months ago

How did you delete the corrupted images? Manually on the server or within the application? I am struggling with the same error and deleting the pictures in the app doe snot help.

grishy commented 9 months ago

@mfuellbier via API. I made something like this

import requests
import os
import json
from tqdm import tqdm

baseaddr = "https://immich.lab.local"

# Disable SSL warnings
import urllib3

urllib3.disable_warnings()

#
# Get all assets without thumbhash
url = f"{baseaddr}/api/asset"

headers = {
    "Accept": "application/json",
    "x-api-key": os.environ["API_KEY"],
}

print("Getting assets...")
response = requests.request("GET", url, headers=headers, verify=False).json()
without_thumbhash = [x for x in response if x["thumbhash"] is None]

for asset in without_thumbhash:
    print(
        f"id='{asset['id']}' fileCreatedAt={asset['fileCreatedAt']} name='{asset['originalFileName']}'"
    )

print(f"Total: {len(without_thumbhash)}")

#
# Download all assets without thumbhash
if False:
    print()
    print("=" * 80)
    print()

    url = f"{baseaddr}/api/asset/download/archive"
    payload = json.dumps({"assetIds": [x["id"] for x in without_thumbhash]})
    headers = {
        "Content-Type": "application/json",
        "Accept": "application/octet-stream",
        "x-api-key": os.environ["API_KEY"],
    }

    print("Downloading...")
    response = requests.request(
        "POST", url, headers=headers, data=payload, verify=False, stream=True
    )

    print("Headers")
    print(response.headers)

    total_size_in_bytes = int(response.headers.get("content-length", 0))
    block_size = 1024  # 1 Kibibyte
    progress_bar = tqdm(total=total_size_in_bytes, unit="iB", unit_scale=True)
    with open("backup.zip", "wb") as file:
        for data in response.iter_content(block_size):
            progress_bar.update(len(data))
            file.write(data)
    progress_bar.close()
    if total_size_in_bytes != 0 and progress_bar.n != total_size_in_bytes:
        print("ERROR, something went wrong")

#
# Delete all assets without thumbhash in Immich
if False:
    print()
    print("=" * 80)
    print()

    url = f"{baseaddr}/api/asset"
    payload = json.dumps({"ids": [x["id"] for x in without_thumbhash]})
    headers = {
        "Content-Type": "application/json",
        "x-api-key": os.environ["API_KEY"],
    }

    print("Deleting...")
    response = requests.request(
        "DELETE", url, headers=headers, data=payload, verify=False
    )
    print(response.text)
mfuellbier commented 8 months ago

Thank you. I tried deleting the files with you script. Deletion worked well, but I cannot reupload the deleted images anymore. Immich says the images would be duplicates. Besides that, I am still not able to upload new images. The logs still show the error "Input file contains unsupported image format". Do you have any idea what the issue might be here?

grishy commented 8 months ago

@mfuellbier idk, I used the API way to upload (immich-go tool). Works pretty well. and fixed all. P.S. did you uncomment if False:?

jrasm91 commented 2 months ago

I believe the original issue (microservices crashing) has been fixed since this issue was logged. Happy to re-open it if it is still a problem. For other issues/troubleshooting feel free to open a new discussion (q&a)