search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
45.08k stars 2.18k forks source link

[BUG] 504 errors importing videos with CLI using --import #4013

Closed yodatak closed 10 months ago

yodatak commented 1 year ago

The bug

in immich-microservices

2023-09-07T16:46:34.760277344+02:00 [Nest] 8  - 09/07/2023, 4:46:34 PM   ERROR [JobService] Unable to run job handler: TypeError: Cannot read properties of null (reading 'to')
2023-09-07T16:46:34.760442799+02:00 [Nest] 8  - 09/07/2023, 4:46:34 PM   ERROR [JobService] TypeError: Cannot read properties of null (reading 'to')
2023-09-07T16:46:34.760450096+02:00     at CommunicationRepository.send (/usr/src/app/dist/infra/repositories/communication.repository.js:20:24)
2023-09-07T16:46:34.760453473+02:00     at JobService.onDone (/usr/src/app/dist/domain/job/job.service.js:164:50)
2023-09-07T16:46:34.760456347+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:46:34.760459978+02:00     at async /usr/src/app/dist/domain/job/job.service.js:107:25
2023-09-07T16:46:34.760462728+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-09-07T16:46:34.760465157+02:00     at async Work
[Nest] 8  - 09/07/2023, 4:48:44 PM   ERROR [JobService] Unable to run job handler: Error: Connection terminated due to connection timeout
[Nest] 8  - 09/07/2023, 4:48:44 PM   ERROR [JobService] Error: Connection terminated due to connection timeout
2023-09-07T16:48:44.470777854+02:00     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
2023-09-07T16:48:44.470781573+02:00     at Object.onceWrapper (node:events:628:28)
2023-09-07T16:48:44.470784056+02:00     at Connection.emit (node:events:514:28)
2023-09-07T16:48:44.470786605+02:00     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
2023-09-07T16:48:44.470789076+02:00     at Socket.emit (node:events:514:28)
2023-09-07T16:48:44.470791493+02:00     at TCP.<anonymous> (node:net:323:12)
[Nest] 8  - 09/07/2023, 4:48:44 PM   ERROR [JobService] Object:
2023-09-07T16:48:44.471332649+02:00 {
2023-09-07T16:48:44.471335870+02:00   "id": "1810065b-7f56-418b-ae57-3162ecdb3f33",
2023-09-07T16:48:44.471338484+02:00   "source": "upload"
2023-09-07T16:48:44.471341241+02:00 }
2023-09-07T16:48:44.471343509+02:00 
[Nest] 8  - 09/07/2023, 4:48:45 PM   ERROR [JobService] Unable to run job handler: Error: Connection terminated due to connection timeout
2023-09-07T16:48:45.551578848+02:00 [Nest] 8  - 09/07/2023, 4:48:45 PM   ERROR [JobService] Error: Connection terminated due to connection timeout
2023-09-07T16:48:45.551586947+02:00     at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
2023-09-07T16:48:45.551589740+02:00     at Object.onceWrapper (node:events:628:28)
2023-09-07T16:48:45.551592245+02:00     at Connection.emit (node:events:514:28)
2023-09-07T16:48:45.551594918+02:00     at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:63:12)
2023-09-07T16:48:45.551597387+02:00     at Socket.emit (node:events:514:28)
2023-09-07T16:48:45.551599834+02:00     at TCP.<anonymous> (node:net:323:12)
2023-09-07T16:48:45.551602460+02:00 [Nest] 8  - 09/07/2023, 4:48:45 PM   ERROR [JobService] Object:
2023-09-07T16:48:45.551604977+02:00 {
2023-09-07T16:48:45.551607547+02:00   "id": "ef3ed216-dea7-429e-910f-35ec4eb9fd20",
2023-09-07T16:48:45.551610294+02:00   "source": "upload"
2023-09-07T16:48:45.551612738+02:00 }
2023-09-07T16:48:45.551615079+02:00 
[Nest] 8  - 09/07/2023, 4:48:47 PM   ERROR [JobService] Unable to run job handler: TypeError: fetch failed
[Nest] 8  - 09/07/2023, 4:48:47 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11576:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
    at async SmartInfoService.handleClassifyImage (/usr/src/app/dist/domain/smart-info/smart-info.service.js:77:22)
    at async /usr/src/app/dist/domain/job/job.service.js:105:37
    at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
    at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
[Nest] 8  - 09/07/2023, 4:48:47 PM   ERROR [JobService] Object:
{
  "id": "56e660e8-238a-4349-af42-2f75521648dc",
  "source": "upload"
}

[Nest] 8  - 09/07/2023, 4:48:47 PM   ERROR [JobService] Unable to run job handler: TypeError: fetch failed
[Nest] 8  - 09/07/2023, 4:48:47 PM   ERROR [JobService] TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11576:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MachineLearningRepository.post (/usr/src/app/dist/infra/repositories/machine-learning.repository.js:27:21)
    at async FacialRecognitionService.handleRecognizeFaces (/usr/src/app/dist/domain/facial-recognition/facial-recognition.services.js:105:23)
2023-09-07T16:48:47.267395051+02:00     at async /usr/src/app/dist/domain/job/job.service.js:105:37
2023-09-07T16:48:47.267405270+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-09-07T16:48:47.267429326+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)
2023-09-07T16:48:47.267439507+02:00 [Nest] 8  - 09/07/2023, 4:48:47 PM   ERROR [JobService] Object:
2023-09-07T16:48:47.267448981+02:00 {
2023-09-07T16:48:47.267458765+02:00   "id": "1691264c-94dd-4d3f-b25d-ce29a91a2a1e",
2023-09-07T16:48:47.267468597+02:00   "source": "upload"
2023-09-07T16:48:47.267478003+02:00 }
2023-09-07T16:48:47.267486760+02:00 
[Nest] 8  - 09/07/2023, 4:48:50 PM     LOG [MediaService] Successfully generated image thumbnail 5b9612f8-b89d-4526-8cd0-17787a48866b
[Nest] 8  - 09/07/2023, 4:48:50 PM   ERROR [JobService] Unable to run job handler: TypeError: Cannot read properties of null (reading 'to')
[Nest] 8  - 09/07/2023, 4:48:50 PM   ERROR [JobService] TypeError: Cannot read properties of null (reading 'to')
2023-09-07T16:48:50.857432731+02:00     at CommunicationRepository.send (/usr/src/app/dist/infra/repositories/communication.repository.js:20:24)
2023-09-07T16:48:50.857436489+02:00     at JobService.onDone (/usr/src/app/dist/domain/job/job.service.js:164:50)
2023-09-07T16:48:50.857438951+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:48:50.857441504+02:00     at async /usr/src/app/dist/domain/job/job.service.js:107:25
2023-09-07T16:48:50.857443889+02:00     at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:346:28)
2023-09-07T16:48:50.857446167+02:00     at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:531:24)

I also goterror in search tab in web 500 error

in immich server

2023-09-07T16:45:49.881940765+02:00 [Nest] 7  - 09/07/2023, 4:45:49 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:45:49.882261150+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:45:49.882270180+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:45:49.882274754+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:45:49.882277310+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:45:49.882279386+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:45:49.882281386+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:45:49.882283935+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:45:49.882285941+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:45:49.882288266+02:00     at Object.next (/usr/src/ap
p/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:45:49.882290528+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:45:49.882292452+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:45:49.991596213+02:00 [Nest] 7  - 09/07/2023, 4:45:49 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:45:49.991687056+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:45:49.991693086+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:45:49.991695052+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:45:49.991696661+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:45:49.991698067+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:45:49.991699431+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:45:49.991701057+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:45:49.991702431+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
    at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
    at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[Nest] 7  - 09/07/2023, 4:45:55 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:45:55.015817553+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:45:55.015820320+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:45:55.015823095+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:45:55.015825871+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:45:55.015828015+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:45:55.015830852+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:45:55.015833377+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:45:55.015836364+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:45:55.015838592+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:45:55.015841214+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:46:00.458230458+02:00 [Nest] 7  - 09/07/2023, 4:46:00 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:46:00.458252527+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:46:00.458255335+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:46:00.458257644+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-
07T16:46:00.458259687+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:46:00.458261842+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:46:00.458264165+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:46:00.458266391+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:46:00.458268392+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:46:00.458270328+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:46:00.458272424+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:46:00.458274496+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:46:12.218016163+02:00 [Nest] 7  - 09/07/2023, 4:46:12 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:46:12.218047522+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:46:12.218050928+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:46:12.218053146+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:46:12.218057615+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:46:12.218059798+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:46:12.218061658+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:46:12.218063630+02:00     at Object.next (/usr/src/app/node_
modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:46:12.218065509+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:46:12.218067335+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:46:12.218069140+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:46:12.218070975+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:46:12.599054790+02:00 [Nest] 7  - 09/07/2023, 4:46:12 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:46:12.599076786+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:46:12.599079762+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:46:12.599082027+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:46:12.599083892+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:46:12.599085793+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:46:12.599087629+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:46:12.599089901+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:46:12.599091948+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:46:12.599094087+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:46:12.599096188+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:46:12.59909
8065+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:48:35.701930921+02:00 [Nest] 7  - 09/07/2023, 4:48:35 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:35.701962834+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:35.701965546+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:48:35.701967771+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:48:35.701974384+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:48:35.701984479+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:48:35.701986839+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:48:35.701992892+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:35.701995201+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:48:35.701997417+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:35.701999483+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:48:35.702002124+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:48:37.783875036+02:00 [Nest] 7  - 09/07/2023, 4:48:37 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:37.783899772+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:37.783903284+
02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:48:37.783906232+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:48:37.783910871+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:48:37.783913160+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:48:37.783915242+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:48:37.783917783+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:37.783920207+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:48:37.783922453+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:37.783924333+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:48:37.783926257+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:48:37.934863402+02:00 [Nest] 7  - 09/07/2023, 4:48:37 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:37.937045849+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:37.937062731+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:48:37.937065249+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:48:37.937067365+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-0
9-07T16:48:37.937069747+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:48:37.937071821+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:48:37.937074207+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:37.937076433+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:48:37.937078538+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:37.937080707+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:48:37.937090936+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[Nest] 7  - 09/07/2023, 4:48:38 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
    at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
    at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:48:38.048257787+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
    at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:48:38.048261662+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
    at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:38.048268787+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:48:38.048270735+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:38.048272606+02:00     at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:48:38.048274870+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2023-09-07T16:48:38.185808219+02:00 [Nest] 7  - 09/07/2023, 4:48:38 PM   ERROR [ExceptionsHandler] Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:38.185883567+02:00 ObjectNotFound: Request failed with HTTP code 404 | Server said: Not found.
2023-09-07T16:48:38.185890507+02:00     at ObjectNotFound.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
2023-09-07T16:48:38.185892978+02:00     at new ObjectNotFound (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ObjectNotFound.js:25:42)
2023-09-07T16:48:38.185895142+02:00     at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:367:21)
2023-09-07T16:48:38.185897145+02:00     at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:220:98
2023-09-07T16:48:38.185899122+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
2023-09-07T16:48:38.185901158+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:38.185902987+02:00     at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
2023-09-07T16:48:38.185904874+02:00     at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
2023-09-07T16:48:38.185906913+02:00     at fulfilled (/usr/src/app/node_
modules/typesense/lib/Typesense/ApiCall.js:5:58)
2023-09-07T16:48:38.185908934+02:00     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

The OS that Immich Server is running on

Fedora k3s

Version of Immich Server

v1.76.1

Version of Immich Mobile App

non used

Platform with the issue

Your docker-compose.yml content

i use k3s

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: immich
  namespace: immich
spec:
  chart:
    spec:
      chart: immich
      interval: 5m
      reconcileStrategy: ChartVersion
      sourceRef:
        kind: HelmRepository
        name: truecharts
        namespace: flux-system
      version: 8.0.8
  interval: 5m
  values:
    TZ: Europe/Paris

    operator:
      register: false
      verify:
        enabled: false
        failOnError: false
    containerOptions:
      NVIDIA_CAPS:
        - all

    # -- Allow specifying a runtimeClassName other than the default one (ie: nvidia)
    runtimeClassName: nvidia

    securityContext:
      container:
        PUID: 1000
        UMASK: '0022'
        runAsGroup: 100
        runAsUser: 1000
      pod:
        fsGroup: 100
        fsGroupChangePolicy: OnRootMismatch

    immich:
      enable_ml: true
      enable_typesense: true
      public_login_page_message: Immich
      disable_reverse_geocoding: false
      reverse_geocoding_precision: 3
      log_level: log

    persistence:
      library:
        enabled: true
        mountPath: /usr/src/app/upload/library
        hostPath: /mnt/XXXX/immich/library
        hostPathType: Directory
        type: hostPath
      uploads:
        enabled: true
        mountPath: /usr/src/app/upload/upload
        hostPath: /mnt/XXXX/immich/upload
        hostPathType: Directory
        type: hostPath
      thumbs:
        enabled: true
        mountPath: /usr/src/app/upload/thumbs
        hostPath: /mnt/XXXX/immich/thumbs
        hostPathType: Directory
        type: hostPath
      profile:
        enabled: true
        mountPath: /usr/src/app/upload/profile
        hostPath: /mnt/XXXX/immich/profile
        hostPathType: Directory
        type: hostPath
      video:
        enabled: true
        mountPath: /usr/src/app/upload/encoded-video
        hostPath: /mnt/XXXX/immich/encoded-video
        hostPathType: Directory
        type: hostPath
      originals-photos:
        enabled: true
        hostPath: /mnt/XXXXX/Photos Videos
        hostPathType: Directory
        mountPath: /mnt/XXXX/Photos Videos
        readOnly: true
        targetSelectAll: true
        type: hostPath

    ingress:
      main:
        enabled: true
        ingressClassName: traefik
        enableFixedMiddlewares: false
        hosts:
          - host: XXXX
            paths:
              - path: /
                pathType: Prefix
        tls:
          - hosts:
              - XXXX
            secretName: photos-tls-secret

Your .env content

none

Reproduction steps

1. i do a big importing of existing readonly galery
2. i some time latter i got this

Additional information

No response

bo0tzz commented 1 year ago

What do the logs of your typesense container say? It seems like that might be overloaded.

yodatak commented 1 year ago

Thanks for help some more logs

{
  "job": {
    "thumbnailGeneration": {
      "concurrency": 8
    },
    "metadataExtraction": {
      "concurrency": 8
    },
    "objectTagging": {
      "concurrency": 4
    },
    "recognizeFaces": {
      "concurrency": 4
    },
    "storageTemplateMigration": {
      "concurrency": 8
    },
    "sidecar": {
      "concurrency": 8
    },
    "backgroundTask": {
      "concurrency": 8
    },
    "clipEncoding": {
      "concurrency": 2
    },
    "search": {
      "concurrency": 8
    },
    "videoConversion": {
      "concurrency": 2
    }
  },
  "ffmpeg": {
    "crf": 23,
    "threads": 0,
    "preset": "ultrafast",
    "targetVideoCodec": "h264",
    "targetAudioCodec": "aac",
    "targetResolution": "720",
    "maxBitrate": "0",
    "twoPass": false,
    "transcode": "required",
    "tonemap": "hable",
    "accel": "disabled"
  },
  "machineLearning": {
    "enabled": true,
    "url": "http://immich-machinelearning:10003",
    "classification": {
      "enabled": true,
      "modelName": "microsoft/resnet-50",
      "minScore": 0.9
    },
    "clip": {
      "enabled": true,
      "modelName": "ViT-B-32::openai"
    },
    "facialRecognition": {
      "enabled": true,
      "modelName": "buffalo_l",
      "minScore": 0.7,
      "maxDistance": 0.6
    }
  },
  "oauth": {
    "enabled": false,
    "issuerUrl": "",
    "clientId": "",
    "clientSecret": "",
    "mobileOverrideEnabled": false,
    "mobileRedirectUri": "",
    "scope": "openid email profile",
    "storageLabelClaim": "preferred_username",
    "buttonText": "Login with OAuth",
    "autoRegister": true,
    "autoLaunch": false
  },
  "passwordLogin": {
    "enabled": true
  },
  "storageTemplate": {
    "template": "{{y}}/{{y}}-{{MM}}-{{dd}}/{{filename}}"
  },
  "thumbnail": {
    "webpSize": 250,
    "jpegSize": 1440
  }
}

the log i got from immich import

node@immich-688d755b49-wr8jc:/usr/src/app$ immich upload --key XXXX --server XXX  "/mnt/XXXXX/XXXX" -y --recursive -t 10 --import
Checking connectivity with Immich instance...
Server status: OK
Checking credentials...
Login status: OK
Successful authentication for user XXXXX
Indexing local assets...
Indexing complete, found 327529 local assets
Comparing local assets with those on the Immich instance...
A total of 38675 assets will be uploaded to the server
Start uploading...
Upload Progress | █████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 11% || 38675/327529 || Current file [/mnt/XX 2/XXX.CR2]
Error asset:  {
  file: '/mnt/XXX/2022/XXXXX.mp4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'https://immich.XXX/api/asset/import',
      data: '{"deviceAssetId":"XXXXX.mp4-1933807498","deviceId":"CLI","fileCreatedAt":"2022-04-22T16:20:52.000Z","fileModifiedAt":"2022-04-22T16:20:52.000Z","isFavorite":"false","fileExtension":".mp4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/2022/Baptiste Solidaires Informatique Urkaine.mp4"}'
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 362,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'POST /api/asset/import HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Content-Length: 362\r\n' +
        'Host: XXXX.immich.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/import',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'XXXX.immich.com',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Timeout',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
Error asset:  {
  file: '/mnt/XXX/Montage/pizza pino/canon eos/MVI_0538.MOV',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'https://immich.XXX/api/asset/import',
      data: '{"deviceAssetId":"MVI_0538.MOV-2716295569","deviceId":"CLI","fileCreatedAt":"2018-06-02T05:21:26.000Z","fileModifiedAt":"2018-06-02T05:21:26.000Z","isFavorite":"false","fileExtension":".MOV","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/Montage/pizza pino/canon eos/MVI_0538.MOV"}'
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 325,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'POST /api/asset/import HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Content-Length: 325\r\n' +
        'Host: XXXX.immich.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/import',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'XXXX.immich.com',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Timeout',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
Error asset:  {
  file: '/mnt/XXX/XXXX/C0012.MP4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'https://immich.XXX/api/asset/import',
      data: '{"deviceAssetId":"C0012.MP4-3674651378","deviceId":"CLI","fileCreatedAt":"2021-07-15T12:12:16.000Z","fileModifiedAt":"2021-07-15T12:12:16.000Z","isFavorite":"false","fileExtension":".MP4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/XXXX/C0012.MP4"}'
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 322,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'POST /api/asset/import HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Content-Length: 322\r\n' +
        'Host: XXXX.immich.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/import',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'XXXX.immich.com',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Timeout',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
Error asset:  {
  file: '/mnt/XXX/XXXX/C0013.MP4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'https://immich.XXX/api/asset/import',
      data: '{"deviceAssetId":"C0013.MP4-2936365794","deviceId":"CLI","fileCreatedAt":"2021-07-15T12:22:20.000Z","fileModifiedAt":"2021-07-15T12:22:20.000Z","isFavorite":"false","fileExtension":".MP4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/XXXX/C0013.MP4"}'
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 322,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'POST /api/asset/import HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Content-Length: 322\r\n' +
        'Host: XXXX.immich.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/import',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'XXXX.immich.com',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Timeout',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
Error asset:  {
  file: '/mnt/XXX/XXXX/C0014.MP4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'https://immich.XXX/api/asset/import',
      data: '{"deviceAssetId":"C0014.MP4-2978313974","deviceId":"CLI","fileCreatedAt":"2021-07-15T12:31:35.000Z","fileModifiedAt":"2021-07-15T12:31:35.000Z","isFavorite":"false","fileExtension":".MP4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/XXXX/C0014.MP4"}'
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 322,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'POST /api/asset/import HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Content-Length: 322\r\n' +
        'Host: XXXX.immich.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/import',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'XXXX.immich.com',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Timeout',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
Error asset:  {
  file: '/mnt/XXX/XXXX/C0015.MP4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'https://immich.XXX/api/asset/import',
      data: '{"deviceAssetId":"C0015.MP4-2646923586","deviceId":"CLI","fileCreatedAt":"2021-07-15T12:35:37.000Z","fileModifiedAt":"2021-07-15T12:35:37.000Z","isFavorite":"false","fileExtension":".MP4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXX/XXXX/C0015.MP4"}'
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 322,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'POST /api/asset/import HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Content-Length: 322\r\n' +
        'Host: XXXX.immich.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/import',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'XXXX.immich.com',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Timeout',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}

and typesence

I20230907 16:29:49.203409   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:29:49.203681   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:31:59.213184129+02:00 I20230907 16:29:52.136260   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:31:59.213185598+02:00 I20230907 16:29:59.204329   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:29:59.204372   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:30:09.205035   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:31:59.213189908+02:00 I20230907 16:30:09.205098   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:30:19.205725   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:31:59.213192823+02:00 I20230907 16:30:19.205758   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:30:29.206576   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:30:29.206611   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:30:39.207240   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:31:59.213198554+02:00 I20230907 16:30:39.207276   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:31:59.213199977+02:00 I20230907 16:30:49.207926   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:31:59.213201404+02:00 I20230907 16:30:49.207965   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:30:53.143424   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:31:59.213209880+02:00 I20230907 16:30:59.208699   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:30:59.208735   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:31:09.209383   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:31:09.209440   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:31:19.210069   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:31:19.210103   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:31:29.210781   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:31:29.210815   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:31:59.213221532+02:00 I20230907 16:31:39.211509   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:31:59.213222980+02:00 I20230907 16:31:39.211549   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:31:59.213224400+02:00 I20230907 16:31:49.212189   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:31:59.213225833+02:00 I20230907 16:31:49.212226   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:31:59.213231839+02:00 I20230907 16:31:54.147421   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:31:59.212867   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:31:59.213135   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:32:09.213796   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:34:09.223465706+02:00 I20230907 16:32:09.213843   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:32:19.214514   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:32:19.214550   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:34:09.223471535+02:00 I20230907 16:32:29.215185   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:32:29.215224   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:32:39.215909   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:32:39.215966   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:32:49.216591   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:34:09.223486831+02:00 I20230907 16:32:49.216634   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:32:55.153040   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:34:09.223490478+02:00 I20230907 16:32:59.217676   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:32:59.217723   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:33:09.218418   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:33:09.218482   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:34:09.223497465+02:00 I20230907 16:33:19.219158   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:33:19.219189   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:33:29.219866   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:34:09.223506681+02:00 I20230907 16:33:29.219909   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:33:39.220940   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:33:39.220988   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:34:09.223511900+02:00 I20230907 16:33:49.221670   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:33:49.221740   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:34:09.223515383+02:00 I20230907 16:33:56.156810   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:33:59.222398   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:33:59.222434   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:36:19.266247834+02:00 I20230907 16:34:09.223111   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:34:09.223410   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:34:19.224054   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:34:19.224094   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:34:29.224741   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:34:29.224776   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:34:39.225435   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:36:19.266601816+02:00 I20230907 16:34:39.225471   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:34:49.226186   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:34:49.226222   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:34:57.163491   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:34:59.227104   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:34:59.227280   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:35:09.244202   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:36:19.266663543+02:00 I20230907 16:35:09.245168   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:35:19.246631   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:35:19.246680   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:35:29.249236   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:35:29.249329   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:36:19.266704149+02:00 I20230907 16:35:39.253510   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:36:19.266712299+02:00 I20230907 16:35:39.253561   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:36:19.266720864+02:00 I20230907 16:35:49.256667   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:36:19.266731455+02:00 I20230907 16:35:49.256719   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:36:19.266741910+02:00 I20230907 16:35:58.187074   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:36:19.266759612+02:00 I20230907 16:35:59.260445   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:36:19.266770748+02:00 I20230907 16:35:59.260546   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:36:19.266781059+02:00 I20230907 16:36:09.263536   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:36:19.266791757+02:00 I20230907 16:36:09.263577   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318818337+02:00 I20230907 16:36:19.265493   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318836701+02:00 I20230907 16:36:19.266094   299 raft_ser
ver.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318839551+02:00 I20230907 16:36:29.271066   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318841793+02:00 I20230907 16:36:29.271200   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318843941+02:00 I20230907 16:36:39.279420   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318846077+02:00 I20230907 16:36:39.279484   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318848339+02:00 I20230907 16:36:49.290627   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318850467+02:00 I20230907 16:36:49.290772   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318852808+02:00 I20230907 16:36:59.197636   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:38:29.318857698+02:00 I20230907 16:36:59.297367   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318859887+02:00 I20230907 16:36:59.297494   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318862011+02:00 I20230907 16:37:09.298318   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318865014+02:00 I20230907 16:37:09.298417   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318867987+02:00 I20230
907 16:37:19.301503   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318870852+02:00 I20230907 16:37:19.301607   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318873536+02:00 I20230907 16:37:29.304710   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318876457+02:00 I20230907 16:37:29.304750   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318879261+02:00 I20230907 16:37:39.309211   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318887078+02:00 I20230907 16:37:39.309265   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318890386+02:00 I20230907 16:37:49.312556   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318892575+02:00 I20230907 16:37:49.312669   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318894697+02:00 I20230907 16:37:59.313561   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318896859+02:00 I20230907 16:37:59.313596   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318900790+02:00 I20230907 16:38:00.220654   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:38:29.318902972+02:00 I20230907 16:38:09.314311   262 raft_server.cpp:564] Term: 2, last_index index: 14
48, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318905118+02:00 I20230907 16:38:09.314344   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318907229+02:00 I20230907 16:38:19.315969   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:38:29.318909397+02:00 I20230907 16:38:19.316007   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:38:29.318923037+02:00 I20230907 16:38:29.318449   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337822037+02:00 I20230907 16:38:29.318509   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337868825+02:00 I20230907 16:38:39.319967   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337872685+02:00 I20230907 16:38:39.320060   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337875300+02:00 I20230907 16:38:49.321424   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337878042+02:00 I20230907 16:38:49.321511   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337880540+02:00 I20230907 16:38:59.322511   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337883079+02:00 I20230907 16:38:59.322556   299 raft_server.
h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337886215+02:00 I20230907 16:39:01.241528   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:40:49.337890293+02:00 I20230907 16:39:09.324499   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337894400+02:00 I20230907 16:39:09.324645   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337904709+02:00 I20230907 16:39:19.325690   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337909459+02:00 I20230907 16:39:19.325826   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337912058+02:00 I20230907 16:39:29.326601   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337914554+02:00 I20230907 16:39:29.326664   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337917030+02:00 I20230907 16:39:39.327740   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337919697+02:00 I20230907 16:39:39.327777   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337922183+02:00 I20230907 16:39:49.328608   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337924687+02:00 I20230907 16:39:49.328646   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337927182+02:00 I20230907 
16:39:59.329387   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337929678+02:00 I20230907 16:39:59.329454   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337932170+02:00 I20230907 16:40:02.255044   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:40:49.337934697+02:00 I20230907 16:40:09.330129   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337937468+02:00 I20230907 16:40:09.330201   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337941170+02:00 I20230907 16:40:19.332188   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337945343+02:00 I20230907 16:40:19.332222   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337949409+02:00 I20230907 16:40:29.333328   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337953706+02:00 I20230907 16:40:29.333364   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:40:49.337957895+02:00 I20230907 16:40:39.335991   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:40:49.337975805+02:00 I20230907 16:40:39.336083   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377697696+02:00 I20230907 16:40:49.337464   262 raft_server.cpp:564] Term: 2, last_index index: 1448, 
committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377714719+02:00 I20230907 16:40:49.337800   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377717017+02:00 I20230907 16:40:59.342478   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377725003+02:00 I20230907 16:40:59.342556   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377727057+02:00 I20230907 16:41:03.264240   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:42:59.377728802+02:00 I20230907 16:41:09.343191   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377730522+02:00 I20230907 16:41:09.343227   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377732455+02:00 I20230907 16:41:19.345552   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377734324+02:00 I20230907 16:41:19.345598   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377736057+02:00 I20230907 16:41:29.346921   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377737773+02:00 I20230907 16:41:29.347064   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377739471+02:00 I20230907 16:41:39.351914   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_
writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377741188+02:00 I20230907 16:41:39.352062   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377742893+02:00 I20230907 16:41:49.352993   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377744614+02:00 I20230907 16:41:49.353029   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377746320+02:00 I20230907 16:41:59.359923   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377748089+02:00 I20230907 16:41:59.359957   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377749813+02:00 I20230907 16:42:04.283854   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:42:59.377751534+02:00 I20230907 16:42:09.364686   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377753936+02:00 I20230907 16:42:09.365665   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377756294+02:00 I20230907 16:42:19.368320   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377758467+02:00 I20230907 16:42:19.368355   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377760745+02:00 I20230907 16:42:29.369127   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:5
9.377763059+02:00 I20230907 16:42:29.369194   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377765245+02:00 I20230907 16:42:39.370123   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377768449+02:00 I20230907 16:42:39.370209   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:42:59.377770181+02:00 I20230907 16:42:49.370857   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:42:59.377771904+02:00 I20230907 16:42:49.370903   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406766861+02:00 I20230907 16:42:59.377444   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406826922+02:00 I20230907 16:42:59.377674   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406847676+02:00 I20230907 16:43:05.301301   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:45:09.406856688+02:00 I20230907 16:43:09.380699   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406864071+02:00 I20230907 16:43:09.380733   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406871361+02:00 I20230907 16:43:19.381362   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406878624+02:00 I20230907 16:43:19.381409   299 raft_server.h:60] Peer ref
resh succeeded!
2023-09-07T18:45:09.406886953+02:00 I20230907 16:43:29.384903   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406896817+02:00 I20230907 16:43:29.384936   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406906368+02:00 I20230907 16:43:39.385644   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406920303+02:00 I20230907 16:43:39.385673   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406930428+02:00 I20230907 16:43:49.386977   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406940005+02:00 I20230907 16:43:49.387043   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406950790+02:00 I20230907 16:43:59.388796   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406958537+02:00 I20230907 16:43:59.388839   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406966513+02:00 I20230907 16:44:06.319248   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:45:09.406973859+02:00 I20230907 16:44:09.390421   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.406981215+02:00 I20230907 16:44:09.390461   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.406988674+02:00 I20230907 16:44:19.39280
2   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.407007241+02:00 I20230907 16:44:19.392920   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.407014721+02:00 I20230907 16:44:29.394214   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.407022017+02:00 I20230907 16:44:29.394250   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.407029277+02:00 I20230907 16:44:39.396353   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.407036554+02:00 I20230907 16:44:39.396409   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.407043846+02:00 I20230907 16:44:49.398945   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.407051136+02:00 I20230907 16:44:49.398979   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.407058499+02:00 I20230907 16:44:59.400401   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:45:09.407065746+02:00 I20230907 16:44:59.400434   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:45:09.407097088+02:00 I20230907 16:45:07.340353   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:47:19.445969365+02:00 I20230907 16:45:09.405902   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_inde
x: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:47:19.445990304+02:00 I20230907 16:45:09.406706   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:47:19.445992623+02:00 I20230907 16:45:19.409269   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:47:19.445994606+02:00 I20230907 16:45:19.409313   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:47:19.445996605+02:00 I20230907 16:45:29.412350   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:47:19.445998353+02:00 I20230907 16:45:29.412381   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:47:19.446000078+02:00 I20230907 16:45:39.413200   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:47:19.446001818+02:00 I20230907 16:45:39.413240   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:45:49.414548   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:45:49.414584   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:47:19.446007029+02:00 I20230907 16:45:59.415594   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:47:19.446013002+02:00 I20230907 16:45:59.415688   292 raft_server.h:60] Peer refresh succeeded!
I20230907 16:46:08.349647   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:46:09.416549   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:46:09.416652   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:46:19.420742   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:47:19.446024520+02:00 I20230907 16:46:19.420869   292 raft_server.h:60] Peer refresh succeeded!
I20230907 16:46:29.422032   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:47:19.446029833+02:00 I20230907 16:46:29.422075   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:46:39.426769   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:46:39.426831   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:46:49.432679   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:46:49.432727   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:46:59.436492   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:46:59.436576   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:47:09.367693   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:47:09.444370   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:47:09.444492   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472058114+02:00 I20230907 16:47:19.445700   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:49:29.472104443+02:00 I20230907 16:47:19.445942   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472110891+02:00 I20230907 16:47:29.447484   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:47:29.447587   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472123594+02:00 I20230907 16:47:39.449258   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:49:29.472144613+02:00 I20230907 16:47:39.449947   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:47:49.451269   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:47:49.451321   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472205154+02:00 I20230907 16:47:59.452847   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:49:29.472209676+02:00 I20230907 16:47:59.452962   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472214101+02:00 I20230907 16:48:09.455587   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:49:29.472218838+02:00 I20230907 16:48:09.456355   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:48:10.378072   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:48:19.459204   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:48:19.459300   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472237254+02:00 I20230907 16:48:29.463717   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:48:29.463759   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:48:39.464480   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:48:39.464511   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:48:49.465201   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:48:49.465250   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:48:59.466076   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:48:59.466217   305 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472274026+02:00 I20230907 16:49:09.466974   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:49:09.467108   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:49:29.472282979+02:00 I20230907 16:49:11.394212   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:49:19.470341   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:49:19.470397   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:49:29.471253   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506833626+02:00 I20230907 16:49:29.471889   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:49:39.472606   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506838513+02:00 I20230907 16:49:39.472642   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:49:49.473484   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506842750+02:00 I20230907 16:49:49.473608   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:49:59.478327   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506847253+02:00 I20230907 16:49:59.478507   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:51:39.506849308+02:00 I20230907 16:50:09.479187   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:50:09.479246   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:50:12.402351   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:51:39.506855669+02:00 I20230907 16:50:19.480391   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506857777+02:00 I20230907 16:50:19.480535   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:50:29.481473   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506861868+02:00 I20230907 16:50:29.481530   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:50:39.482787   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506866126+02:00 I20230907 16:50:39.482834   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:51:39.506868371+02:00 I20230907 16:50:49.485591   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:50:49.485635   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:51:39.506872714+02:00 I20230907 16:50:59.490762   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506879323+02:00 I20230907 16:50:59.491313   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:51:39.506881606+02:00 I20230907 16:51:09.503599   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506883677+02:00 I20230907 16:51:09.503719   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:51:39.506885957+02:00 I20230907 16:51:13.412426   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:51:39.506888093+02:00 I20230907 16:51:19.504729   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506890200+02:00 I20230907 16:51:19.504766   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:51:39.506892458+02:00 I20230907 16:51:29.505527   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:51:39.506894517+02:00 I20230907 16:51:29.505563   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:51:39.506902531+02:00 I20230907 16:51:39.506449   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:51:39.506489   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:53:49.519409613+02:00 I20230907 16:51:49.507345   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:53:49.519418209+02:00 I20230907 16:51:49.507408   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:53:49.519425139+02:00 I20230907 16:51:59.508363   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:53:49.519432892+02:00 I20230907 16:51:59.508404   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:53:49.519440954+02:00 I20230907 16:52:09.509209   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:53:49.519448509+02:00 I20230907 16:52:09.509264   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:52:14.417016   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:53:49.519462411+02:00 I20230907 16:52:19.510066   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:52:19.510102   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:52:29.511366   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:52:29.511407   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:52:39.512140   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:52:39.512214   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:52:49.513013   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:52:49.513049   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:53:49.519520164+02:00 I20230907 16:52:59.513851   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:53:49.519529132+02:00 I20230907 16:52:59.513895   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:53:49.519538057+02:00 I20230907 16:53:09.514724   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:53:49.519543827+02:00 I20230907 16:53:09.514762   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:53:49.519549488+02:00 I20230907 16:53:15.422190   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:53:49.519555152+02:00 I20230907 16:53:19.515636   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:53:49.519561307+02:00 I20230907 16:53:19.515671   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:53:29.516753   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:53:49.519577233+02:00 I20230907 16:53:29.516788   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:53:49.519584966+02:00 I20230907 16:53:39.517629   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:53:39.517669   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:53:49.518606   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532519966+02:00 I20230907 16:53:49.518702   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:56:09.532535674+02:00 I20230907 16:53:59.519462   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:53:59.519497   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:54:09.520290   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532562445+02:00 I20230907 16:54:09.520351   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:56:09.532567609+02:00 I20230907 16:54:16.427273   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:56:09.532570241+02:00 I20230907 16:54:19.522101   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532579192+02:00 I20230907 16:54:19.522168   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:56:09.532581307+02:00 I20230907 16:54:29.522970   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532583035+02:00 I20230907 16:54:29.523031   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:54:39.523911   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532586577+02:00 I20230907 16:54:39.523945   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:56:09.532588307+02:00 I20230907 16:54:49.524821   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532590030+02:00 I20230907 16:54:49.524860   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:54:59.525784   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:54:59.525847   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:56:09.532595385+02:00 I20230907 16:55:09.526659   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532597092+02:00 I20230907 16:55:09.526722   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:55:17.432358   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T18:56:09.532600564+02:00 I20230907 16:55:19.527577   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532602275+02:00 I20230907 16:55:19.527614   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:55:29.528379   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532605705+02:00 I20230907 16:55:29.528414   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:55:39.529318   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:56:09.532609135+02:00 I20230907 16:55:39.529350   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:56:09.532610944+02:00 I20230907 16:55:49.530324   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:55:49.530357   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:56:09.532614423+02:00 I20230907 16:55:59.531132   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:55:59.531173   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563906133+02:00 I20230907 16:56:09.532176   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:56:09.532493   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:56:18.441802   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:56:19.533317   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:56:19.533390   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:56:29.534129   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563940058+02:00 I20230907 16:56:29.534178   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:56:39.542182   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563944291+02:00 I20230907 16:56:39.544174   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:56:49.550580   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563948527+02:00 I20230907 16:56:49.550616   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563950653+02:00 I20230907 16:56:59.553225   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563952761+02:00 I20230907 16:56:59.553280   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563954878+02:00 I20230907 16:57:09.554186   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:57:09.554231   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563959081+02:00 I20230907 16:57:19.452324   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:57:19.557760   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563963373+02:00 I20230907 16:57:19.557806   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:57:29.558519   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563967681+02:00 I20230907 16:57:29.558558   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563969994+02:00 I20230907 16:57:39.559978   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:57:39.560016   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563978480+02:00 I20230907 16:57:49.561477   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563980603+02:00 I20230907 16:57:49.561517   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563982731+02:00 I20230907 16:57:59.562211   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:57:59.562254   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T18:58:19.563986932+02:00 I20230907 16:58:09.562929   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T18:58:19.563989164+02:00 I20230907 16:58:09.562968   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:58:19.563656   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T19:00:29.580843277+02:00 I20230907 16:58:19.563899   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:58:20.459970   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
2023-09-07T19:00:29.580854976+02:00 I20230907 16:58:29.564522   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:58:29.564555   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:58:39.565266   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:58:39.565328   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T19:00:29.580867508+02:00 I20230907 16:58:49.566066   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T19:00:29.580870828+02:00 I20230907 16:58:49.566104   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:58:59.566893   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T19:00:29.580876885+02:00 I20230907 16:58:59.566926   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:59:09.567569   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T19:00:29.580883090+02:00 I20230907 16:59:09.567603   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:59:19.570607   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:59:19.570647   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:59:21.463987   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
I20230907 16:59:29.571308   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T19:00:29.580904231+02:00 I20230907 16:59:29.571343   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:59:39.572181   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T19:00:29.580910195+02:00 I20230907 16:59:39.572216   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T19:00:29.580913147+02:00 I20230907 16:59:49.573062   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 16:59:49.573096   299 raft_server.h:60] Peer refresh succeeded!
I20230907 16:59:59.574018   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
2023-09-07T19:00:29.580922240+02:00 I20230907 16:59:59.574054   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T19:00:29.580925305+02:00 I20230907 17:00:09.578420   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 17:00:09.578545   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T19:00:29.580931264+02:00 I20230907 17:00:19.579238   262 raft_server.cpp:564] Term: 2, last_index index: 1448, committed_index: 1448, known_applied_index: 1448, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1296829
I20230907 17:00:19.579300   299 raft_server.h:60] Peer refresh succeeded!
2023-09-07T19:00:29.580951212+02:00 I20230907 17:00:22.468595   263 batched_indexer.cpp:285] Running GC for aborted requests, req map size: 0
jrasm91 commented 1 year ago

I mean there are all types of messages and errors here. If you restart the system is there a persistent issue you are running into?

yodatak commented 1 year ago

in mass importing some file i got this error again i use http3 if it can change somethink


Upload Progress | █████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 11% || 38675/327529 || Current file [/mnt/XXXXXX/IMG_4162.CR2]
Error asset:  {
  file: '/mnt/rXXXXXX/C0012.MP4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'https://XXXXXX/api/asset/import',
      data: '{"deviceAssetId":"C0012.MP4-3674651378","deviceId":"CLI","fileCreatedAt":"2021-07-15T12:12:16.000Z","fileModifiedAt":"2021-07-15T12:12:16.000Z","isFavorite":"false","fileExtension":".MP4","duration":"0:00:00.000000","isReadOnly":"true","assetPath":"/mnt/XXXXXX/C0012.MP4"}'
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 322,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'POST /api/asset/import HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'x-api-key: qcplVLVTAlBcPcIyOeukGuEVHZNyqPOn2wfCsU8aw\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Content-Length: 322\r\n' +
        'Host: XXXX\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/import',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'XXX',
      protocol: 'https:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Timeout',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
jrasm91 commented 1 year ago

504 is a gateway timeout which can happen with mp4s and --import. The server takes too long to hash the file and nginx throws an error. They are successfully imported though.

yodatak commented 1 year ago

Intersting it failled again on the same file , how could i fix it with nginx ?

jrasm91 commented 1 year ago

You need to add this in the nginx.conf to file in the /api location block

http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout

yodatak commented 1 year ago

In immich proxy i got this error that goes effectily to timeout issues ^^ Deployment: immich-proxy Container: immich

2023/09/07 21:24:28 [error] 43#43: *103492 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.42.0.78, server: , request: "POST /api/asset/import HTTP/1.1", upstream: "http://1XXXX:10001/asset/import", host: "XXXXX"
2023-09-07T21:24:28.262313247+02:00 2023/09/07 21:24:28 [error] 43#43: *103155 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 1XXXX, server: , request: "POST /api/asset/import HTTP/1.1", upstream: "http://XXXX:10001/asset/import", host: "XXXXXX"
2023-09-07T21:24:28.317295066+02:00 2023/09/07 21:24:28 [error] 45#45: *103025 upstream timed out (110: Operation timed out) while reading response header from upstream, client: XXXX, server: , request: "POST /api/asset/import HTTP/1.1", upstream: "http://XXXX:10001/asset/import", host: "XXXX"
2023/09/07 21:24:28 [error] 43#43: *103079 upstream timed out (110: Operation timed out) while reading response header from upstream, client: XXXXXX, server: , request: "POST /api/asset/import HTTP/1.1", upstream: "http://10XXXXXX:10001/asset/import", host: "XXXXX"
yodatak commented 1 year ago

I try this and i will say if its fix but we need to modiy the default.conf file in container

nginx.config

    keepalive_timeout  65;

defaut.conf

map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}

map $http_x_forwarded_proto $forwarded_protocol {
  default $scheme;

  # Only allow the values 'http' and 'https' for the X-Forwarded-Proto header.
  http http;
  https https;
}

upstream server {
  server immich-server:10001;
  keepalive 2;
}

upstream web {
  server immich-web:10000;
  keepalive 2;
}

server {
  listen 8080;

  access_log off;
  client_max_body_size 50000M;

  # Compression
  gzip on;
  gzip_comp_level 2;
  gzip_min_length 1000;
  gzip_proxied any;
  gzip_vary on;
  gunzip on;

  # text/html is included by default
  gzip_types
    application/javascript
    application/json
    font/ttf
    image/svg+xml
    text/css;

  proxy_buffering off;
  proxy_request_buffering off;
  proxy_buffer_size 16k;
  proxy_busy_buffers_size 24k;
  proxy_buffers 64 4k;
  proxy_force_ranges on;

  proxy_http_version 1.1;
  proxy_set_header Host $http_host;
  proxy_set_header X-Forwarded-Host $http_host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $forwarded_protocol;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection $connection_upgrade;

  location /api {

    rewrite /api/(.*) /$1 break;
    proxy_read_timeout 300s;
    proxy_pass http://server;
  }

  location / {

    proxy_pass http://web;
  }
}

and i reload the container in runtime nginx -s reload

we must fix it in https://github.com/immich-app/immich/blob/main/nginx/templates/default.conf.template

jrasm91 commented 1 year ago

Long timeouts aren't necessarily good, but for this specific issue/endpoint it would probably be worthwhile

jrasm91 commented 1 year ago

We could make the timeout just for the import endpoint.

yodatak commented 10 months ago

Anny news on this i got this issue on different ending point

Error: Connection terminated due to connection timeout

2023-10-26T20:25:55.462204908+02:00 Error: connect ETIMEDOUT at Socket. (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41) 2023-10-26T20:25:55.462209120+02:00 at Object.onceWrapper (node:events:628:28) at Socket.emit (node:events:514:28) at Socket._onTimeout (node:net:589:8) 2023-10-26T20:25:55.462215992+02:00 at listOnTimeout (node:internal/timers:573:17) 2023-10-26T20:25:55.462218781+02:00 at process.processTimers (node:internal/timers:514:7) { 2023-10-26T20:25:55.462220819+02:00 errorno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect' } Error: connect ETIMEDOUT at Socket. (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41) 2023-10-26T20:25:55.462233716+02:00 at Object.onceWrapper (node:events:628:28) 2023-10-26T20:25:55.462237799+02:00 at Socket.emit (node:events:514:28) at Socket._onTimeout (node:net:589:8) at listOnTimeout (node:internal/timers:573:17) 2023-10-26T20:25:55.462244319+02:00 at process.processTimers (node:internal/timers:514:7) { errorno: 'ETIMEDOUT', 2023-10-26T20:25:55.462248581+02:00 code: 'ETIMEDOUT', 2023-10-26T20:25:55.462250657+02:00 syscall: 'connect' 2023-10-26T20:25:55.462252751+02:00 } 2023-10-26T20:25:55.462255039+02:00 Error: connect ETIMEDOUT 2023-10-26T20:25:55.462257197+02:00 at Socket. (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41) 2023-10-26T20:25:55.462259347+02:00 at Object.onceWrapper (node:events:628:28) 2023-10-26T20:25:55.462261472+02:00 at Socket.emit (node:events:514:28) at Socket._onTimeout (node:net:589:8) at listOnTimeout (node:internal/timers:573:17) 2023-10-26T20:25:55.462270760+02:00 at process.processTimers (node:internal/timers:514:7) { errorno: 'ETIMEDOUT', 2023-10-26T20:25:55.462275142+02:00 code: 'ETIMEDOUT', syscall: 'connect' } 2023-10-26T20:25:55.462281600+02:00 Error: connect ETIMEDOUT at Socket. (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41) 2023-10-26T20:25:55.462285891+02:00 at Object.onceWrapper (node:events:628:28) at Socket.emit (node:events:514:28) 2023-10-26T20:25:55.462290155+02:00 at Socket._onTimeout (node:net:589:8) at listOnTimeout (node:internal/timers:573:17) 2023-10-26T20:25:55.462294432+02:00 at process.processTimers (node:internal/timers:514:7) { 2023-10-26T20:25:55.462296511+02:00 errorno: 'ETIMEDOUT', code: 'ETIMEDOUT', 2023-10-26T20:25:55.462300825+02:00 syscall: 'connect' 2023-10-26T20:25:55.462302901+02:00 } Error: connect ETIMEDOUT 2023-10-26T20:25:55.462307119+02:00 at Socket. (/usr/src/app/node_modules/ioredis/built/Redis.js:170:41) 2023-10-26T20:25:55.462309280+02:00 at Object.onceWrapper (node:events:628:28) at Socket.emit (node:events:514:28) 2023-10-26T20:25:55.462313786+02:00 at Socket._onTimeout (node:net:589:8) 2023-10-26T20:25:55.462316104+02:00 at listOnTimeout (node:internal/timers:573:17) 2023-10-26T20:25:55.462318139+02:00 at process.processTimers (node:internal/timers:514:7) { errorno: 'ETIMEDOUT', 2023-10-26T20:25:55.462322457+02:00 code: 'ETIMEDOUT', syscall: 'connect' 2023-10-26T20:25:55.462326839+02:00 } [Nest] 8 - 10/26/2023, 8:25:55 PM LOG [MetadataService] Initialized local reverse geocoder with cities500 [Nest] 8 - 10/26/2023, 8:25:55 PM LOG [SearchService] Running bootstrap [Nest] 8 - 10/26/2023, 8:25:55 PM LOG [TypesenseRepository] Schema up to date: assets/assets-v10 2023-10-26T20:25:56.028128335+02:00 [Nest] 8 - 10/26/2023, 8:25:56 PM LOG [TypesenseRepository] Schema up to date: albums/albums-v2 [Nest] 8 - 10/26/2023, 8:25:56 PM LOG [TypesenseRepository] Schema up to date: faces/faces-v1 [Nest] 8 - 10/26/2023, 8:25:56 PM LOG [TypesenseRepository] Alias mapping: [{"collection_name":"albums-v2","name":"albums"}] [Nest] 8 - 10/26/2023, 8:25:56 PM LOG [TypesenseRepository] Collections needing migration: {"assets":true,"albums":false,"faces":true} [Nest] 8 - 10/26/2023, 8:25:56 PM LOG [NestApplication] Nest application successfully started +303ms 2023-10-26T20:25:56.359155681+02:00 [Nest] 8 - 10/26/2023, 8:25:56 PM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:10004 [v1.82.1] [PRODUCTION] 2023-10-26T20:27:41.200426155+02:00 [Nest] 8 - 10/26/2023, 8:27:41 PM ERROR [JobService] Unable to run job handler (search/search-index-assets): Error: Connection terminated due to connection timeout [Nest] 8 - 10/26/2023, 8:27:41 PM ERROR [JobService] Error: Connection terminated due to connection timeout 2023-10-26T20:27:41.200470079+02:00 at Connection. (/usr/src/app/node_modules/pg/lib/client.js:132:73) at Object.onceWrapper (node:events:628:28) 2023-10-26T20:27:41.200474479+02:00 at Connection.emit (node:events:514:28) at Socket. (/usr/src/app/node_modules/pg/lib/connection.js:63:12) 2023-10-26T20:27:41.200478859+02:00 at Socket.emit (node:events:514:28) at TCP. (node:net:337:12) 2023-10-26T20:27:41.201124331+02:00 [Nest] 8 - 10/26/2023, 8:27:41 PM ERROR [JobService] Object: {} 2023-10-26T20:27:41.201137732+02:00

<--- Last few GCs ---> 2023-10-26T20:29:50.703361758+02:00 2023-10-26T20:29:50.703366124+02:00 [8:0x592a4310000] 254858 ms: Mark-Compact (reduce) 2046.4 (2083.4) -> 2045.9 (2083.9) MB, 4529.62 / 0.01 ms (+ 293.9 ms in 57 steps since start of marking, biggest step 43.7 ms, walltime since start of marking 4882 ms) (average mu = 0.119, current mu =[8:0x592a4310000] 257716 ms: Mark-Compact (reduce) 2047.0 (2083.9) -> 2045.4 (2083.9) MB, 2854.69 / 0.00 ms (average mu = 0.073, current mu = 0.001) allocation failure; scavenge might not succeed

2023-10-26T20:29:50.703371229+02:00 2023-10-26T20:29:50.703376160+02:00 <--- JS stacktrace ---> 2023-10-26T20:29:50.703378497+02:00 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0xc99960 node::Abort() [immich_microservices] 2: 0xb6ffcb [immich_microservices] 3: 0xebe910 v8::Utils::ReportOOMFailure(v8::internal::Isolate, char const, v8::OOMDetails const&) [immich_microservices] 4: 0xebebf7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate, char const, v8::OOMDetails const&) [immich_microservices] 2023-10-26T20:29:50.721524217+02:00 5: 0x10d06a5 [immich_microservices] 6: 0x10d0c34 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [immich_microservices] 7: 0x10e7b24 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const) [immich_microservices] 8: 0x10e833c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [immich_microservices] 9: 0x10be641 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 10: 0x10bf7d5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [immich_microservices] 11: 0x109be16 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [immich_microservices] 12: 0x108d914 v8::internal::FactoryBase::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [immich_microservices] 13: 0x1090196 v8::internal::FactoryBase::NewRawOneByteString(int, v8::internal::AllocationType) [immich_microservices] 14: 0x10a7324 v8::internal::Factory::NewStringFromUtf8(v8::base::Vector const&, v8::internal::AllocationType) [immich_microservices] 2023-10-26T20:29:50.871139582+02:00 15: 0xed1202 v8::String::NewFromUtf8(v8::Isolate, char const*, v8::NewStringType, int) [immich_microservices] 16: 0xdb7249 [immich_microservices] 17: 0xc6da6f [immich_microservices] 18: 0x7fbfc608d546 Aborted (core dumped)

alextran1502 commented 10 months ago

not relevance anymore with the library feature