search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
44.56k stars 2.17k forks source link

[BUG] Immich Stops Working when using CLI #2957

Closed lonewolfpk closed 10 months ago

lonewolfpk commented 1 year ago

The bug

I recently discovered an issue that immich stops working almost completely (Functionality is broken). When Using CLI (uploading), I have just go to the administration tab in immich server. Then suddenly the upload shows error 500 undefined. and my server suddenly went offline. I tried refreshing the page but it didnt work. complete reinstallation of immich is the only solution i have found. I tried this several time to check this can be replicated. You can replicate every time you do this.

The OS that Immich Server is running on

Windows 11 Pro

Version of Immich Server

1.63.2

Version of Immich Mobile App

Not applicable

Platform with the issue

Your docker-compose.yml content

Not Applicable.

Your .env content

Not Applicable

Reproduction steps

1.Start CLI upload
2.Go to administration tab
3.Check your Jobs or storage (anything in administration tab)
4.you encounter the crash of 500 undefined error
5.From this point you cannot use immich unless done a complete reset of server.

Additional information

No response

alextran1502 commented 1 year ago

Can you please post the logs of the server when the issue happens? How do you install and run Immich?

alextran1502 commented 1 year ago

What is the resource do you allocate for Immich?

lonewolfpk commented 1 year ago

Win 11 pro( running inside a VM - sorry to mention before) Intel i5-10300h - 75% is allocated for VM 16 GB ram (8 GB for VM) Didn't allocated anything specifically for immich ( maybe using default resources values)

lonewolfpk commented 1 year ago

I found that the microservices (postgres,typesense,redist) stopped when I do these things. The service initially started but stops when uploading and and inside it shows error logs ( I reinstalled so logs got erased some). I installed docker in another drive and it fixed the issue.

lonewolfpk commented 1 year ago

This issue started happening again. This time I have the log reports. This is report from the cli utility: (like the below it is showing for the remaining assets that need to be uploaded.)

Error asset:  {
  file: 'E:\\admin\\2020\\2020-11\\IMG20201114200256.jpg',
  reason: Error: error request aborted
      at createError (E:\docker\docker apps\immich\immich repo\CLI-main\node_modules\axios\lib\core\createError.js:16:15)
      at IncomingMessage.handlerStreamAborted (E:\docker\docker apps\immich\immich repo\CLI-main\node_modules\axios\lib\adapters\http.js:301:18)
      at IncomingMessage.emit (node:events:511:28)
      at IncomingMessage._destroy (node:_http_incoming:224:10)
      at _destroy (node:internal/streams/destroy:109:10)
      at IncomingMessage.destroy (node:internal/streams/destroy:71:5)
      at Socket.socketCloseListener (node:_http_client:455:11)
      at Socket.emit (node:events:523:35)
      at TCP.<anonymous> (node:net:334:12) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'http://192.168.0.161:2283/api/asset/upload',
      data: [FormData]
    },
    code: 'ERR_REQUEST_ABORTED',
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: false,
      chunkedEncoding: true,
      shouldKeepAlive: true,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: null,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [Socket],
      _header: 'POST /api/asset/upload HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: multipart/form-data; boundary=--------------------------009856132569760699902137\r\n' +
        'x-api-key: MSwyznUkClH8WkYst45GxZg2lpREUpTIAMCl01cb80\r\n' +
        'User-Agent: axios/0.26.0\r\n' +
        'Host: 192.168.0.161:2283\r\n' +
        'Connection: keep-alive\r\n' +
        'Transfer-Encoding: chunked\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/upload',
      _ended: false,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: [Function: emitRequestTimeout],
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: true,
      host: '192.168.0.161',
      protocol: 'http:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: undefined,
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: undefined
}

This is the report from the docker desktop server log:
[Nest] 1  - 06/28/2023, 3:35:39 PM   ERROR [ExpressAdapter] Premature close
2023-06-28 08:35:39 Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
2023-06-28 08:35:39     at new NodeError (node:internal/errors:399:5)
2023-06-28 08:35:39     at ServerResponse.onclose (node:internal/streams/end-of-stream:154:30)
2023-06-28 08:35:39     at ServerResponse.emit (node:events:525:35)
2023-06-28 08:35:39     at emitCloseNT (node:_http_server:984:10)
2023-06-28 08:35:39     at Socket.onServerResponseClose (node:_http_server:277:5)
2023-06-28 08:35:39     at Socket.emit (node:events:525:35)
2023-06-28 08:35:39     at TCP.<anonymous> (node:net:322:12)

This is from immich proxy: ( i did not reset any connection BTW)
2023/06/28 12:45:18 [error] 42#42: *859 recv() failed (104: Connection reset by peer) while reading upstream, client: 172.18.0.1, server: , request: "POST /api/asset/upload HTTP/1.1", upstream: "http://172.18.0.8:3001/asset/upload", host: "192.168.0.161:2283"
alextran1502 commented 1 year ago

Is the server stopped working again, or your just cannot upload those files? Can you still access the web?

lonewolfpk commented 1 year ago

I can access the web interface. From that point nothing is uploading.

lonewolfpk commented 1 year ago

Everytime this occurs. Error asset: { file: 'D:\admin\2021\2021-05\VID20210508005632.mp4', reason: Error: Request failed with status code 504 at createError (E:\Docker\Docker Apps\Immich\Immich-App\CLI-main\node_modules\axios\lib\core\createError.js:16:15) at settle (E:\Docker\Docker Apps\Immich\Immich-App\CLI-main\node_modules\axios\lib\core\settle.js:17:12) at IncomingMessage.handleStreamEnd (E:\Docker\Docker Apps\Immich\Immich-App\CLI-main\node_modules\axios\lib\adapters\http.js:322:11) at IncomingMessage.emit (node:events:523:35) at endReadableNT (node:internal/streams/readable:1367:12) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { config: { transitional: [Object], adapter: [Function: httpAdapter], transformRequest: [Array], transformResponse: [Array], timeout: 0, xsrfCookieName: 'XSRF-TOKEN', xsrfHeaderName: 'X-XSRF-TOKEN', maxContentLength: Infinity, maxBodyLength: Infinity, validateStatus: [Function: validateStatus], headers: [Object], method: 'post', maxRedirects: 0, url: 'http://192.168.0.90:2283/api/asset/upload', data: [FormData] }, request: ClientRequest { _events: [Object: null prototype], _eventsCount: 3, _maxListeners: undefined, outputData: [], outputSize: 0, writable: true, destroyed: true, _last: false, chunkedEncoding: true, shouldKeepAlive: true, maxRequestsOnConnectionReached: false, _defaultKeepAlive: true, useChunkedEncodingByDefault: true, sendDate: false, _removedConnection: false, _removedContLen: false, _removedTE: false, strictContentLength: false, _contentLength: null, _hasBody: true, _trailer: '', finished: false, _headerSent: true, _closed: true, socket: [Socket], _header: 'POST /api/asset/upload HTTP/1.1\r\n' + 'Accept: application/json, text/plain, /\r\n' + 'Content-Type: multipart/form-data; boundary=--------------------------540162392401991048968072\r\n' + 'x-api-key: UtJmjR1ZjCnnJyGPp4DXl3umt3CHaOsXXU1QDgbv0\r\n' + 'User-Agent: axios/0.26.0\r\n' + 'Host: 192.168.0.90:2283\r\n' + 'Connection: keep-alive\r\n' + 'Transfer-Encoding: chunked\r\n' + '\r\n', _keepAliveTimeout: 0, _onPendingData: [Function: nop], agent: [Agent], socketPath: undefined, method: 'POST', maxHeaderSize: undefined, insecureHTTPParser: undefined, joinDuplicateHeaders: undefined, path: '/api/asset/upload', _ended: true, res: [IncomingMessage], aborted: false, timeoutCb: [Function: emitRequestTimeout], upgradeOrConnect: false, parser: null, maxHeadersCount: null, reusedSocket: true, host: '192.168.0.90', protocol: 'http:',

  [Symbol(kBytesWritten)]: 0,
  [Symbol(kNeedDrain)]: true,
  [Symbol(corked)]: 0,
  [Symbol(kOutHeaders)]: [Object: null prototype],
  [Symbol(errored)]: null,
  [Symbol(kHighWaterMark)]: 16384,
  [Symbol(kRejectNonStandardBodyWrites)]: false,
  [Symbol(kUniqueHeaders)]: null
},
response: {
  status: 504,
  statusText: 'Gateway Time-out',
  headers: [Object],
  config: [Object],
  request: [ClientRequest],
  data: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
},
isAxiosError: true,
toJSON: [Function: toJSON]

}, response: '\r\n' + '504 Gateway Time-out\r\n' + '\r\n' + '

504 Gateway Time-out

\r\n' + '
nginx/1.25.0
\r\n' + '\r\n' + '\r\n' }

jrasm91 commented 1 year ago

Does this also happen when you upload a file through the web interface? Did it always crash on the first file or only when you upload multiple? Can you change the docker compose file t remove the restart: always lines and let the container stay off if it crashes, then verify which one crashed.

lonewolfpk commented 1 year ago

It crashes in the middle. I was uploading 10k+ media. The microservices crashed

dcglabs commented 1 year ago

I got same issue with using cli upload tool. With docker compose setup. docker run -it --rm -v "$(pwd):/import" ghcr.io/immich-app/immich-cli:latest upload --recursive /import/ --key 5r86rOgwH0BTHKRnW3RcV2Wf3gUc5iBdzzocEDzw --album dcg_photo --server http://192.168.21.87:8080/api This upload was for 60k items ~400GB. I had done a couple different batches of 10k without issue. This was last bit of output from the docker cli run. I wonder if doing massive cli upload you should disable all the thumbnail, face, tagging, etc items? This is on a core i3-13100 with 64gb ram. It seemed to get through more than 50% of upload before crashing. Not sure when it did. The album dcg_photo does exist but no items are associated with it. 

Error asset:  {
  file: '/import/dcg_cloud/Go Pro/100GOPRO/.@__thumb/defaultGOPR0037.MP4',
  reason: Error: Request failed with status code 500
      at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:539:35)
      at endReadableNT (node:internal/streams/readable:1345:12)
      at processTicksAndRejections (node:internal/process/task_queues:83:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'http://192.168.21.87:8080/api/asset/upload',
      data: [FormData]
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: true,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: null,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [Socket],
      _header: 'POST /api/asset/upload HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: multipart/form-data; boundary=--------------------------203081941416512938797114\r\n' +
        'x-api-key: 5r86rOgwH0BTHKRnW3RcV2Wf3gUc5iBdzzocEDzw\r\n' +
        'User-Agent: axios/0.26.0\r\n' +
        'Host: 192.168.21.87:8080\r\n' +
        'Connection: close\r\n' +
        'Transfer-Encoding: chunked\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/api/asset/upload',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: '192.168.21.87',
      protocol: 'http:',
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    response: {
      status: 500,
      statusText: 'Internal Server Error',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: [Object]
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: { statusCode: 500, message: 'Internal server error' }
}
Error asset:  {
  file: '/import/dcg_cloud/Go Pro/100GOPRO/GP050042.MP4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:539:35)
      at endReadableNT (node:internal/streams/readable:1345:12)
      at processTicksAndRejections (node:internal/process/task_queues:83:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'http://192.168.21.87:8080/api/asset/upload',
      data: [FormData]
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: true,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: null,
      _hasBody: true,
      _trailer: '',
      finished: false,
      _headerSent: true,
      _closed: true,
      socket: [Socket],
      _header: 'POST /api/asset/upload HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: multipart/form-data; boundary=--------------------------404389337864740434891698\r\n' +
        'x-api-key: 5r86rOgwH0BTHKRnW3RcV2Wf3gUc5iBdzzocEDzw\r\n' +
        'User-Agent: axios/0.26.0\r\n' +
        'Host: 192.168.21.87:8080\r\n' +
        'Connection: close\r\n' +
        'Transfer-Encoding: chunked\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/api/asset/upload',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: '192.168.21.87',
      protocol: 'http:',
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: true,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    response: {
      status: 504,
      statusText: 'Gateway Time-out',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
isaaclyman commented 1 year ago

Happens for me as well (docker compose setup, running on a Windows desktop). I've found I can avoid the issue if I pause all jobs on the web interface and upload only a couple hundred assets at a time. I'm guessing the various jobs use up all the available connections on my network card, causing everything else to time out and spiral into an infinite loop.

When this happens, Docker is unable to kill the immich_microservices and/or immich_server containers. I have to restart the machine to bring the server back up.

In the error state, I get a 500 error from the webpage and this log repeats constantly in the immich_server container logs:

[Nest] 7  - 07/31/2023, 7:22:35 PM   ERROR [ExceptionsHandler] Connection terminated due to connection timeout
Error: Connection terminated due to connection timeout
    at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
    at Object.onceWrapper (node:events:627:28)
    at Connection.emit (node:events:513:28)
    at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:62:12)
    at Socket.emit (node:events:513:28)
    at TCP.<anonymous> (node:net:322:12)

I also typically see errors indicating that one of the containers can't resolve download.geonames.org. I wonder if the file it's referencing could be cached?

isaaclyman commented 1 year ago

Update: Just saw this happen while I was not uploading any assets (none of the CLI/web upload/mobile app were running). The only jobs running were:

marank commented 1 year ago

I'M experiencing the same. I wanted to upload ~25k images and videos, and after a while, my VM gets unreliable. On the same VM, I have Uptime-Kuma running, and I get error messages from it like the following: "Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?" This happend never before.

The VM is running on Proxmox on an Intel(R) Xeon(R) CPU E5-2680 v3 with 8 cores and 32GB RAM assigned to it.

Here are some relevant logs from the docker stack:

immich_microservices     | Request #1692557416139: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557416139: Sleeping for 4s and then retrying request...
immich_microservices     | [Nest] 7  - 08/20/2023, 6:50:28 PM     LOG [MediaService] Encoding success 2458c407-cbcb-4576-b4b6-228c94dae604
immich_microservices     | Request #1692557427899: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557427899: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557427924: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557427924: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557416139: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557416139: Sleeping for 4s and then retrying request...
immich_microservices     | /usr/src/app/dist/microservices/app.service.js:74
immich_microservices     |                 throw error;
immich_microservices     |                 ^
immich_microservices     | 
immich_microservices     | 
immich_microservices     | Error: timeout exceeded when trying to connect
immich_microservices     |     at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:200:27)
immich_microservices     |     at listOnTimeout (node:internal/timers:569:17)
immich_microservices     |     at process.processTimers (node:internal/timers:512:7)
immich_microservices     | 
immich_microservices     | 
immich_microservices     | Node.js v18.17.0
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:03 PM     LOG [NestFactory] Starting Nest application...
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:03 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +98ms
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:03 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:03 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:03 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +1ms
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:03 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +11ms
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:03 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:03 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:06 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +3481ms
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:06 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:06 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:06 PM     LOG [InstanceLoader] InfraModule dependencies initialized +6ms
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:06 PM     LOG [InstanceLoader] DomainModule dependencies initialized +5ms
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:06 PM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:06 PM     LOG [MetadataExtractionProcessor] Reverse geocoding is enabled
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:06 PM     LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:16 PM     LOG [MediaService] Start encoding video 8793748a-70e7-49c0-9300-a2e71365a1ab {"inputOptions":[],"outputOptions":["-vcodec h264","-acodec aac","-movflags faststart","-fps_mode passthrough","-v verbose","-vf format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false}
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:16 PM     LOG [MediaService] Successfully generated image thumbnail bf639d57-484a-4839-aa34-df957fe2e521
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:22 PM     LOG [MediaService] Encoding success 8793748a-70e7-49c0-9300-a2e71365a1ab
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:28 PM     LOG [MetadataExtractionProcessor] Reverse Geocoding Initialized
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:29 PM     LOG [NestApplication] Nest application successfully started +1725ms
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:29 PM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.74.0] [PRODUCTION] 
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:31 PM     LOG [MediaService] Start encoding video ad550acd-b469-44ef-952b-c7024ca5f7b9 {"inputOptions":[],"outputOptions":["-vcodec h264","-acodec aac","-movflags faststart","-fps_mode passthrough","-v verbose","-vf format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false}
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:35 PM     LOG [MediaService] Successfully generated image thumbnail bbefc993-8069-4888-991b-7cfcde71b8b1
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:38 PM     LOG [MediaService] Encoding success ad550acd-b469-44ef-952b-c7024ca5f7b9
immich_microservices     | Request #1692557914758: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557914758: Sleeping for 4s and then retrying request...
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:45 PM     LOG [MediaService] Start encoding video ac256d69-7f52-4145-a904-be5cdd44f379 {"inputOptions":[],"outputOptions":["-vcodec h264","-acodec aac","-movflags faststart","-fps_mode passthrough","-v verbose","-vf format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false}
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:47 PM     LOG [MediaService] Successfully generated image thumbnail e7f9c7c2-3106-455f-aba4-abb5f93d8fcc
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:47 PM     LOG [MediaService] Successfully generated image thumbnail 6fb80b9a-cdb4-4fd5-9360-0c2a4d9a46e8
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:47 PM     LOG [MediaService] Successfully generated image thumbnail 060b1572-124b-4244-99e7-f3dfd1056381
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:47 PM     LOG [MediaService] Successfully generated image thumbnail d7566ee1-2692-4c77-b874-b382ddf5224f
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:48 PM     LOG [MediaService] Successfully generated image thumbnail eb422585-0ee4-4288-b36e-27258983baf9
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:48 PM     LOG [MediaService] Successfully generated image thumbnail f3896b02-c7cd-4418-bdde-b5d079e7fe66
immich_microservices     | [Nest] 7  - 08/20/2023, 6:58:48 PM     LOG [MediaService] Successfully generated image thumbnail 13456103-fc30-45cc-a730-daf7533b45f8
immich_microservices     | Request #1692557914758: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557914758: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557931136: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557931136: Sleeping for 4s and then retrying request...
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:03 PM     LOG [MediaService] Encoding success ac256d69-7f52-4145-a904-be5cdd44f379
immich_microservices     | Request #1692557937273: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557937273: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557914758: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557914758: Sleeping for 4s and then retrying request...
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:13 PM     LOG [MediaService] Successfully generated image thumbnail f98feafe-1538-4d20-a406-8f934a5e3a92
immich_microservices     | Request #1692557944752: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557944752: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557944762: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557944762: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557931136: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557931136: Sleeping for 4s and then retrying request...
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:16 PM     LOG [MediaService] Start encoding video 2458c407-cbcb-4576-b4b6-228c94dae604 {"inputOptions":[],"outputOptions":["-vcodec h264","-acodec aac","-movflags faststart","-fps_mode passthrough","-v verbose","-vf format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false}
immich_microservices     | Request #1692557937273: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557937273: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557953219: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557953219: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557953224: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557953224: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557953230: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557953230: Sleeping for 4s and then retrying request...
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:23 PM     LOG [MediaService] Encoding success 2458c407-cbcb-4576-b4b6-228c94dae604
immich_microservices     | Request #1692557956315: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557956315: Sleeping for 4s and then retrying request...
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:27 PM     LOG [MediaService] Start encoding video ff2e3ca5-1975-480a-a9c7-cba4a73a0302 {"inputOptions":[],"outputOptions":["-vcodec h264","-acodec aac","-movflags faststart","-fps_mode passthrough","-v verbose","-vf format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false}
immich_microservices     | Request #1692557944752: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557944752: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557944762: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557944762: Sleeping for 4s and then retrying request...
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:29 PM     LOG [MediaService] Successfully generated image thumbnail f7b46139-fece-4f15-99aa-e3457af66391
immich_microservices     | Request #1692557931136: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557931136: Sleeping for 4s and then retrying request...
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:30 PM     LOG [MediaService] Successfully generated image thumbnail f4f80143-e815-4e20-93f5-8b1bd52912b2
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:30 PM     LOG [MediaService] Successfully generated image thumbnail 013254be-6d54-48f3-b842-887808c0b0c6
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:30 PM     LOG [MediaService] Successfully generated image thumbnail f90260fe-c89d-4e18-ac30-fd1bfac86b18
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:30 PM     LOG [MediaService] Successfully generated image thumbnail dbe3d55b-18ff-4145-bc13-44050c74802a
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:30 PM     LOG [MediaService] Successfully generated image thumbnail 09da5613-aaa3-4b75-b16c-821d041ac3c8
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:31 PM     LOG [MediaService] Successfully generated image thumbnail c22ce114-2f7a-4253-aa7f-57b789dd0a0b
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:31 PM     LOG [MediaService] Successfully generated image thumbnail 912d44fe-c282-4116-975c-417e978e55dd
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:31 PM     LOG [MediaService] Successfully generated image thumbnail 01070ba6-bb58-4bce-9d00-87694b2c0e14
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:31 PM     LOG [MediaService] Successfully generated image thumbnail 6a9a5d7a-bedb-4e0a-94fb-ab889666921a
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:32 PM     LOG [MediaService] Successfully generated image thumbnail f85ad065-89d3-4424-b5c2-3bee18e79dd7
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:32 PM     LOG [MediaService] Successfully generated image thumbnail ff7ab7d8-9f0a-409b-8cbe-b09eee179aa6
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:32 PM     LOG [MediaService] Successfully generated image thumbnail 9810deed-8831-451d-ac0e-639c79263b8f
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:40 PM     LOG [MediaService] Encoding success ff2e3ca5-1975-480a-a9c7-cba4a73a0302
immich_microservices     | [Nest] 7  - 08/20/2023, 6:59:41 PM     LOG [MediaService] Successfully generated image thumbnail 3fb27823-b5cf-4c01-aea9-e36517e5b1d5
immich_microservices     | Request #1692557944762: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557944762: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557931136: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557931136: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557976148: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557976148: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1692557981149: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
immich_microservices     | Request #1692557981149: Sleeping for 4s and then retrying request...
immich_microservices     | /usr/src/app/dist/microservices/app.service.js:74
immich_microservices     |                 throw error;
immich_microservices     |                 ^
immich_microservices     | 
immich_microservices     | 
immich_microservices     | Error: timeout exceeded when trying to connect
immich_microservices     |     at Timeout._onTimeout (/usr/src/app/node_modules/pg-pool/index.js:200:27)
immich_microservices     |     at listOnTimeout (node:internal/timers:569:17)
immich_microservices     |     at process.processTimers (node:internal/timers:512:7)
immich_microservices     | 
immich_microservices     | 
immich_microservices     | Node.js v18.17.0
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [NestFactory] Starting Nest application...
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +41ms
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +12ms
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +194ms
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [InstanceLoader] InfraModule dependencies initialized +5ms
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [InstanceLoader] DomainModule dependencies initialized +6ms
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +0ms
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [MetadataExtractionProcessor] Reverse geocoding is enabled
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:43 PM     LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:51 PM     LOG [MediaService] Start encoding video 1ed80f64-ba9a-4231-9a93-35e856d062e7 {"inputOptions":[],"outputOptions":["-vcodec h264","-acodec aac","-movflags faststart","-fps_mode passthrough","-v verbose","-vf format=yuv420p","-preset ultrafast","-crf 23"],"twoPass":false}
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:52 PM     LOG [MetadataExtractionProcessor] Reverse Geocoding Initialized
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:52 PM     LOG [NestApplication] Nest application successfully started +78ms
immich_microservices     | [Nest] 7  - 08/20/2023, 7:03:52 PM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.74.0] [PRODUCTION] 

Logs from the immich cli:

Error asset:  {
  file: '/mnt/media/photos/Jahre/2019/Italien 2019/20190906_204022.mp4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'http://10.0.0.4:2283/api/asset/upload',
      data: [FormData]
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: true,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: null,
      _hasBody: true,
      _trailer: '',
      finished: false,
      _headerSent: true,
      _closed: true,
      socket: [Socket],
      _header: 'POST /api/asset/upload HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: multipart/form-data; boundary=--------------------------736417810135127844557825\r\n' +
        'x-api-key: <redacted>\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Host: 10.0.0.4:2283\r\n' +
        'Connection: close\r\n' +
        'Transfer-Encoding: chunked\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/upload',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: '10.0.0.4',
      protocol: 'http:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: true,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Time-out',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
Error asset:  {
  file: '/mnt/media/photos/Jahre/2019/Italien 2019/20190906_210410.mp4',
  reason: Error: Request failed with status code 504
      at createError (/usr/src/app/node_modules/axios/lib/core/createError.js:16:15)
      at settle (/usr/src/app/node_modules/axios/lib/core/settle.js:17:12)
      at IncomingMessage.handleStreamEnd (/usr/src/app/node_modules/axios/lib/adapters/http.js:322:11)
      at IncomingMessage.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1359:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: Infinity,
      maxBodyLength: Infinity,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      maxRedirects: 0,
      url: 'http://10.0.0.4:2283/api/asset/upload',
      data: [FormData]
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: true,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: null,
      _hasBody: true,
      _trailer: '',
      finished: false,
      _headerSent: true,
      _closed: true,
      socket: [Socket],
      _header: 'POST /api/asset/upload HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: multipart/form-data; boundary=--------------------------284543707829071763435817\r\n' +
        'x-api-key: <redacted>\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Host: 10.0.0.4:2283\r\n' +
        'Connection: close\r\n' +
        'Transfer-Encoding: chunked\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/asset/upload',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: '10.0.0.4',
      protocol: 'http:',
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: true,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    response: {
      status: 504,
      statusText: 'Gateway Time-out',
      headers: [Object],
      config: [Object],
      request: [ClientRequest],
      data: '<html>\r\n' +
        '<head><title>504 Gateway Time-out</title></head>\r\n' +
        '<body>\r\n' +
        '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
        '<hr><center>nginx/1.25.0</center>\r\n' +
        '</body>\r\n' +
        '</html>\r\n'
    },
    isAxiosError: true,
    toJSON: [Function: toJSON]
  },
  response: '<html>\r\n' +
    '<head><title>504 Gateway Time-out</title></head>\r\n' +
    '<body>\r\n' +
    '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
    '<hr><center>nginx/1.25.0</center>\r\n' +
    '</body>\r\n' +
    '</html>\r\n'
}
mikex99 commented 11 months ago

I'm having this issue on windows without using the cli, just uploading images from mobile app. When It happens, the web interface starts throwing error 500 and I cannot stop the immich_server. The immich_server is full of repeated logs. [Nest] 7 - 09/27/2023, 2:27:29 PM ERROR [ExceptionsHandler] Connection terminated due to connection timeout Error: Connection terminated due to connection timeout at Connection. (/usr/src/app/node_modules/pg/lib/client.js:132:73) at Object.onceWrapper (node:events:628:28) at Connection.emit (node:events:514:28) at Socket. (/usr/src/app/node_modules/pg/lib/connection.js:63:12) at Socket.emit (node:events:514:28) at TCP. (node:net:323:12)