martabal
commented
11 months ago Did you add / remove environment variables from the original template ? If no, can you make sure you have the right file permissions for /import and /photos ?
Closed iankaufmann closed 11 months ago
martabal
commented
11 months ago Did you add / remove environment variables from the original template ? If no, can you make sure you have the right file permissions for /import and /photos ?
iankaufmann
commented
11 months ago Did you add / remove environment variables from the original template ? If no, can you make sure you have the right file permissions for /import and /photos ?
The only environment variables I added were paths to the external libraries that I wanted to add.
Permissions on /import and /photos (from the UnRAID side) are nobody:users and 0777
I would think if it were a permissions issue, there would be issues loading the images 100% of the time instead of just sometimes. (Although, when this happens I also get errors on /api calls like "server-info").
Just now, I clicked the "Explore" tab and this is what the network tab looked like:
4 of the 12 person thumbnails just failed to load with 502.
martabal
commented
11 months ago Yeah, I'm asking because there was already reports of strange behaviors for those who limited the RAM usage. And lately the only errors with external libraries were related to file permissions but you configured it properly
iankaufmann
commented
11 months ago Interesting... well, yeah I'm not sure what I could be doing wrong. The external mappings are on the local array (not NFS or anything). Also this could be an upstream issue and not related to this project. I haven't tried the "official" install to compare. This repo is obviously more attractive to run on UnRAID vs. running so many connected containers (although it seems the official method is heading more in this direction on the next release).
Most searches for "immich 502" are people saying they get that no matter what, and can't even load the WebUI at all. I can't find any other threads where it is intermittent like this.
I'll scroll through the time line and it will work for a bit, all of a sudden a bunch of 502 errors, but then if I scroll the blank images out of the viewport and then back in, they load.
I wonder if there is some tweak to the internal nginx config I could make.
martabal
commented
11 months ago This error in your logs [hooks.server.ts]:handleError Not found: /sw.js let me think this is an error with the web, but you have 502 errors with the server config endpoint. Can you share your nginx logs (they are in /config/log/nginx ?
iankaufmann
commented
11 months ago OK, so this has been quite the journey the last couple hours. Thanks for taking interest and trying to help!
After looking in the nginx logs like you suggested, I saw a bunch of errors related to websockets. The weird thing was that it said "host: photos.{my-domain}".
But I haven't been accessing this via hostname (reverse proxy) since yesterday evening. As part of trying to debug this issue, I started using the ip:port directly just to rule out the proxy. I have no devices or browsers accessing Immich via the hostname/proxy.
But it looks like maybe... possibly... having the HAProxy backend configured on PFSense was "holding on" to a bunch of websocket connections? I have no idea why anything related to that hostname would be in the logs from the last couple hours.
I went into PFSense and I disabled the backend that was pointed to Immich... and I just spent the last 45 minutes flying through Immich (on the ip:port url) and tagging faces and I did not encounter a single 502 error the whole time. (In the past, it would error out every 3-4 attempts to tag someone)
Really weird, but it seems like something about HAProxy is not playing nice with Immich's websockets... flooding it with reconnects or something?
Nginx error.log:
2023/11/19 14:50:32 [error] 1030#1030: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/server-info HTTP/1.1", upstream: "http://127.0.0.1:3001/server-info", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 14:50:32 [error] 1030#1030: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/server-info HTTP/1.1", upstream: "http://127.0.0.1:3001/server-info", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 14:50:36 [error] 1030#1030: *1 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/socket.io/?EIO=4&transport=polling&t=OlfMNJf HTTP/1.1", upstream: "http://localhost/socket.io/?EIO=4&transport=polling&t=OlfMNJf", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 14:50:42 [error] 1030#1030: *1 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/socket.io/?EIO=4&transport=polling&t=OlfMOoG HTTP/1.1", upstream: "http://localhost/socket.io/?EIO=4&transport=polling&t=OlfMOoG", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 14:50:43 [error] 1030#1030: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/server-info HTTP/1.1", upstream: "http://127.0.0.1:3001/server-info", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 14:50:43 [error] 1030#1030: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/server-info HTTP/1.1", upstream: "http://127.0.0.1:3001/server-info", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 14:50:48 [error] 1030#1030: *1 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/socket.io/?EIO=4&transport=polling&t=OlfMQHC HTTP/1.1", upstream: "http://localhost/socket.io/?EIO=4&transport=polling&t=OlfMQHC", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 14:50:53 [error] 1030#1030: *1 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/server-info HTTP/1.1", upstream: "http://localhost/server-info", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 14:50:54 [error] 1030#1030: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/socket.io/?EIO=4&transport=polling&t=OlfMRlk HTTP/1.1", upstream: "http://127.0.0.1:3001/socket.io/?EIO=4&transport=polling&t=OlfMRlk", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 14:50:54 [error] 1030#1030: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/socket.io/?EIO=4&transport=polling&t=OlfMRlk HTTP/1.1", upstream: "http://127.0.0.1:3001/socket.io/?EIO=4&transport=polling&t=OlfMRlk", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 14:51:00 [error] 1030#1030: *1 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/socket.io/?EIO=4&transport=polling&t=OlfMTEQ HTTP/1.1", upstream: "http://localhost/socket.io/?EIO=4&transport=polling&t=OlfMTEQ", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 14:51:03 [error] 1030#1030: *1 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/server-info HTTP/1.1", upstream: "http://localhost/server-info", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 14:51:05 [error] 1030#1030: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.30, server: , request: "GET /photos HTTP/1.1", upstream: "http://127.0.0.1:3000/photos", host: "10.32.1.100:7777"
2023/11/19 14:51:05 [error] 1030#1030: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.30, server: , request: "GET /photos HTTP/1.1", upstream: "http://127.0.0.1:3000/photos", host: "10.32.1.100:7777"
2023/11/19 14:51:05 [error] 1030#1030: *1 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://localhost/favicon.ico", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 14:51:06 [error] 1030#1030: *1 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /photos HTTP/1.1", upstream: "http://localhost/photos", host: "10.32.1.100:7777"
2023/11/19 14:51:06 [error] 1030#1030: *1 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://localhost/favicon.ico", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 14:51:19 [error] 1031#1031: *10 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 14:51:19 [error] 1031#1031: *10 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 14:53:12 [error] 1031#1031: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.30, server: , request: "GET /photos HTTP/1.1", upstream: "http://127.0.0.1:3000/photos", host: "10.32.1.100:7777"
2023/11/19 14:53:12 [error] 1031#1031: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.30, server: , request: "GET /photos HTTP/1.1", upstream: "http://127.0.0.1:3000/photos", host: "10.32.1.100:7777"
2023/11/19 14:53:12 [error] 1031#1031: *1 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://localhost/favicon.ico", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 14:53:22 [error] 1032#1032: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 14:53:22 [error] 1032#1032: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 14:53:43 [error] 1033#1033: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/config HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/config", host: "photos.{my-domain}", referrer: "https://photos.{my-domain}/library/browse"
2023/11/19 14:53:43 [error] 1033#1033: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/config HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/config", host: "photos.{my-domain}", referrer: "https://photos.{my-domain}/library/browse"
2023/11/19 14:53:44 [error] 1033#1033: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /sw.js HTTP/1.1", upstream: "http://127.0.0.1:3000/sw.js", host: "photos.{my-domain}"
2023/11/19 14:53:44 [error] 1033#1033: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /sw.js HTTP/1.1", upstream: "http://127.0.0.1:3000/sw.js", host: "photos.{my-domain}"
2023/11/19 14:54:23 [error] 1034#1034: *12 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 14:54:23 [error] 1034#1034: *12 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 14:54:57 [error] 1035#1035: *15 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.30, server: , request: "GET /photos HTTP/1.1", upstream: "http://127.0.0.1:3000/photos", host: "10.32.1.100:7777"
2023/11/19 14:54:57 [error] 1035#1035: *15 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.30, server: , request: "GET /photos HTTP/1.1", upstream: "http://127.0.0.1:3000/photos", host: "10.32.1.100:7777"
2023/11/19 14:54:57 [error] 1035#1035: *15 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /photos HTTP/1.1", upstream: "http://localhost/photos", host: "10.32.1.100:7777"
2023/11/19 14:54:58 [error] 1035#1035: *15 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /photos HTTP/1.1", upstream: "http://localhost/photos", host: "10.32.1.100:7777"
2023/11/19 14:54:58 [error] 1035#1035: *15 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /photos HTTP/1.1", upstream: "http://localhost/photos", host: "10.32.1.100:7777"
2023/11/19 14:55:23 [error] 1036#1036: *18 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 14:55:23 [error] 1036#1036: *18 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 14:56:23 [error] 1031#1031: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 14:56:23 [error] 1031#1031: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 14:57:23 [error] 1032#1032: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 14:57:23 [error] 1032#1032: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 14:58:23 [error] 1026#1026: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 14:58:23 [error] 1026#1026: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 14:59:23 [error] 1027#1027: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 14:59:23 [error] 1027#1027: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 15:00:24 [error] 1028#1028: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 15:00:24 [error] 1028#1028: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 15:01:24 [error] 1029#1029: *10 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 15:01:24 [error] 1029#1029: *10 connect() failed (111: Connection refused) while connecting to upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 15:02:25 [error] 231#231: *29 upstream prematurely closed connection while reading response header from upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 15:02:26 [error] 231#231: *29 upstream prematurely closed connection while reading response header from upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 15:03:27 [error] 230#230: *334 upstream prematurely closed connection while reading response header from upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 15:03:28 [error] 230#230: *334 upstream prematurely closed connection while reading response header from upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 15:03:28 [error] 230#230: *336 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/asset/time-bucket?size=MONTH&isArchived=false&withStacked=true&withPartners=true&timeBucket=2021-12-01T00%3A00%3A00.000Z HTTP/1.1", upstream: "http://localhost/asset/time-bucket?size=MONTH&isArchived=false&withStacked=true&withPartners=true&timeBucket=2021-12-01T00%3A00%3A00.000Z", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 15:03:36 [error] 230#230: *336 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/server-info HTTP/1.1", upstream: "http://localhost/server-info", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 15:04:29 [error] 230#230: *350 upstream prematurely closed connection while reading response header from upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 15:04:30 [error] 230#230: *350 upstream prematurely closed connection while reading response header from upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 15:04:36 [error] 230#230: *344 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/server-info HTTP/1.1", upstream: "http://localhost/server-info", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"
2023/11/19 15:05:31 [error] 230#230: *358 upstream prematurely closed connection while reading response header from upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 15:05:32 [error] 230#230: *358 upstream prematurely closed connection while reading response header from upstream, client: 10.32.1.1, server: , request: "GET /api/v1/ws HTTP/1.1", upstream: "http://127.0.0.1:3001/v1/ws", host: "photos.{my-domain}"
2023/11/19 15:05:36 [error] 230#230: *344 no live upstreams while connecting to upstream, client: 10.32.1.30, server: , request: "GET /api/server-info HTTP/1.1", upstream: "http://localhost/server-info", host: "10.32.1.100:7777", referrer: "http://10.32.1.100:7777/photos"Nothing has shown up in the nginx error.log since I disabled the HAProxy backend.
The only thing I'm seeing now (in the logs for the Immich container) are a bunch of errors after tagging a face and merging it with another:
[migrations] started
[migrations] no migrations found
╔═══════════════════════════════╗
__ _____ _____ __
/ / |_ _/ ____| / /
/ / | || | __ / /
/ / | || | |_ | / /
/ / _| || |__| | / /
/_/ |_____\_____| /_/
Baseimage from linuxserver.io
╠═══════════════════════════════╣
To support this applications developer(s) visit:
Immich: https://immich.app/docs/overview/support-the-project
╠═══════════════════════════════╣
User/Group ID:
User UID: 99
User GID: 100
╚═══════════════════════════════╝
[custom-init] No custom files found, skipping...
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [NestFactory] Starting Nest application...
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +11ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [InstanceLoader] ConfigModule dependencies initialized +2ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +47ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [InstanceLoader] InfraModule dependencies initialized +1ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [InstanceLoader] AppModule dependencies initialized +2ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [InstanceLoader] DomainModule dependencies initialized +5ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] ActivityController {/activity}: +39ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/activity, GET} route +1ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/activity/statistics, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/activity, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/activity/:id, DELETE} route +1ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] AssetsController {/assets}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/assets, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] AssetController {/asset}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/map-marker, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/memory-lane, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/random, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/download/info, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/download/archive, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/download/:id, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/statistics, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/time-buckets, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/time-bucket, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/jobs, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset, PUT} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset, DELETE} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/restore, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/trash/empty, POST} route +1ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/trash/restore, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/stack/parent, PUT} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/:id, PUT} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] AssetController {/asset}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/upload, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/file/:id, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/thumbnail/:id, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/curated-objects, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/curated-locations, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/search-terms, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/:deviceId, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/assetById/:id, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/exist, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/asset/bulk-upload-check, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] AppController {/}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/refresh-config, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] AlbumController {/album}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/album/count, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/album, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/album, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/album/:id, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/album/:id, PATCH} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/album/:id, DELETE} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/album/:id/assets, PUT} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/album/:id/assets, DELETE} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/album/:id/users, PUT} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/album/:id/user/:userId, DELETE} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] APIKeyController {/api-key}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/api-key, POST} route +1ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/api-key, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/api-key/:id, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/api-key/:id, PUT} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/api-key/:id, DELETE} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] AuditController {/audit}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/audit/deletes, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/audit/file-report, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/audit/file-report/checksum, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/audit/file-report/fix, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] AuthController {/auth}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/auth/login, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/auth/admin-sign-up, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/auth/devices, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/auth/devices, DELETE} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/auth/devices/:id, DELETE} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/auth/validateToken, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/auth/change-password, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/auth/logout, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] JobController {/jobs}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/jobs, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/jobs/:id, PUT} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] LibraryController {/library}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/library, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/library, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/library/:id, PUT} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/library/:id, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/library/:id, DELETE} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/library/:id/statistics, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/library/:id/scan, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/library/:id/removeOffline, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] OAuthController {/oauth}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/oauth/mobile-redirect, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/oauth/config, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/oauth/authorize, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/oauth/callback, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/oauth/link, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/oauth/unlink, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] PartnerController {/partner}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/partner, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/partner/:id, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/partner/:id, PUT} route +1ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/partner/:id, DELETE} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] SearchController {/search}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/search/person, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/search, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/search/explore, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] ServerInfoController {/server-info}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/server-info, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/server-info/ping, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/server-info/version, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/server-info/features, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/server-info/theme, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/server-info/config, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/server-info/statistics, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/server-info/media-types, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] SharedLinkController {/shared-link}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/shared-link, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/shared-link/me, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/shared-link/:id, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/shared-link, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/shared-link/:id, PATCH} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/shared-link/:id, DELETE} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/shared-link/:id/assets, PUT} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/shared-link/:id/assets, DELETE} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] SystemConfigController {/system-config}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/system-config, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/system-config/defaults, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/system-config, PUT} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/system-config/storage-template-options, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/system-config/map/style.json, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] TagController {/tag}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/tag, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/tag, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/tag/:id, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/tag/:id, PATCH} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/tag/:id, DELETE} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/tag/:id/assets, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/tag/:id/assets, PUT} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/tag/:id/assets, DELETE} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] UserController {/user}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/user, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/user/info/:id, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/user/me, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/user, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/user/profile-image, DELETE} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/user/:id, DELETE} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/user/:id/restore, POST} route +1ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/user, PUT} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/user/profile-image, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/user/profile-image/:id, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RoutesResolver] PersonController {/person}: +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/person, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/person, PUT} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/person/:id, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/person/:id, PUT} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/person/:id/statistics, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/person/:id/thumbnail, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/person/:id/assets, GET} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [RouterExplorer] Mapped {/person/:id/merge, POST} route +0ms
[Nest] 194 - 11/19/2023, 3:37:27 PM LOG [SearchService] Running bootstrap
Request #1700429847705: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700429847705: Sleeping for 4s and then retrying request...
[Nest] 194 - 11/19/2023, 3:37:31 PM LOG [TypesenseRepository] Creating schema: assets/assets-v10
[Nest] 194 - 11/19/2023, 3:37:31 PM LOG [TypesenseRepository] Creating schema: albums/albums-v2
[Nest] 194 - 11/19/2023, 3:37:31 PM LOG [TypesenseRepository] Creating schema: faces/faces-v1
[Nest] 194 - 11/19/2023, 3:37:31 PM LOG [TypesenseRepository] Alias mapping: []
[Nest] 194 - 11/19/2023, 3:37:31 PM LOG [TypesenseRepository] Collections needing migration: {"assets":true,"albums":true,"faces":true}
[Nest] 194 - 11/19/2023, 3:37:32 PM LOG [AppService] Feature Flags: {
"clipEncode": true,
"facialRecognition": true,
"tagImage": true,
"map": true,
"reverseGeocoding": true,
"sidecar": true,
"search": true,
"trash": true,
"oauth": false,
"oauthAutoLaunch": false,
"passwordLogin": true,
"configFile": false
}
[Nest] 194 - 11/19/2023, 3:37:32 PM LOG [NestApplication] Nest application successfully started +3ms
[Nest] 194 - 11/19/2023, 3:37:32 PM LOG [ImmichServer] Immich Server is listening on http://[::1]:3001 [v1.87.0] [PRODUCTION]
[Nest] 194 - 11/19/2023, 3:37:32 PM LOG [CommunicationRepository] Websocket Connect: eqOzdxJjzBlKNcbYAAAB
[ig-init] done.
Listening on 0.0.0.0:3000
[Nest] 1552 - 11/19/2023, 3:37:36 PM LOG [NestFactory] Starting Nest application...
[Nest] 1552 - 11/19/2023, 3:37:36 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +10ms
[Nest] 1552 - 11/19/2023, 3:37:36 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 1552 - 11/19/2023, 3:37:36 PM LOG [InstanceLoader] ConfigHostModule dependencies initialized +0ms
[Nest] 1552 - 11/19/2023, 3:37:36 PM LOG [InstanceLoader] DiscoveryModule dependencies initialized +1ms
[Nest] 1552 - 11/19/2023, 3:37:36 PM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 1552 - 11/19/2023, 3:37:36 PM LOG [InstanceLoader] ConfigModule dependencies initialized +1ms
[Nest] 1552 - 11/19/2023, 3:37:36 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 1552 - 11/19/2023, 3:37:36 PM LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 1552 - 11/19/2023, 3:37:36 PM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +39ms
[Nest] 1552 - 11/19/2023, 3:37:36 PM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 1552 - 11/19/2023, 3:37:36 PM LOG [InstanceLoader] InfraModule dependencies initialized +1ms
[Nest] 1552 - 11/19/2023, 3:37:36 PM LOG [InstanceLoader] DomainModule dependencies initialized +5ms
[Nest] 1552 - 11/19/2023, 3:37:36 PM LOG [InstanceLoader] MicroservicesModule dependencies initialized +1ms
[11/19/23 15:37:36] INFO Starting gunicorn 21.2.0
[11/19/23 15:37:36] INFO Listening at: http://0.0.0.0:3003 (1546)
[11/19/23 15:37:36] INFO Using worker: uvicorn.workers.UvicornWorker
[11/19/23 15:37:36] INFO Booting worker with pid: 1656
[Nest] 1552 - 11/19/2023, 3:37:40 PM LOG [MetadataService] Initialized local reverse geocoder with cities500
[Nest] 1552 - 11/19/2023, 3:37:40 PM LOG [SearchService] Running bootstrap
[Nest] 1552 - 11/19/2023, 3:37:40 PM LOG [TypesenseRepository] Schema up to date: assets/assets-v10
[Nest] 1552 - 11/19/2023, 3:37:40 PM LOG [TypesenseRepository] Schema up to date: albums/albums-v2
[Nest] 1552 - 11/19/2023, 3:37:40 PM LOG [TypesenseRepository] Schema up to date: faces/faces-v1
[Nest] 1552 - 11/19/2023, 3:37:40 PM LOG [TypesenseRepository] Alias mapping: []
[Nest] 1552 - 11/19/2023, 3:37:40 PM LOG [TypesenseRepository] Collections needing migration: {"assets":true,"albums":true,"faces":true}
[Nest] 1552 - 11/19/2023, 3:37:40 PM LOG [NestApplication] Nest application successfully started +11ms
[Nest] 1552 - 11/19/2023, 3:37:40 PM LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.87.0] [PRODUCTION]
[Nest] 1552 - 11/19/2023, 3:37:40 PM LOG [SearchService] Indexing 0 albums
[Nest] 1552 - 11/19/2023, 3:37:40 PM LOG [TypesenseRepository] Using new schema: (unset) => albums-v2
[11/19/23 15:37:42] INFO Created in-memory cache with unloading after 300s
of inactivity.
[11/19/23 15:37:42] INFO Initialized request thread pool with 32 threads.
[Nest] 1552 - 11/19/2023, 3:37:45 PM LOG [SearchService] Indexing 19815 faces
[Nest] 194 - 11/19/2023, 3:37:50 PM LOG [CommunicationRepository] Websocket Disconnect: eqOzdxJjzBlKNcbYAAAB
[Nest] 194 - 11/19/2023, 3:37:51 PM ERROR [StreamableFile] Premature close
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:406:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:159:30)
at ServerResponse.emit (node:events:526:35)
at emitCloseNT (node:_http_server:1023:10)
at Socket.onServerResponseClose (node:_http_server:278:5)
at Socket.emit (node:events:526:35)
at TCP.<anonymous> (node:net:337:12)
[Nest] 194 - 11/19/2023, 3:37:51 PM ERROR [StreamableFile] Premature close
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:406:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:159:30)
at ServerResponse.emit (node:events:526:35)
at emitCloseNT (node:_http_server:1023:10)
at Socket.onServerResponseClose (node:_http_server:278:5)
at Socket.emit (node:events:526:35)
at TCP.<anonymous> (node:net:337:12)
[Nest] 194 - 11/19/2023, 3:37:52 PM ERROR [StreamableFile] Premature close
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:406:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:159:30)
at ServerResponse.emit (node:events:526:35)
at emitCloseNT (node:_http_server:1023:10)
at Socket.onServerResponseClose (node:_http_server:278:5)
at Socket.emit (node:events:526:35)
at TCP.<anonymous> (node:net:337:12)
[Nest] 194 - 11/19/2023, 3:37:52 PM ERROR [StreamableFile] Premature close
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:406:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:159:30)
at ServerResponse.emit (node:events:526:35)
at emitCloseNT (node:_http_server:1023:10)
at Socket.onServerResponseClose (node:_http_server:278:5)
at Socket.emit (node:events:526:35)
at TCP.<anonymous> (node:net:337:12)
[Nest] 194 - 11/19/2023, 3:37:52 PM ERROR [StreamableFile] Premature close
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:406:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:159:30)
at ServerResponse.emit (node:events:526:35)
at emitCloseNT (node:_http_server:1023:10)
at Socket.onServerResponseClose (node:_http_server:278:5)
at Socket.emit (node:events:526:35)
at TCP.<anonymous> (node:net:337:12)
[Nest] 194 - 11/19/2023, 3:37:52 PM ERROR [StreamableFile] Premature close
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:406:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:159:30)
at ServerResponse.emit (node:events:526:35)
at emitCloseNT (node:_http_server:1023:10)
at Socket.onServerResponseClose (node:_http_server:278:5)
at Socket.emit (node:events:526:35)
at TCP.<anonymous> (node:net:337:12)
[Nest] 194 - 11/19/2023, 3:37:52 PM ERROR [StreamableFile] Premature close
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:406:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:159:30)
at ServerResponse.emit (node:events:526:35)
at emitCloseNT (node:_http_server:1023:10)
at Socket.onServerResponseClose (node:_http_server:278:5)
at Socket.emit (node:events:526:35)
at TCP.<anonymous> (node:net:337:12)
[Nest] 194 - 11/19/2023, 3:37:52 PM ERROR [StreamableFile] Premature close
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:406:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:159:30)
at ServerResponse.emit (node:events:526:35)
at emitCloseNT (node:_http_server:1023:10)
at Socket.onServerResponseClose (node:_http_server:278:5)
at Socket.emit (node:events:526:35)
at TCP.<anonymous> (node:net:337:12)
[Nest] 194 - 11/19/2023, 3:37:52 PM ERROR [StreamableFile] Premature close
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:406:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:159:30)
at ServerResponse.emit (node:events:526:35)
at emitCloseNT (node:_http_server:1023:10)
at Socket.onServerResponseClose (node:_http_server:278:5)
at Socket.emit (node:events:526:35)
at TCP.<anonymous> (node:net:337:12)
[Nest] 194 - 11/19/2023, 3:37:53 PM ERROR [StreamableFile] Premature close
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:406:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:159:30)
at ServerResponse.emit (node:events:526:35)
at emitCloseNT (node:_http_server:1023:10)
at Socket.onServerResponseClose (node:_http_server:278:5)
at Socket.emit (node:events:526:35)
at TCP.<anonymous> (node:net:337:12)
[Nest] 194 - 11/19/2023, 3:37:53 PM ERROR [StreamableFile] Premature close
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:406:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:159:30)
at ServerResponse.emit (node:events:526:35)
at emitCloseNT (node:_http_server:1023:10)
at Socket.onServerResponseClose (node:_http_server:278:5)
at Socket.emit (node:events:526:35)
at TCP.<anonymous> (node:net:337:12)
Request #1700429871109: Request to Node 0 failed due to "ERR_REQUEST_ABORTED error request aborted"
Request #1700429871109: Sleeping for 4s and then retrying request...
Request #1700429872062: Request to Node 0 failed due to "ECONNRESET read ECONNRESET"
Request #1700429872062: Sleeping for 4s and then retrying request...
[Nest] 194 - 11/19/2023, 3:37:53 PM ERROR [StreamableFile] Premature close
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:406:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:159:30)
at ServerResponse.emit (node:events:526:35)
at emitCloseNT (node:_http_server:1023:10)
at Socket.onServerResponseClose (node:_http_server:278:5)
at Socket.emit (node:events:526:35)
at TCP.<anonymous> (node:net:337:12)
[Nest] 194 - 11/19/2023, 3:37:53 PM ERROR [StreamableFile] Premature close
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:406:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:159:30)
at ServerResponse.emit (node:events:526:35)
at emitCloseNT (node:_http_server:1023:10)
at Socket.onServerResponseClose (node:_http_server:278:5)
at Socket.emit (node:events:526:35)
at TCP.<anonymous> (node:net:337:12)
[Nest] 194 - 11/19/2023, 3:37:53 PM ERROR [StreamableFile] Premature close
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:406:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:159:30)
at ServerResponse.emit (node:events:526:35)
at emitCloseNT (node:_http_server:1023:10)
at Socket.onServerResponseClose (node:_http_server:278:5)
at Socket.emit (node:events:526:35)
at TCP.<anonymous> (node:net:337:12)
[Nest] 194 - 11/19/2023, 3:37:53 PM ERROR [StreamableFile] Premature close
Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
at new NodeError (node:internal/errors:406:5)
at ServerResponse.onclose (node:internal/streams/end-of-stream:159:30)
at ServerResponse.emit (node:events:526:35)
at emitCloseNT (node:_http_server:1023:10)
at Socket.onServerResponseClose (node:_http_server:278:5)
at Socket.emit (node:events:526:35)
at TCP.<anonymous> (node:net:337:12)
[Nest] 194 - 11/19/2023, 3:37:54 PM LOG [CommunicationRepository] Websocket Connect: PAlJVkkLohHwpx_vAAAD
Request #1700429871109: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700429871109: Sleeping for 4s and then retrying request...
Request #1700429872062: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700429872062: Sleeping for 4s and then retrying request...
Request #1700429871109: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1700429871109: Sleeping for 4s and then retrying request...
Request #1700429884826: Request to Node 0 failed due to "ECONNREFUSED connect ECONNREFUSED 127.0.0.1:8108"
Request #1700429884826: Sleeping for 4s and then retrying request...
Request #1700429871109: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700429871109: Sleeping for 4s and then retrying request...
Request #1700429884826: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700429884826: Sleeping for 4s and then retrying request...
[Nest] 194 - 11/19/2023, 3:38:13 PM LOG [PersonService] Merging 2b98525d-4a98-46d2-8c50-8f30d57080d2 into *redacted*
[Nest] 194 - 11/19/2023, 3:38:13 PM LOG [PersonService] Merged 2b98525d-4a98-46d2-8c50-8f30d57080d2 into *redacted*
[Nest] 1552 - 11/19/2023, 3:38:21 PM LOG [SearchService] Indexing 19815 faces
[Nest] 1552 - 11/19/2023, 3:38:24 PM LOG [TypesenseRepository] Using new schema: (unset) => faces-v1
[Nest] 194 - 11/19/2023, 3:38:30 PM LOG [PersonService] Merging d4fcfdd0-6dea-47b9-9560-de95998fad6a into *redacted*
[Nest] 194 - 11/19/2023, 3:38:30 PM LOG [PersonService] Merged d4fcfdd0-6dea-47b9-9560-de95998fad6a into *redacted*
[Nest] 1552 - 11/19/2023, 3:38:30 PM LOG [TypesenseRepository] Using new schema: (unset) => assets-v10
[Nest] 1552 - 11/19/2023, 3:38:35 PM LOG [SearchService] Indexing 19815 faces
[Nest] 194 - 11/19/2023, 3:38:50 PM LOG [PersonService] Merging Matt Bastion into Matt Bastian
[Nest] 194 - 11/19/2023, 3:38:50 PM LOG [PersonService] Merged Matt Bastion into Matt Bastian
[Nest] 1552 - 11/19/2023, 3:38:55 PM LOG [SearchService] Indexing 19815 faces
[Nest] 1552 - 11/19/2023, 3:38:56 PM LOG [TypesenseRepository] Using new schema: faces-v1 => faces-v1
[Nest] 194 - 11/19/2023, 3:38:57 PM LOG [PersonService] Merging d0ad8bb9-1c59-434f-b962-63f101952eed into *redacted*
[Nest] 194 - 11/19/2023, 3:38:57 PM LOG [PersonService] Merged d0ad8bb9-1c59-434f-b962-63f101952eed into *redacted*
[Nest] 194 - 11/19/2023, 3:39:02 PM LOG [PersonService] Merging fe4e4027-af76-4653-b006-348c7a846279 into *redacted*
[Nest] 194 - 11/19/2023, 3:39:02 PM LOG [PersonService] Merged fe4e4027-af76-4653-b006-348c7a846279 into *redacted*
[Nest] 1552 - 11/19/2023, 3:39:03 PM LOG [SearchService] Indexing 19815 faces
[Nest] 194 - 11/19/2023, 3:39:08 PM LOG [PersonService] Merging d45d66de-5d2f-421f-bbee-b07cf8672760 into *redacted*
[Nest] 194 - 11/19/2023, 3:39:08 PM LOG [PersonService] Merged d45d66de-5d2f-421f-bbee-b07cf8672760 into *redacted*
[Nest] 1552 - 11/19/2023, 3:39:09 PM LOG [SearchService] Indexing 19815 faces
Request #1700429949964: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1700429949964: Sleeping for 4s and then retrying request...
Request #1700429951301: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1700429951301: Sleeping for 4s and then retrying request...
[Nest] 1552 - 11/19/2023, 3:39:17 PM LOG [SearchService] Indexing 19815 faces
[Nest] 194 - 11/19/2023, 3:39:22 PM LOG [PersonService] Merging 659c1ff0-c5de-492a-b76d-df2c3d787b88 into *redacted*
[Nest] 194 - 11/19/2023, 3:39:22 PM LOG [PersonService] Merged 659c1ff0-c5de-492a-b76d-df2c3d787b88 into *redacted*
[Nest] 1552 - 11/19/2023, 3:39:31 PM LOG [TypesenseRepository] Using new schema: faces-v1 => faces-v1
[Nest] 1552 - 11/19/2023, 3:39:38 PM LOG [SearchService] Indexing 19815 faces
[Nest] 1552 - 11/19/2023, 3:40:07 PM LOG [TypesenseRepository] Using new schema: faces-v1 => faces-v1
Request #1700430009134: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1700430009134: Sleeping for 4s and then retrying request...
Request #1700430008439: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1700430008439: Sleeping for 4s and then retrying request...
Request #1700430008933: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1700430008933: Sleeping for 4s and then retrying request...
[Nest] 194 - 11/19/2023, 3:40:09 PM LOG [PersonService] Merging b67ef847-e79a-41b7-ae64-398fdf6a1272 into *redacted*
Request #1700430008094: Request to Node 0 failed due to "ERR_REQUEST_ABORTED error request aborted"
Request #1700430008094: Sleeping for 4s and then retrying request...
[Nest] 194 - 11/19/2023, 3:40:09 PM LOG [PersonService] Merged b67ef847-e79a-41b7-ae64-398fdf6a1272 into *redacted*
Request #1700430009610: Request to Node 0 failed due to "ECONNREFUSED connect ECONNREFUSED 127.0.0.1:8108"
Request #1700430009610: Sleeping for 4s and then retrying request...
Request #1700430009134: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009134: Sleeping for 4s and then retrying request...
Request #1700430008439: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430008439: Sleeping for 4s and then retrying request...
Request #1700430008933: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430008933: Sleeping for 4s and then retrying request...
Request #1700430008094: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430008094: Sleeping for 4s and then retrying request...
Request #1700430009610: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009610: Sleeping for 4s and then retrying request...
Request #1700430008439: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430008439: Sleeping for 4s and then retrying request...
Request #1700430008933: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430008933: Sleeping for 4s and then retrying request...
Request #1700430008094: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430008094: Sleeping for 4s and then retrying request...
Request #1700430009610: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009610: Sleeping for 4s and then retrying request...
Request #1700430008094: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430008094: Sleeping for 4s and then retrying request...
Request #1700430009610: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009610: Sleeping for 4s and then retrying request...
Request #1700430009610: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009610: Sleeping for 4s and then retrying request...
Request #1700430009610: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009610: Sleeping for 4s and then retrying request...
Request #1700430008933: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1700430008933: Sleeping for 4s and then retrying request...
Request #1700430009610: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009610: Sleeping for 4s and then retrying request...
Request #1700430008933: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430008933: Sleeping for 4s and then retrying request...
Request #1700430008094: Request to Node 0 failed due to "ECONNRESET socket hang up"
Request #1700430008094: Sleeping for 4s and then retrying request...
Request #1700430008933: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430008933: Sleeping for 4s and then retrying request...
Request #1700430008094: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430008094: Sleeping for 4s and then retrying request...
Request #1700430008094: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430008094: Sleeping for 4s and then retrying request...
Request #1700430009610: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
Request #1700430009610: Sleeping for 4s and then retrying request...
Request #1700430009610: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009610: Sleeping for 4s and then retrying request...
Request #1700430009610: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009610: Sleeping for 4s and then retrying request...
Request #1700430009610: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009610: Sleeping for 4s and then retrying request...
Request #1700430009610: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009610: Sleeping for 4s and then retrying request...
Request #1700430009610: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009610: Sleeping for 4s and then retrying request...
Request #1700430009610: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009610: Sleeping for 4s and then retrying request...
Request #1700430009134: Request to Node 0 failed due to "ECONNRESET read ECONNRESET"
Request #1700430009134: Sleeping for 4s and then retrying request...
Request #1700430009610: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009610: Sleeping for 4s and then retrying request...
Request #1700430009134: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009134: Sleeping for 4s and then retrying request...
Request #1700430008439: Request to Node 0 failed due to "ECONNRESET read ECONNRESET"
Request #1700430008439: Sleeping for 4s and then retrying request...
Request #1700430009134: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009134: Sleeping for 4s and then retrying request...
Request #1700430008439: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430008439: Sleeping for 4s and then retrying request...
Request #1700430009134: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430009134: Sleeping for 4s and then retrying request...
Request #1700430008439: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700430008439: Sleeping for 4s and then retrying request...
Request #1700430009610: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
Request #1700430009610: Sleeping for 4s and then retrying request...
[Nest] 1552 - 11/19/2023, 3:41:33 PM ERROR [JobService] Unable to run job handler (search/search-index-faces): Error: timeout of 10000ms exceeded
[Nest] 1552 - 11/19/2023, 3:41:33 PM ERROR [JobService] Error: timeout of 10000ms exceeded
at createError (/app/immich/server/node_modules/typesense/node_modules/axios/lib/core/createError.js:16:15)
at RedirectableRequest.handleRequestTimeout (/app/immich/server/node_modules/typesense/node_modules/axios/lib/adapters/http.js:369:16)
at RedirectableRequest.emit (node:events:514:28)
at Timeout.<anonymous> (/app/immich/server/node_modules/follow-redirects/index.js:179:12)
at listOnTimeout (node:internal/timers:573:17)
at process.processTimers (node:internal/timers:514:7)
[Nest] 1552 - 11/19/2023, 3:41:33 PM ERROR [JobService] Object:
{}
Request #1700430008933: Request to Node 0 failed due to "ECONNRESET read ECONNRESET"
Request #1700430008933: Sleeping for 4s and then retrying request...Even though all of those errors are there... I didn't notice any issues on the frontend. But it seems to be trying to connect to something every 4 seconds and failing.
martabal
commented
11 months ago Yeah sounds like a network issue. Immich uses websockets and it seems you have issues with it. I never used HaProxy so I can't really help you ...
Running this on UnRAID with 3 containers: this, postgres15, redis
Everything works perfectly... when it works. I have no issues getting the webUI to load or anything, and generally it will work smoothly for a few minutes. But then all of a sudden I get popups saying "Failed to load assets". From that point on, for about 30 seconds to a minute, nothing works (browsing, loading thumbnails).
The requests all fail with 502 Bad Gateway.
I have ruled out it being because I'm behind a reverse-proxy (although if I can get this working, I would like to put it behind HaProxy on PFSense). This is happening when accessing via the ip:port directly.
It's just like it gets "overwhelmed" every so often (quite often) and then if I wait a bit and reload everything is back.
I am running this on a machine with a 13900k and 64gb of ram. The machine isn't breaking a sweat.
There are no running jobs, and I've restarted the containers a few times.
Also not seeing much in the logs.
I just can't figure out what could be going on. It's showing enough promise that I really want to use this, but the 502 errors make it unusable, and it also causes the Android companion app to log out once one of these errors is encountered.
I am loading one "external" library with about 15k photos in it.