search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
52.02k stars 2.76k forks source link

[BUG] Request failed with status code 503 #2608

Closed zenithyr closed 1 year ago

zenithyr commented 1 year ago

The bug

The application worked perfectly fine in docker until yesterday when I upgraded to the latest with the added typesense feature. After docker pull & docker compose up -d, the web portal now complains "Request failed with status code 503".

Tried to use the latest yaml and .env files with no luck. Commenting out typesense from docker-compose.yml and .env completely resulted the same error.

UPLOAD_LOCATION does point to a folder filled with data from previously. Not sure if it's related.

Please find container logs in the sections below.

The OS that Immich Server is running on

OpenMediaVault 6.1.0-0.deb11.7-amd64

Version of Immich Server

ghcr.io/immich-app/immich-server:release

Version of Immich Mobile App

not used here

Platform with the issue

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:release
    command: ["start-server.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:release
    command: ["start-microservices.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:release
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:release
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.0
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
    logging:
      driver: none
    volumes:
      - tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:release
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

###################################################################################
# Database
###################################################################################

# NOTE: The following four database variables support Docker secrets by adding a *_FILE suffix to the variable name
# See the docker-compose documentation on secrets for additional details: https://docs.docker.com/compose/compose-file/compose-file-v3/#secrets
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=postgres
DB_DATABASE_NAME=immich

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich_redis

# REDIS_URL will be used to pass custom options to ioredis.
# Example for Sentinel
# {"sentinels":[{"host":"redis-sentinel-node-0","port":26379},{"host":"redis-sentinel-node-1","port":26379},{"host":"redis-sentinel-node-2","port":26379}],"name":"redis-sentinel"}
# REDIS_URL=ioredis://eyJzZW50aW5lbHMiOlt7Imhvc3QiOiJyZWRpcy1zZW50aW5lbDEiLCJwb3J0IjoyNjM3OX0seyJob3N0IjoicmVkaXMtc2VudGluZWwyIiwicG9ydCI6MjYzNzl9XSwibmFtZSI6Im15bWFzdGVyIn0=

# Optional Redis settings:

# Note: these parameters are not automatically passed to the Redis Container
# to do so, please edit the docker-compose.yml file as well. Redis is not configured
# via environment variables, only redis.conf or the command line

# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_USERNAME=
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION=<omitted for sensitivity>

###################################################################################
# Typesense
###################################################################################
TYPESENSE_API_KEY=some_random_text
# TYPESENSE_ENABLED=false
# TYPESENSE_URL uses base64 encoding for the nodes json.
# Example JSON that was used:
# [
#      { 'host': 'typesense-1.example.net', 'port': '443', 'protocol': 'https' },
#      { 'host': 'typesense-2.example.net', 'port': '443', 'protocol': 'https' },
#      { 'host': 'typesense-3.example.net', 'port': '443', 'protocol': 'https' },
#  ]
# TYPESENSE_URL=ha://WwogICAgeyAnaG9zdCc6ICd0eXBlc2Vuc2UtMS5leGFtcGxlLm5ldCcsICdwb3J0JzogJzQ0MycsICdwcm90b2NvbCc6ICdodHRwcycgfSwKICAgIHsgJ2hvc3QnOiAndHlwZXNlbnNlLTIuZXhhbXBsZS5uZXQnLCAncG9ydCc6ICc0NDMnLCAncHJvdG9jb2wnOiAnaHR0cHMnIH0sCiAgICB7ICdob3N0JzogJ3R5cGVzZW5zZS0zLmV4YW1wbGUubmV0JywgJ3BvcnQnOiAnNDQzJywgJ3Byb3RvY29sJzogJ2h0dHBzJyB9LApd

###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

# DISABLE_REVERSE_GEOCODING=false
# REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>demo@demo.de</i><br>Password: <i>demo</i>"
####################################################################################

PUBLIC_LOGIN_PAGE_MESSAGE=

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

####################################################################################
# Alternative API's External Address - Optional
#
# This is an advanced feature used to control the public server endpoint returned to clients during Well-known discovery.
# You should only use this if you want mobile apps to access the immich API over a custom URL. Do not include trailing slash.
# NOTE: At this time, the web app will not be affected by this setting and will continue to use the relative path: /api
# Examples: http://localhost:3001, http://immich-api.example.com, etc
####################################################################################

#IMMICH_API_URL_EXTERNAL=http://localhost:3001

Reproduction steps

1.`docker compose down`

[+] Running 9/9
 ✔ Container immich_machine_learning  Removed                                                                                                                                                                                             1.6s
 ✔ Container immich_web               Removed                                                                                                                                                                                            10.7s
 ✔ Container immich_proxy             Removed                                                                                                                                                                                             0.6s
 ✔ Container immich_microservices     Removed                                                                                                                                                                                            10.8s
 ✔ Container immich_server            Removed                                                                                                                                                                                            10.7s
 ✔ Container immich_typesense         Removed                                                                                                                                                                                             2.3s
 ✔ Container immich_redis             Removed                                                                                                                                                                                             0.7s
 ✔ Container immich_postgres          Removed                                                                                                                                                                                             0.8s
 ✔ Network immich_default             Removed

2.`docker volume prune`

```bash
WARNING! This will remove anonymous local volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Volumes:
69eb48024894f73603c532e308b4bd96e345bb2ef880457b01ff1609407161e6
acd4231a655103ff269cc402d85340530637af2acfb74058a4c9c097d4204416
f657a14cc239edd8d9731d92146701d5cb163ebe6e5f1230657a49535ff20eae
2d373eb5dceee436ad5db6638897a48ef42473a83593d95e4efd0547ca5e5a6b
c6f89118528443ba258fb991cf0fd0a766c4bf01e6f12b1eb6ce94700059357d

3.docker compose up -d

[+] Building 0.0s (0/0)
[+] Running 9/9
 ✔ Network immich_default             Created                                                                                                                                                                                             0.1s
 ✔ Container immich_postgres          Started                                                                                                                                                                                             1.8s
 ✔ Container immich_web               Started                                                                                                                                                                                             1.7s
 ✔ Container immich_machine_learning  Started                                                                                                                                                                                             2.1s
 ✔ Container immich_typesense         Started                                                                                                                                                                                             1.7s
 ✔ Container immich_redis             Started                                                                                                                                                                                             2.2s
 ✔ Container immich_microservices     Started                                                                                                                                                                                             3.1s
 ✔ Container immich_server            Started                                                                                                                                                                                             3.2s
 ✔ Container immich_proxy             Started
  1. curl http://127.0.0.1:2283 # or from remotely by http://<ip>:2283
<omitted for brevity>
        <body>
                <div class="error">
                        <span class="status">500</span>
                        <div class="message">
                                <h1>Request failed with status code 503</h1>
                        </div>
                </div>
        </body>

### Additional information

proxy
<details></summary>immich_proxy log</summary>

```bash
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-set-env-variables.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2023/05/30 01:56:17 [notice] 1#1: using the "epoll" event method
2023/05/30 01:56:17 [notice] 1#1: nginx/1.23.4
2023/05/30 01:56:17 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
2023/05/30 01:56:17 [notice] 1#1: OS: Linux 6.1.0-0.deb11.7-amd64
2023/05/30 01:56:17 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2023/05/30 01:56:17 [notice] 1#1: start worker processes
2023/05/30 01:56:17 [notice] 1#1: start worker process 40
2023/05/30 01:56:17 [notice] 1#1: start worker process 41
2023/05/30 01:56:17 [notice] 1#1: start worker process 42
2023/05/30 01:56:17 [notice] 1#1: start worker process 43

server

immich_server log ```bash [Nest] 1 - 05/30/2023, 2:01:11 AM LOG [RouterExplorer] Mapped {/album/count-by-user-id, GET} route +0ms [Nest] 1 - 05/30/2023, 2:01:11 AM LOG [RouterExplorer] Mapped {/album/:id/users, PUT} route +0ms [Nest] 1 - 05/30/2023, 2:01:11 AM LOG [RouterExplorer] Mapped {/album/:id/assets, PUT} route +1ms [Nest] 1 - 05/30/2023, 2:01:11 AM LOG [RouterExplorer] Mapped {/album/:id, GET} route +0ms [Nest] 1 - 05/30/2023, 2:01:11 AM LOG [RouterExplorer] Mapped {/album/:id/assets, DELETE} route +0ms [Nest] 1 - 05/30/2023, 2:01:11 AM LOG [RouterExplorer] Mapped {/album/:id/user/:userId, DELETE} route +1ms [Nest] 1 - 05/30/2023, 2:01:11 AM LOG [RouterExplorer] Mapped {/album/:id/download, GET} route +0ms [Nest] 1 - 05/30/2023, 2:01:11 AM LOG [RouterExplorer] Mapped {/album/create-shared-link, POST} route +0ms [Nest] 1 - 05/30/2023, 2:01:11 AM LOG [SearchService] Running bootstrap Request #1685412071893: Request to Node 0 failed due to "undefined Request failed with HTTP code 503" Request #1685412071893: Sleeping for 4s and then retrying request... Request #1685412071893: Request to Node 0 failed due to "undefined Request failed with HTTP code 503" ```

microservers

immich_microservers log ```bash [Nest] 1 - 05/30/2023, 1:56:20 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized +1ms [Nest] 1 - 05/30/2023, 1:56:20 AM WARN [MetadataExtractionProcessor] Reverse geocoding is enabled [Nest] 1 - 05/30/2023, 1:56:20 AM LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding [Nest] 1 - 05/30/2023, 1:56:36 AM LOG [MetadataExtractionProcessor] Reverse Geocoding Initialized [Nest] 1 - 05/30/2023, 1:56:36 AM LOG [NestApplication] Nest application successfully started +53ms [Nest] 1 - 05/30/2023, 1:56:36 AM LOG [ImmichMicroservice] Running Immich Microservices in PRODUCTION environment - version 1.58.0 - Listening on port: 3002 ```

web

immich_web log ```bash Listening on 0.0.0.0:3000 ```

typesense

immich_typesense log ```bash Error response from daemon: configured logging driver does not support reading ```
zenithyr commented 1 year ago

curl http://<ip>:2283/api/server-info/stats returns

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.23.4</center>
</body>
</html>
alextran1502 commented 1 year ago

Can you put in the .env file LOG_LEVEL=debug and see if you can pull additional information from the server container? It looks like the server cannot start up because, somehow the docker network in your stack cannot communicate with Typesense.

Is it your standard procedure to perform docker compose down then docker volume prune?

alextran1502 commented 1 year ago

UPLOAD_LOCATION does point to a folder filled with data from previously. Not sure if it's related.

Is it the same data though?

zenithyr commented 1 year ago

Thanks for the quick turnaround.

Can you put in the .env file LOG_LEVEL=debug and see if you can pull additional information from the server container? It looks like the server cannot start up because, somehow the docker network in your stack cannot communicate with Typesense.

  1. echo "LOG_LEVEL=debug" >> .env. Then ensured the line is present in the file.
  2. tail -n5 .env 
    
# Examples: http://localhost:3001, http://immich-api.example.com, etc
####################################################################################

IMMICH_API_URL_EXTERNAL=http://localhost:3001

LOG_LEVEL=debug


3. `docker compose down; docker compose up -d`
4. `docker ps | grep immich-server`
```bash
dcbcd7fac27c   ghcr.io/immich-app/immich-server:release             "/bin/sh start-micro…"   22 seconds ago      Up 17 seconds           3001/tcp                                                                                                                          immich_microservices
3bc421247963   ghcr.io/immich-app/immich-server:release             "/bin/sh start-serve…"   22 seconds ago      Up 17 seconds           3001/tcp                                                                                                                          immich_server
  1. docker logs 3bc421247963 
    [Nest] 1  - 05/30/2023, 2:53:01 AM     LOG [RouterExplorer] Mapped {/album/:id, GET} route +0ms
[Nest] 1  - 05/30/2023, 2:53:01 AM     LOG [RouterExplorer] Mapped {/album/:id/assets, DELETE} route +1ms
[Nest] 1  - 05/30/2023, 2:53:01 AM     LOG [RouterExplorer] Mapped {/album/:id/user/:userId, DELETE} route +0ms
[Nest] 1  - 05/30/2023, 2:53:01 AM     LOG [RouterExplorer] Mapped {/album/:id/download, GET} route +1ms
[Nest] 1  - 05/30/2023, 2:53:01 AM     LOG [RouterExplorer] Mapped {/album/create-shared-link, POST} route +0ms
[Nest] 1  - 05/30/2023, 2:53:01 AM     LOG [SearchService] Running bootstrap
Request #1685415181852: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685415181852: Sleeping for 4s and then retrying request...
Request #1685415181852: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685415181852: Sleeping for 4s and then retrying request...
Request #1685415181852: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
Request #1685415181852: Sleeping for 4s and then retrying request...
Request #1685415181852: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
Request #1685415181852: Sleeping for 4s and then retrying request...
Request #1685415181852: Request to Node 0 failed due to "ECONNABORTED timeout of 10000ms exceeded"
Request #1685415181852: Sleeping for 4s and then retrying request...

seems DEBUG level isn't working. Anything I was missing?

Is it your standard procedure to perform docker compose down then docker volume prune?

No, just this once in order to have a clean volume. I usually do docker compose down followed by docker compose up -d

dhanadhan commented 1 year ago

Same, well not really. I am getting 502 bad gateway after upgrading to 1.58.0. Re-pulled the new image and changed the docker-compose lines as mentioned in the release note, but now the web page won't load nor the android app would load my server url :(

Edit: my status page correctly shows the content info, but not loading the web page even with localip:2283 My issue is probably isolated, but not sure what it is after upgrading to 1.58 from 1.57.

Edit 2: solved it by adding the line 45 "entrypoint: ["/bin/sh", "./entrypoint.sh"]" back in to docker-compose.

zenithyr commented 1 year ago

UPLOAD_LOCATION does point to a folder filled with data from previously. Not sure if it's related.

Is it the same data though?

Yes. But in case it's the culprit, I changed it to a new path, restarted the container, and got the same error :( After restart I can see the new directory being created with nothing in it.

Please instruct how to print out more logs, thanks.

alextran1502 commented 1 year ago

@dhanadhan Have you read the release note to make the appropriate changes to the docker-compose file since we changed a few things in it?

dhanadhan commented 1 year ago

@dhanadhan Have you read the release note to make the appropriate changes to the docker-compose file since we changed a few things in it?

Yes, there were 3 changes. It didn't load the web page. But I have just resolved it by adding the line 45 back in.

alextran1502 commented 1 year ago

@dhanadhan It wasn't the intention to keep that line and have the container working, it looks like you haven't pulled the new version?

zenithyr commented 1 year ago

In case it helps, this is the list of containers. They all look normal.

CONTAINER ID   IMAGE                                                COMMAND                   CREATED              STATUS                  PORTS                                                                                                                             NAMES
beb903abea2f   ghcr.io/immich-app/immich-proxy:release              "/docker-entrypoint.…"   About a minute ago   Up About a minute       0.0.0.0:2283->8080/tcp, :::2283->8080/tcp                                                                                         immich_proxy
dd7a61c63d65   ghcr.io/immich-app/immich-server:release             "/bin/sh start-serve…"   About a minute ago   Up About a minute       3001/tcp                                                                                                                          immich_server
655dabb49f9c   ghcr.io/immich-app/immich-server:release             "/bin/sh start-micro…"   About a minute ago   Up About a minute       3001/tcp                                                                                                                          immich_microservices
294c8e350e51   postgres:14                                          "docker-entrypoint.s…"   About a minute ago   Up About a minute       5432/tcp                                                                                                                          immich_postgres
4c8c30da73da   ghcr.io/immich-app/immich-machine-learning:release   "python src/main.py"      About a minute ago   Up About a minute                                                                                                                                         immich_machine_learning
8dcefe72d626   ghcr.io/immich-app/immich-web:release                "/bin/sh entrypoint.…"   About a minute ago   Up About a minute       3000/tcp                                                                                                                          immich_web
5076a8cc60e5   typesense/typesense:0.24.0                           "/opt/typesense-serv…"   20 minutes ago       Up 20 minutes           8108/tcp                                                                                                                          immich_typesense
4d9843482874   redis:6.2                                            "docker-entrypoint.s…"   20 minutes ago       Up 20 minutes           6379/tcp                                                                                                                          immich_redis
alextran1502 commented 1 year ago

@zenithyr Can you remove the logging directive in the typesense container in the docker-compose file? Your typesense container has the issue.

FYI, after using docker compose down it could remove all the containers so that when you perform docker volume prune.

In case it helps, this is the list of containers. They all look normal.

This doesn't look normal as the typesense and redis container doesn't get restart with the same timestamp as other containers. Please bring all the containers down, and make sure that typesense and redis get brought down as well so when you bring up the whole stack, they would be able to communicate with each other.

dhanadhan commented 1 year ago

@dhanadhan It wasn't the intention to keep that line and have the container working, it looks like you haven't pulled the new version?

Pretty sure i pulled the new image. Web page shows v1.58.0 correctly on the bottom left and working as of now with having line 45 "entrypoint: ["/bin/sh", "./entrypoint.sh"]" in the docker compose. Do I keep this change? It will probably stop working if I remove it.

zenithyr commented 1 year ago

@zenithyr Can you remove the logging directive in the typesense container in the docker-compose file? Your typesense container has the issue.

FYI, after using docker compose down it could remove all the containers so that when you perform docker volume prune.

In case it helps, this is the list of containers. They all look normal.

This doesn't look normal as the typesense and redis container doesn't get restart with the same timestamp as other containers. Please bring all the containers down, and make sure that typesense and redis get brought down as well so when you bring up the whole stack, they would be able to communicate with each other.

  1. Update docker-compose.yml

    typesense:
container_name: immich_typesense
image: typesense/typesense:0.24.0
environment:
  - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
  - TYPESENSE_DATA_DIR=/data
#logging:
#  driver: none
volumes:
  - tsdata:/data
restart: always

  2. made sure to have down redis and typesense then up again.

    
docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9cbb6c142af1 ghcr.io/immich-app/immich-proxy:release "/docker-entrypoint.…" 7 seconds ago Up 2 seconds 0.0.0.0:2283->8080/tcp, :::2283->8080/tcp immich_proxy d6054f9db2e0 ghcr.io/immich-app/immich-server:release "/bin/sh start-micro…" 7 seconds ago Up 4 seconds 3001/tcp immich_microservices a22e338d218f ghcr.io/immich-app/immich-server:release "/bin/sh start-serve…" 7 seconds ago Up 3 seconds 3001/tcp immich_server 01d7b7e89f54 redis:6.2 "docker-entrypoint.s…" 8 seconds ago Up 5 seconds 6379/tcp immich_redis 13b047cbec76 postgres:14 "docker-entrypoint.s…" 8 seconds ago Up 5 seconds 5432/tcp immich_postgres c49927afb7d4 ghcr.io/immich-app/immich-machine-learning:release "python src/main.py" 8 seconds ago Up 5 seconds immich_machine_learning 5b561bf8dbc2 ghcr.io/immich-app/immich-web:release "/bin/sh entrypoint.…" 8 seconds ago Up 5 seconds 3000/tcp immich_web a32e40169bb4 typesense/typesense:0.24.0 "/opt/typesense-serv…" 8 seconds ago Up 5 seconds 8108/tcp immich_typesense


The issue persists :(. 
typesense logs looks like below

```bash
I20230530 03:21:37.153359   165 raft_meta.cpp:546] Saved single stable meta, path /data/state/meta term 21 votedfor 0.0.0.0:0:0 time: 9569
I20230530 03:21:47.154302   165 raft_server.cpp:545] Term: 21, last_index index: 10, committed_index: 0, known_applied_index: 3, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
W20230530 03:21:47.154343   165 raft_server.cpp:569] Single-node with no leader. Resetting peers.
I20230530 03:21:47.723843   169 node.cpp:1484] node default_group:192.168.16.3:8107:8108 term 21 start pre_vote
I20230530 03:21:47.723978   169 node.cpp:1549] node default_group:192.168.16.3:8107:8108 term 21 start vote and grant vote self
I20230530 03:21:47.740236   169 raft_meta.cpp:546] Saved single stable meta, path /data/state/meta term 22 votedfor 192.168.16.3:8107:8108 time: 16131
I20230530 03:21:47.740278   169 node.cpp:1783] node default_group:192.168.16.3:8107:8108 term 22 become leader of group 192.168.16.3:8107:8108
I20230530 03:21:47.748555   169 raft_server.h:273] Configuration of this group is 172.23.0.3:8107:8108
I20230530 03:21:47.748601   169 raft_server.h:273] Configuration of this group is 172.25.0.5:8107:8108
I20230530 03:21:47.748620   169 raft_server.h:273] Configuration of this group is 172.26.0.6:8107:8108
I20230530 03:21:47.748637   169 raft_server.h:273] Configuration of this group is 172.27.0.5:8107:8108
I20230530 03:21:47.748653   169 raft_server.h:273] Configuration of this group is 172.30.0.3:8107:8108
I20230530 03:21:47.748670   169 raft_server.h:273] Configuration of this group is 172.31.0.3:8107:8108
I20230530 03:21:47.748687   169 raft_server.h:273] Configuration of this group is 192.168.0.5:8107:8108
I20230530 03:21:47.748694   169 raft_server.h:273] Configuration of this group is 192.168.16.3:8107:8108
I20230530 03:21:47.748701   169 node.cpp:3142] node default_group:192.168.16.3:8107:8108 reset ConfigurationCtx, new_peers: 192.168.16.3:8107:8108, old_peers: 192.168.16.3:8107:8108
I20230530 03:21:47.748711   169 raft_server.h:256] Node becomes leader, term: 22
I20230530 03:21:57.155246   165 raft_server.cpp:545] Term: 22, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
I20230530 03:21:57.155386   207 raft_server.h:60] Peer refresh succeeded!
I20230530 03:22:07.156229   165 raft_server.cpp:545] Term: 22, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
I20230530 03:22:07.156322   207 raft_server.h:60] Peer refresh succeeded!
I20230530 03:22:17.157153   165 raft_server.cpp:545] Term: 22, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
I20230530 03:22:17.157231   207 raft_server.h:60] Peer refresh succeeded!
I20230530 03:22:27.158087   165 raft_server.cpp:545] Term: 22, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
I20230530 03:22:27.158147   207 raft_server.h:60] Peer refresh succeeded!
I20230530 03:22:37.159025   165 raft_server.cpp:545] Term: 22, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
I20230530 03:22:37.159099   207 raft_server.h:60] Peer refresh succeeded!
I20230530 03:22:38.035077   166 batched_indexer.cpp:279] Running GC for aborted requests, req map size: 0
I20230530 03:22:47.159945   165 raft_server.cpp:545] Term: 22, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
I20230530 03:22:47.160013   207 raft_server.h:60] Peer refresh succeeded!
I20230530 03:22:57.160825   165 raft_server.cpp:545] Term: 22, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
I20230530 03:22:57.160929   207 raft_server.h:60] Peer refresh succeeded!
I20230530 03:23:07.161732   165 raft_server.cpp:545] Term: 22, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
I20230530 03:23:07.161800   207 raft_server.h:60] Peer refresh succeeded!
I20230530 03:23:17.162842   165 raft_server.cpp:545] Term: 22, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
I20230530 03:23:17.162955   207 raft_server.h:60] Peer refresh succeeded!
I20230530 03:23:27.163966   165 raft_server.cpp:545] Term: 22, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
I20230530 03:23:27.164098   207 raft_server.h:60] Peer refresh succeeded!
I20230530 03:23:37.165251   165 raft_server.cpp:545] Term: 22, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
I20230530 03:23:37.165385   207 raft_server.h:60] Peer refresh succeeded!
I20230530 03:23:39.040940   166 batched_indexer.cpp:279] Running GC for aborted requests, req map size: 0
I20230530 03:23:47.166237   165 raft_server.cpp:545] Term: 22, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
I20230530 03:23:47.166321   207 raft_server.h:60] Peer refresh succeeded!
I20230530 03:23:57.167263   165 raft_server.cpp:545] Term: 22, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
I20230530 03:23:57.167353   207 raft_server.h:60] Peer refresh succeeded!
I20230530 03:24:07.168507   165 raft_server.cpp:545] Term: 22, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
I20230530 03:24:07.168588   207 raft_server.h:60] Peer refresh succeeded!
I20230530 03:24:17.169873   165 raft_server.cpp:545] Term: 22, last_index index: 11, committed_index: 11, known_applied_index: 11, applying_index: 0, queued_writes: 0, pending_queue_size: 0, local_sequence: 1
I20230530 03:24:17.170038   207 raft_server.h:60] Peer refresh succeeded!
alextran1502 commented 1 year ago

@zenithyr What is the log now for the server? Maybe try restart your system?

zenithyr commented 1 year ago

@zenithyr What is the log now for the server? Maybe try restart your system?

Same as before

[Nest] 1  - 05/30/2023, 3:58:39 AM     LOG [RouterExplorer] Mapped {/album/:id, GET} route +0ms
[Nest] 1  - 05/30/2023, 3:58:39 AM     LOG [RouterExplorer] Mapped {/album/:id/assets, DELETE} route +0ms
[Nest] 1  - 05/30/2023, 3:58:39 AM     LOG [RouterExplorer] Mapped {/album/:id/user/:userId, DELETE} route +1ms
[Nest] 1  - 05/30/2023, 3:58:39 AM     LOG [RouterExplorer] Mapped {/album/:id/download, GET} route +0ms
[Nest] 1  - 05/30/2023, 3:58:39 AM     LOG [RouterExplorer] Mapped {/album/create-shared-link, POST} route +1ms
[Nest] 1  - 05/30/2023, 3:58:39 AM     LOG [SearchService] Running bootstrap
Request #1685419119631: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685419119631: Sleeping for 4s and then retrying request...

I then tried to set TYPESENSE_ENABLED=false in the .env file, and commented out typesense from docker-compose completely. Down the stack and up again. I saw no error from the server

[Nest] 1  - 05/30/2023, 4:04:52 AM     LOG [RouterExplorer] Mapped {/album/create-shared-link, POST} route +1ms
[Nest] 1  - 05/30/2023, 4:04:52 AM     LOG [NestApplication] Nest application successfully started +46ms
[Nest] 1  - 05/30/2023, 4:04:52 AM     LOG [ImmichServer] Running Immich Server in PRODUCTION environment - version 1.58.0 - Listening on port: 3001
[Nest] 1  - 05/30/2023, 4:04:52 AM    WARN [ImmichServer] Machine learning is enabled
[Nest] 1  - 05/30/2023, 4:04:52 AM    WARN [ImmichServer] Search is disabled

But this time, while I still got the same 503 error by curl http://<ip>:2283, I got some response on curl http://<ip>:2283/api/server-info/stats: {"statusCode":401,"message":"Authentication required","error":"Unauthorized"}

It looks like the server cannot start up because, somehow the docker network in your stack cannot communicate with Typesense.

So is it safe to assume the 503 isn't related to typesense?

alextran1502 commented 1 year ago

Okay, why don't we do this, please follow exactly the commands below.

  1. docker compose down
  2. docker container prune
  3. docker compose pull
  4. docker compose up

Then let me know if all the logs look good.

Ah I notice you did docker pull which is not the correct command to pull updates for all containers. It has to be docker compose pull

zenithyr commented 1 year ago

Based on the previous configurations, with typesense disabled:

user@omv ~/d/immich> docker compose down

Bring down ```bash [+] Running 8/8 ✔ Container immich_proxy Removed 1.0s ✔ Container immich_microservices Removed 11.0s ✔ Container immich_machine_learning Removed 1.9s ✔ Container immich_web Removed 10.9s ✔ Container immich_server Removed 10.8s ✔ Container immich_postgres Removed 0.8s ✔ Container immich_redis Removed 0.9s ✔ Network immich_default Removed 0.4s ```

user@omv ~/d/immich> docker container prune

Prue ```bash WARNING! This will remove all stopped containers. Are you sure you want to continue? [y/N] y Total reclaimed space: 0B ```

user@omv ~/d/immich> docker compose pull

Pull. It failed the first time so I retried. ```bash [+] Pulling 7/7 ✔ immich-microservices Skipped - Image is already being pulled by immich-server 0.0s ✔ database Pulled 5.7s ✘ immich-machine-learning Error 10.1s ✔ immich-web Pulled 3.7s ✔ redis Pulled 6.1s ✔ immich-proxy Pulled 3.7s ✔ immich-server Pulled 3.7s Error response from daemon: Get "https://ghcr.io/v2/": net/http: TLS handshake timeout user@omv ~/d/immich [18]> docker compose pull [+] Pulling 7/7 ✔ immich-microservices Skipped - Image is already being pulled by immich-server 0.0s ✔ database Pulled 14.2s ✔ immich-proxy Pulled 13.6s ✔ immich-web Pulled 13.6s ✔ immich-server Pulled 13.6s ✔ redis Pulled 14.2s ✔ immich-machine-learning Pulled 13.5s ```

user@omv ~/d/immich> docker compose up

Bring up ```bash [+] Building 0.0s (0/0) [+] Running 8/8 ✔ Network immich_default Created 0.2s ✔ Container immich_machine_learning Created 0.2s ✔ Container immich_postgres Created 0.2s ✔ Container immich_web Created 0.2s ✔ Container immich_redis Created 0.3s ✔ Container immich_microservices Created 0.1s ✔ Container immich_server Created 0.1s ✔ Container immich_proxy Created 0.1s Attaching to immich_machine_learning, immich_microservices, immich_postgres, immich_proxy, immich_redis, immich_server, immich_web immich_postgres | immich_postgres | PostgreSQL Database directory appears to contain a database; Skipping initialization immich_postgres | immich_postgres | 2023-05-30 04:19:20.701 UTC [1] LOG: starting PostgreSQL 14.8 (Debian 14.8-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit immich_postgres | 2023-05-30 04:19:20.701 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 immich_postgres | 2023-05-30 04:19:20.701 UTC [1] LOG: listening on IPv6 address "::", port 5432 immich_redis | 1:C 30 May 2023 04:19:20.710 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo immich_redis | 1:C 30 May 2023 04:19:20.710 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=1, just started immich_redis | 1:C 30 May 2023 04:19:20.710 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf immich_redis | 1:M 30 May 2023 04:19:20.711 * monotonic clock: POSIX clock_gettime immich_redis | 1:M 30 May 2023 04:19:20.713 * Running mode=standalone, port=6379. immich_redis | 1:M 30 May 2023 04:19:20.713 # Server initialized immich_redis | 1:M 30 May 2023 04:19:20.713 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. immich_redis | 1:M 30 May 2023 04:19:20.713 * Ready to accept connections immich_postgres | 2023-05-30 04:19:20.728 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" immich_postgres | 2023-05-30 04:19:20.754 UTC [28] LOG: database system was shut down at 2023-05-30 04:18:12 UTC immich_postgres | 2023-05-30 04:19:20.800 UTC [1] LOG: database system is ready to accept connections immich_web | Listening on 0.0.0.0:3000 immich_proxy | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration immich_proxy | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ immich_proxy | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh immich_proxy | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf immich_proxy | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-set-env-variables.envsh immich_proxy | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh immich_proxy | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf immich_proxy | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh immich_proxy | /docker-entrypoint.sh: Configuration complete; ready for start up immich_proxy | 2023/05/30 04:19:24 [notice] 1#1: using the "epoll" event method immich_proxy | 2023/05/30 04:19:24 [notice] 1#1: nginx/1.23.4 immich_proxy | 2023/05/30 04:19:24 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) immich_proxy | 2023/05/30 04:19:24 [notice] 1#1: OS: Linux 6.1.0-0.deb11.7-amd64 immich_proxy | 2023/05/30 04:19:24 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 immich_proxy | 2023/05/30 04:19:24 [notice] 1#1: start worker processes immich_proxy | 2023/05/30 04:19:24 [notice] 1#1: start worker process 40 immich_proxy | 2023/05/30 04:19:24 [notice] 1#1: start worker process 41 immich_proxy | 2023/05/30 04:19:24 [notice] 1#1: start worker process 42 immich_proxy | 2023/05/30 04:19:24 [notice] 1#1: start worker process 43 immich_microservices | [Nest] 1 - 05/30/2023, 4:19:26 AM LOG [NestFactory] Starting Nest application... immich_microservices | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +116ms immich_microservices | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +2ms immich_microservices | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +1ms immich_microservices | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] ConfigModule dependencies initialized +24ms immich_microservices | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] BullModule dependencies initialized +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [NestFactory] Starting Nest application... immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +98ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] BullModule dependencies initialized +2ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] DownloadModule dependencies initialized +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] DiscoveryModule dependencies initialized +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] ConfigModule dependencies initialized +17ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] BullModule dependencies initialized +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] BullModule dependencies initialized +1ms immich_microservices | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +335ms immich_microservices | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms immich_microservices | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] InfraModule dependencies initialized +15ms immich_microservices | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] DomainModule dependencies initialized +0ms immich_microservices | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] MicroservicesModule dependencies initialized +2ms immich_microservices | [Nest] 1 - 05/30/2023, 4:19:27 AM WARN [MetadataExtractionProcessor] Reverse geocoding is enabled immich_microservices | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding immich_machine_learning | INFO: Started server process [1] immich_machine_learning | INFO: Waiting for application startup. immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +455ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] AlbumModule dependencies initialized +13ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] TagModule dependencies initialized +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] AssetModule dependencies initialized +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] InfraModule dependencies initialized +6ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] DomainModule dependencies initialized +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:27 AM LOG [InstanceLoader] AppModule dependencies initialized +6ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] AppController {/}: +507ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/refresh-config, POST} route +7ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] AlbumController {/album}: +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/album, GET} route +2ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/album, POST} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/album/:id, PATCH} route +2ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/album/:id, DELETE} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] APIKeyController {/api-key}: +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/api-key, POST} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/api-key, GET} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/api-key/:id, GET} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/api-key/:id, PUT} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/api-key/:id, DELETE} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] AssetController {/asset}: +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/map-marker, GET} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] AuthController {/auth}: +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/auth/login, POST} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/auth/admin-sign-up, POST} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/auth/devices, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/auth/devices, DELETE} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/auth/devices/:id, DELETE} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/auth/validateToken, POST} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/auth/change-password, POST} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/auth/logout, POST} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] JobController {/jobs}: +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/jobs, GET} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/jobs/:jobId, PUT} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] OAuthController {/oauth}: +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/oauth/mobile-redirect, GET} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/oauth/config, POST} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/oauth/callback, POST} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/oauth/link, POST} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/oauth/unlink, POST} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] PartnerController {/partner}: +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/partner, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/partner/:id, POST} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/partner/:id, DELETE} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] SearchController {/search}: +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/search, GET} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/search/config, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/search/explore, GET} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] ServerInfoController {/server-info}: +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/server-info, GET} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/server-info/ping, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/server-info/version, GET} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/server-info/stats, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] SharedLinkController {/share}: +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/share, GET} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/share/me, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/share/:id, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/share/:id, DELETE} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/share/:id, PATCH} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] SystemConfigController {/system-config}: +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/system-config, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/system-config/defaults, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/system-config, PUT} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/system-config/storage-template-options, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] UserController {/user}: +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/user, GET} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/user/info/:userId, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/user/me, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/user, POST} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/user/count, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/user/:userId, DELETE} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/user/:userId/restore, POST} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/user, PUT} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/user/profile-image, POST} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/user/profile-image/:userId, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] PersonController {/person}: +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/person, GET} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/person/:id, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/person/:id, PUT} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/person/:id/thumbnail, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/person/:id/assets, GET} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] AssetController {/asset}: +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/upload, POST} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/download/:assetId, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/download-files, POST} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/download-library, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/file/:assetId, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/thumbnail/:assetId, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/curated-objects, GET} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/curated-locations, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/search-terms, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/search, POST} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/count-by-time-bucket, POST} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/count-by-user-id, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/stat/archive, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset, GET} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/time-bucket, POST} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/:deviceId, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/assetById/:assetId, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/:assetId, PUT} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset, DELETE} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/check, POST} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/exist, POST} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/bulk-upload-check, POST} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/shared-link, POST} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/shared-link/add, PATCH} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/asset/shared-link/remove, PATCH} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] TagController {/tag}: +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/tag, POST} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/tag, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/tag/:id, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/tag/:id, PATCH} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/tag/:id, DELETE} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RoutesResolver] AlbumController {/album}: +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/album/count-by-user-id, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/album/:id/users, PUT} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/album/:id/assets, PUT} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/album/:id, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/album/:id/assets, DELETE} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/album/:id/user/:userId, DELETE} route +1ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/album/:id/download, GET} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [RouterExplorer] Mapped {/album/create-shared-link, POST} route +0ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [NestApplication] Nest application successfully started +47ms immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM LOG [ImmichServer] Running Immich Server in PRODUCTION environment - version 1.58.0 - Listening on port: 3001 immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM WARN [ImmichServer] Machine learning is enabled immich_server | [Nest] 1 - 05/30/2023, 4:19:28 AM WARN [ImmichServer] Search is disabled immich_machine_learning | Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. immich_machine_learning | The `max_size` parameter is deprecated and will be removed in v4.26. Please specify in `size['longest_edge'] instead`. immich_machine_learning | Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration. immich_machine_learning | /opt/venv/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:54: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider' immich_machine_learning | warnings.warn( immich_machine_learning | Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} immich_machine_learning | model ignore: /cache/models/buffalo_l/1k3d68.onnx landmark_3d_68 immich_machine_learning | Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} immich_machine_learning | model ignore: /cache/models/buffalo_l/2d106det.onnx landmark_2d_106 immich_machine_learning | Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} immich_machine_learning | find model: /cache/models/buffalo_l/det_10g.onnx detection [1, 3, '?', '?'] 127.5 128.0 immich_machine_learning | Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} immich_machine_learning | model ignore: /cache/models/buffalo_l/genderage.onnx genderage immich_machine_learning | Applied providers: ['CPUExecutionProvider'], with options: {'CPUExecutionProvider': {}} immich_machine_learning | find model: /cache/models/buffalo_l/w600k_r50.onnx recognition ['None', 3, 112, 112] 127.5 127.5 immich_machine_learning | set det-size: (640, 640) immich_machine_learning | INFO: Application startup complete. immich_machine_learning | INFO: Uvicorn running on http://0.0.0.0:3003 (Press CTRL+C to quit) immich_microservices | [Nest] 1 - 05/30/2023, 4:19:44 AM LOG [MetadataExtractionProcessor] Reverse Geocoding Initialized immich_microservices | [Nest] 1 - 05/30/2023, 4:19:44 AM LOG [NestApplication] Nest application successfully started +39ms immich_microservices | [Nest] 1 - 05/30/2023, 4:19:44 AM LOG [ImmichMicroservice] Running Immich Microservices in PRODUCTION environment - version 1.58.0 - Listening on port: 3002 ```

And then refresh the webpage or curl, got nothing but that status code 503...

jrasm91 commented 1 year ago

Your immich-server is starting up fine. Looking at the logs it looks like the immich-proxy is started before immich-server is ready to accept connections, which is why you get a 503. Just restart the proxy container (after everything else has started) and you should be good to go.

zenithyr commented 1 year ago

Your immich-server is starting up fine. Looking at the logs it looks like the immich-proxy is started before immich-server is ready to accept connections, which is why you get a 503. Just restart the proxy container (after everything else has started) and you should be good to go.

Thanks for jumping in. I tried the following.

  1. docker ps
ONTAINER ID   IMAGE                                                COMMAND                   CREATED          STATUS                  PORTS                                                                                                                             NAMES
dc12bee5c3f5   ghcr.io/immich-app/immich-proxy:release              "/docker-entrypoint.…"   14 minutes ago   Up 3 seconds            0.0.0.0:2283->8080/tcp, :::2283->8080/tcp                                                                                         immich_proxy
6e684e1fc572   ghcr.io/immich-app/immich-server:release             "/bin/sh start-serve…"   14 minutes ago   Up 5 seconds            3001/tcp                                                                                                                          immich_server
cc434814b002   ghcr.io/immich-app/immich-server:release             "/bin/sh start-micro…"   14 minutes ago   Up 4 seconds            3001/tcp                                                                                                                          immich_microservices
59d170627db2   redis:6.2                                            "docker-entrypoint.s…"   14 minutes ago   Up 6 seconds            6379/tcp                                                                                                                          immich_redis
91d664d8b569   ghcr.io/immich-app/immich-web:release                "/bin/sh entrypoint.…"   14 minutes ago   Up 6 seconds            3000/tcp                                                                                                                          immich_web
08b07ae8acfb   postgres:14                                          "docker-entrypoint.s…"   14 minutes ago   Up 6 seconds            5432/tcp                                                                                                                          immich_postgres
b2380714154f   ghcr.io/immich-app/immich-machine-learning:release   "python src/main.py"      14 minutes ago   Up 6 seconds                                                                                                                                              immich_machine_learning
  1. docker container restart dc12bee5c3f5
  2. docker ps
CONTAINER ID   IMAGE                                                COMMAND                   CREATED          STATUS                  PORTS                                                                                                                             NAMES
dc12bee5c3f5   ghcr.io/immich-app/immich-proxy:release              "/docker-entrypoint.…"   14 minutes ago   Up 1 second             0.0.0.0:2283->8080/tcp, :::2283->8080/tcp                                                                                         immich_proxy
6e684e1fc572   ghcr.io/immich-app/immich-server:release             "/bin/sh start-serve…"   14 minutes ago   Up 20 seconds           3001/tcp                                                                                                                          immich_server
cc434814b002   ghcr.io/immich-app/immich-server:release             "/bin/sh start-micro…"   14 minutes ago   Up 20 seconds           3001/tcp                                                                                                                          immich_microservices
59d170627db2   redis:6.2                                            "docker-entrypoint.s…"   14 minutes ago   Up 21 seconds           6379/tcp                                                                                                                          immich_redis
91d664d8b569   ghcr.io/immich-app/immich-web:release                "/bin/sh entrypoint.…"   14 minutes ago   Up 21 seconds           3000/tcp                                                                                                                          immich_web
08b07ae8acfb   postgres:14                                          "docker-entrypoint.s…"   14 minutes ago   Up 21 seconds           5432/tcp                                                                                                                          immich_postgres
b2380714154f   ghcr.io/immich-app/immich-machine-learning:release   "python src/main.py"      14 minutes ago   Up 21 seconds                                                                                                                                             immich_machine_learning

So the proxy is indeed restarted. And that 503 still didn't go away....

jrasm91 commented 1 year ago

What are the latest immich-web logs and what hardware are you running this on?

zenithyr commented 1 year ago

What are the latest immich-web logs and what hardware are you running this on?

docker logs 91d664d8b569
Listening on 0.0.0.0:3000
Listening on 0.0.0.0:3000

I am running this on top of the OpenMediaVault 6.1.0-0.deb11.7-amd64, which sits on PromoxVE. Memory usage 8.7G/24G. The UPLOAD_LOCATION is on a RAID-0 ZFS drive.

There are some other docker containers like qBitTorrent, PhotoView running fine so far.

Things I can think of:

user@omv ~/d/immich> uname -r
6.1.0-0.deb11.7-amd64

user@omv ~/d/immich> lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   36 bits physical, 48 bits virtual
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           58
Model name:                      Intel(R) Xeon(R) CPU E3-1265L V2 @ 2.50GHz
Stepping:                        9
CPU MHz:                         2500.016
BogoMIPS:                        5000.03
Virtualization:                  VT-x
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       128 KiB
L1i cache:                       128 KiB
L2 cache:                        16 MiB
L3 cache:                        16 MiB
NUMA node0 CPU(s):               0-3
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Mitigation; PTE Inversion; VMX flush not necessary, SMT disabled
Vulnerability Mds:               Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Mmio stale data:   Unknown: No mitigations
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Unknown: Dependent on hypervisor status
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_f
                                 req pni pclmulqdq vmx ssse3 cx16 pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cpuid_fault pti tpr_shadow vnmi flexpriority ept vpid fsg
                                 sbase tsc_adjust smep erms xsaveopt arat umip arch_capabilities

user@omv ~/d/immich> grep MemTotal /proc/meminfo
MemTotal:       24610032 kB

user@omv ~/d/immich> free -h
       total        used        free      shared  buff/cache   available
       23Gi       8.6Gi        13Gi        18Mi       1.8Gi        14Gi
      974Mi          0B       974Mi

user@omv ~/d/immich> lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Device 1234:1111 (rev 02)
00:03.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon
00:05.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
00:12.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:1e.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
00:1f.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
01:01.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI
01:02.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI
01:03.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI
01:04.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI
02:1b.0 USB controller: Red Hat, Inc. QEMU XHCI Host Controller (rev 01)

One thing to note is that I have set up http proxy for docker server

user@omv ~/d/immich> sudo cat /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://<my local proxy ip>:<my port>/"
Environment="HTTPS_PROXY=http://<my local proxy ip>:<my port>/"
Environment="NO_PROXY=localhost,127.0.0.1,.example.com"

But even if I commented out above settings and run sudo systemctl daemon-reload;sudo systemctl restart docker. The 503 error is still there.

Before the version upgrade (I forgot the old version), everything went on pretty well. And there hasn't been any hardware changes in between.

alextran1502 commented 1 year ago

Can you please try to setup a new instance of Immich using stock setup? I have a feeling your proxy settings cause the issue

zenithyr commented 1 year ago

Can you please try to setup a new instance of Immich using stock setup? I have a feeling your proxy settings cause the issue

Sure. I dropped all proxy entries in the http-proxy.conf file. 

user@omv ~> docker info | grep -I proxy # to confirm there is no proxy now.
(no output)

restarted by docker compose down and up -d. Then

user@omv ~> curl http://127.0.0.1:2283 | tail -n10
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1304  100  1304    0     0    793      0  0:00:01  0:00:01 --:--:--   793
        </head>
        <body>
                <div class="error">
                        <span class="status">500</span>
                        <div class="message">
                                <h1>Request failed with status code 503</h1>
                        </div>
                </div>
        </body>
</html>

Seems nobody else ran into similar issues that I am having. I will try to revert to an older version later. Will keep you posted. Please also let me know if you have anything else to try. Thanks.

jrasm91 commented 1 year ago

Where is the 503 error coming from? You should see it in the web logs, proxy logs, or server logs. If those are clean then it is coming from something in front of immich-proxy.

zenithyr commented 1 year ago

Older version didn't work either. So I dropped everything including images, and start refresh from docker compose pull.

Then I follow the guidance by docker logs immich_server --follow. After the whole stack is up, even before I make any web request, the server keeps printing out errors, and ends with something below

[Nest] 1  - 05/30/2023, 4:17:06 PM     LOG [SearchService] Running bootstrap
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
Request #1685463426275: Request to Node 0 failed due to "undefined Request failed with HTTP code 503"
Request #1685463426275: Sleeping for 4s and then retrying request...
/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23
        var _this = _super.call(this, message) || this;
                           ^

ServerError: Request failed with HTTP code 503
    at ServerError.TypesenseError [as constructor] (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/TypesenseError.js:23:28)
    at new ServerError (/usr/src/app/node_modules/typesense/lib/Typesense/Errors/ServerError.js:25:42)
    at ApiCall.customErrorForResponse (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:347:21)
    at /usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:204:58
    at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:33:23)
    at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
    at step (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:18:139)
    at Object.next (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:14:53)
    at fulfilled (/usr/src/app/node_modules/typesense/lib/Typesense/ApiCall.js:5:58)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  httpStatus: 503
}

I wonder if it's possible to find what that mystery request is so I can reproduce easier by go into the running container. Any hint is welcome.

jrasm91 commented 1 year ago

This isn't a mystery request, this is the server trying to connect to typesense on start up.

zenithyr commented 1 year ago

I tested the same set up on a brand new virtual host and there is no problem. Closing the ticket for now.

zenithyr commented 1 year ago

OK I found it. @alextran1502 is correct, the 503 comes from the proxy.

Thanks to @jrasm91 who helped look into it!

qrzbing commented 1 year ago

I met same problem too, and immich works after disabled proxy. How can I let proxy and immich work together?

AndyXheli commented 12 months ago

Hey @alextran1502 Im starting to see this on v1.88.2 i updated from v1.88.1 and now seem to take a bit for th server to come up. Then after a 1 or 2 web ui starts to work

immich_server            | Request #1700678670571: Sleeping for 4s and then retrying request...
immich_microservices     | [Nest] 7  - 11/22/2023, 6:44:43 PM     LOG [MetadataService] Initialized local reverse geocoder with cities500
immich_microservices     | [Nest] 7  - 11/22/2023, 6:44:43 PM     LOG [SearchService] Running bootstrap
immich_microservices     | Request #1700678683661: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_microservices     | Request #1700678683661: Sleeping for 4s and then retrying request...
immich_server            | Request #1700678670571: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server            | Request #1700678670571: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1700678683661: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_microservices     | Request #1700678683661: Sleeping for 4s and then retrying request...
immich_server            | Request #1700678670571: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server            | Request #1700678670571: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1700678683661: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_microservices     | Request #1700678683661: Sleeping for 4s and then retrying request...
immich_server            | Request #1700678670571: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server            | Request #1700678670571: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1700678683661: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_microservices     | Request #1700678683661: Sleeping for 4s and then retrying request...
immich_server            | Request #1700678670571: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server            | Request #1700678670571: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1700678683661: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_microservices     | Request #1700678683661: Sleeping for 4s and then retrying request...
immich_server            | Request #1700678670571: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server            | Request #1700678670571: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1700678683661: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_microservices     | Request #1700678683661: Sleeping for 4s and then retrying request...
immich_server            | Request #1700678670571: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server            | Request #1700678670571: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1700678683661: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_microservices     | Request #1700678683661: Sleeping for 4s and then retrying request...
immich_server            | Request #1700678670571: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server            | Request #1700678670571: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1700678683661: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_microservices     | Request #1700678683661: Sleeping for 4s and then retrying request...
immich_server            | Request #1700678670571: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server            | Request #1700678670571: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1700678683661: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_microservices     | Request #1700678683661: Sleeping for 4s and then retrying request...
immich_server            | Request #1700678670571: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server            | Request #1700678670571: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1700678683661: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_microservices     | Request #1700678683661: Sleeping for 4s and then retrying request...
immich_typesense         | W20231122 18:45:21.251355   646 controller.cpp:1454] SIGINT was installed with 1
immich_typesense         | W20231122 18:45:21.251394   646 raft_server.cpp:570] Single-node with no leader. Resetting peers.
immich_typesense         | W20231122 18:45:21.251399   646 node.cpp:894] node default_group:172.19.0.3:8107:8108 set_peer from 172.20.0.3:8107:8108 to 172.19.0.3:8107:8108
immich_server            | Request #1700678670571: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server            | Request #1700678670571: Sleeping for 4s and then retrying request...
immich_microservices     | Request #1700678683661: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_microservices     | Request #1700678683661: Sleeping for 4s and then retrying request...
immich_server            | Request #1700678670571: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
immich_server            | Request #1700678670571: Sleeping for 4s and then retrying request...
immich_microservices     | [Nest] 7  - 11/22/2023, 6:45:27 PM     LOG [TypesenseRepository] Schema up to date: assets/assets-v10
immich_microservices     | [Nest] 7  - 11/22/2023, 6:45:27 PM     LOG [TypesenseRepository] Schema up to date: albums/albums-v2
immich_microservices     | [Nest] 7  - 11/22/2023, 6:45:27 PM     LOG [TypesenseRepository] Schema up to date: faces/faces-v1
immich_microservices     | [Nest] 7  - 11/22/2023, 6:45:27 PM     LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}]
immich_microservices     | [Nest] 7  - 11/22/2023, 6:45:27 PM     LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false}
immich_microservices     | [Nest] 7  - 11/22/2023, 6:45:27 PM     LOG [NestApplication] Nest application successfully started +21ms
immich_microservices     | [Nest] 7  - 11/22/2023, 6:45:27 PM     LOG [ImmichMicroservice] Immich Microservices is listening on http://[::1]:3002 [v1.88.2] [PRODUCTION]
immich_server            | [Nest] 7  - 11/22/2023, 6:45:30 PM     LOG [TypesenseRepository] Schema up to date: assets/assets-v10
immich_server            | [Nest] 7  - 11/22/2023, 6:45:30 PM     LOG [TypesenseRepository] Schema up to date: albums/albums-v2
immich_server            | [Nest] 7  - 11/22/2023, 6:45:30 PM     LOG [TypesenseRepository] Schema up to date: faces/faces-v1
immich_server            | [Nest] 7  - 11/22/2023, 6:45:30 PM     LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}]
immich_server            | [Nest] 7  - 11/22/2023, 6:45:30 PM     LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false}
immich_server            | [Nest] 7  - 11/22/2023, 6:45:30 PM     LOG [AppService] Feature Flags: {
immich_server            |   "clipEncode": true,
immich_server            |   "facialRecognition": true,
immich_server            |   "tagImage": true,
immich_server            |   "map": true,
immich_server            |   "reverseGeocoding": true,
immich_server            |   "sidecar": true,
immich_server            |   "search": true,
immich_server            |   "trash": true,
immich_server            |   "oauth": false,
immich_server            |   "oauthAutoLaunch": false,
immich_server            |   "passwordLogin": true,
immich_server            |   "configFile": false
immich_server            | }
immich_server            | [Nest] 7  - 11/22/2023, 6:45:30 PM     LOG [NestApplication] Nest application successfully started +13ms
immich_server            | [Nest] 7  - 11/22/2023, 6:45:30 PM     LOG [ImmichServer] Immich Server is listening on http://[::1]:3001 [v1.88.2] [PRODUCTION]
immich_server            | [Nest] 7  - 11/22/2023, 6:45:34 PM     LOG [CommunicationRepository] Websocket Connect:    EqfRp3bwfldEe9qvAAAB

image

AndyXheli commented 12 months ago

Heres what i see after starting up the docker 

root@immich:/var/immich-app# docker logs immich_server --follow
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [NestFactory] Starting Nest application...
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +46ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +6ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [InstanceLoader] ScheduleModule dependencies initialized +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +227ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [InstanceLoader] InfraModule dependencies initialized +3ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [InstanceLoader] AppModule dependencies initialized +7ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [InstanceLoader] DomainModule dependencies initialized +17ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] ActivityController {/api/activity}: +131ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/activity, GET} route +3ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/activity/statistics, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/activity, POST} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/activity/:id, DELETE} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] AssetsController {/api/assets}: +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/assets, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] AssetController {/api/asset}: +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/map-marker, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/memory-lane, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/random, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/download/info, POST} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/download/archive, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/download/:id, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/statistics, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/time-buckets, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/time-bucket, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/jobs, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset, PUT} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset, DELETE} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/restore, POST} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/trash/empty, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/trash/restore, POST} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/stack/parent, PUT} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/:id, PUT} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] AssetController {/api/asset}: +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/upload, POST} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/file/:id, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/thumbnail/:id, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/curated-objects, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/curated-locations, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/search-terms, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/:deviceId, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/assetById/:id, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/exist, POST} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/asset/bulk-upload-check, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] AppController {/api}: +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/.well-known/immich, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/custom.css, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/refresh-config, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] AlbumController {/api/album}: +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/album/count, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/album, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/album, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/album/:id, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/album/:id, PATCH} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/album/:id, DELETE} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/album/:id/assets, PUT} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/album/:id/assets, DELETE} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/album/:id/users, PUT} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/album/:id/user/:userId, DELETE} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] APIKeyController {/api/api-key}: +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/api-key, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/api-key, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/api-key/:id, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/api-key/:id, PUT} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/api-key/:id, DELETE} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] AuditController {/api/audit}: +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/audit/deletes, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/audit/file-report, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/audit/file-report/checksum, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/audit/file-report/fix, POST} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] AuthController {/api/auth}: +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/auth/login, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/auth/admin-sign-up, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/auth/devices, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/auth/devices, DELETE} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/auth/devices/:id, DELETE} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/auth/validateToken, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/auth/change-password, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/auth/logout, POST} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] JobController {/api/jobs}: +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/jobs, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/jobs/:id, PUT} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] LibraryController {/api/library}: +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/library, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/library, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/library/:id, PUT} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/library/:id, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/library/:id, DELETE} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/library/:id/statistics, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/library/:id/scan, POST} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/library/:id/removeOffline, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] OAuthController {/api/oauth}: +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/oauth/mobile-redirect, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/oauth/config, POST} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/oauth/authorize, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/oauth/callback, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/oauth/link, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/oauth/unlink, POST} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] PartnerController {/api/partner}: +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/partner, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/partner/:id, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/partner/:id, PUT} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/partner/:id, DELETE} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] SearchController {/api/search}: +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/search/person, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/search, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/search/explore, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] ServerInfoController {/api/server-info}: +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/server-info, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/server-info/ping, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/server-info/version, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/server-info/features, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/server-info/theme, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/server-info/config, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/server-info/statistics, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/server-info/media-types, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] SharedLinkController {/api/shared-link}: +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/shared-link, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/shared-link/me, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/shared-link, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id, PATCH} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id, DELETE} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id/assets, PUT} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/shared-link/:id/assets, DELETE} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] SystemConfigController {/api/system-config}: +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/system-config, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/system-config/defaults, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/system-config, PUT} route +1ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/system-config/storage-template-options, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/system-config/map/style.json, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RoutesResolver] TagController {/api/tag}: +0ms
[Nest] 7  - 11/22/2023, 6:52:19 PM     LOG [RouterExplorer] Mapped {/api/tag, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/tag, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/tag/:id, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/tag/:id, PATCH} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/tag/:id, DELETE} route +1ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/tag/:id/assets, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/tag/:id/assets, PUT} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/tag/:id/assets, DELETE} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RoutesResolver] UserController {/api/user}: +1ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/user, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/user/info/:id, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/user/me, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/user, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/user/profile-image, DELETE} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/user/:id, DELETE} route +1ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/user/:id/restore, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/user, PUT} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/user/profile-image, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/user/profile-image/:id, GET} route +1ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RoutesResolver] PersonController {/api/person}: +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/person, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/person, PUT} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/person/:id, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/person/:id, PUT} route +1ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/person/:id/statistics, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/person/:id/thumbnail, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/person/:id/assets, GET} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [RouterExplorer] Mapped {/api/person/:id/merge, POST} route +0ms
[Nest] 7  - 11/22/2023, 6:52:20 PM     LOG [SearchService] Running bootstrap
Request #1700679140035: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700679140035: Sleeping for 4s and then retrying request...
Request #1700679140035: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700679140035: Sleeping for 4s and then retrying request...
Request #1700679140035: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700679140035: Sleeping for 4s and then retrying request...
Request #1700679140035: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700679140035: Sleeping for 4s and then retrying request...
Request #1700679140035: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700679140035: Sleeping for 4s and then retrying request...
Request #1700679140035: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700679140035: Sleeping for 4s and then retrying request...
Request #1700679140035: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700679140035: Sleeping for 4s and then retrying request...
Request #1700679140035: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700679140035: Sleeping for 4s and then retrying request...
Request #1700679140035: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700679140035: Sleeping for 4s and then retrying request...
Request #1700679140035: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700679140035: Sleeping for 4s and then retrying request...
Request #1700679140035: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700679140035: Sleeping for 4s and then retrying request...
Request #1700679140035: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700679140035: Sleeping for 4s and then retrying request...
Request #1700679140035: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700679140035: Sleeping for 4s and then retrying request...
Request #1700679140035: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700679140035: Sleeping for 4s and then retrying request...
Request #1700679140035: Request to Node 0 failed due to "undefined Request failed with HTTP code 503 | Server said: Not Ready or Lagging"
Request #1700679140035: Sleeping for 4s and then retrying request...
[Nest] 7  - 11/22/2023, 6:53:20 PM     LOG [TypesenseRepository] Schema up to date: assets/assets-v10
[Nest] 7  - 11/22/2023, 6:53:20 PM     LOG [TypesenseRepository] Schema up to date: albums/albums-v2
[Nest] 7  - 11/22/2023, 6:53:20 PM     LOG [TypesenseRepository] Schema up to date: faces/faces-v1
[Nest] 7  - 11/22/2023, 6:53:20 PM     LOG [TypesenseRepository] Alias mapping: [{"collection_name":"faces-v1","name":"faces"},{"collection_name":"albums-v2","name":"albums"},{"collection_name":"assets-v10","name":"assets"}]
[Nest] 7  - 11/22/2023, 6:53:20 PM     LOG [TypesenseRepository] Collections needing migration: {"assets":false,"albums":false,"faces":false}
[Nest] 7  - 11/22/2023, 6:53:20 PM     LOG [AppService] Feature Flags: {
  "clipEncode": true,
  "facialRecognition": true,
  "tagImage": true,
  "map": true,
  "reverseGeocoding": true,
  "sidecar": true,
  "search": true,
  "trash": true,
  "oauth": false,
  "oauthAutoLaunch": false,
  "passwordLogin": true,
  "configFile": false
}
[Nest] 7  - 11/22/2023, 6:53:20 PM     LOG [NestApplication] Nest application successfully started +13ms
alextran1502 commented 12 months ago

@AndyXheli Typesense on start-up would index the library to serve the quick search purpose, which might take a few minutes depending on the size of your gallery

AndyXheli commented 12 months ago

Ah okay! yes i have about 300GB worth of data