API component binds to localhost

rkojedzinszky commented 1 month ago

The bug

After upgrading from v1.117.0 to v1.118.*, the api component binds to localhost, not to wildcard address. Log shows:

The OS that Immich Server is running on

Debian

Version of Immich Server

v1.118.1

Version of Immich Mobile App

v1.117.0

Platform with the issue

[X] Server
[ ] Web
[ ] Mobile

Your docker-compose.yml content

n/a

Your .env content

n/a

Reproduction steps

Upgrade from v1.117.0 to v1.118.*
Observe logs

Relevant log output

[Nest] 17  - 10/16/2024, 11:54:41 AM     LOG [Api:Bootstrap] Immich Server is listening on http://[::1]:2283 [v1.118.1] [production]

Additional information

Documentation shows that IMMICH_HOST should control this, howewer, the default value is 0.0.0.0. Also, setting this explicitly still dont help. Reading the code, it turns out that app uses the HOST environment variable, not IMMICH_HOST. Howewer, setting this explicitly to 0.0.0.0 also does not help. The application does process the environment variable, as the log message changes to

[Nest] 17  - 10/16/2024, 11:59:09 AM     LOG [Api:Bootstrap] Immich Server is listening on http://127.0.0.1:2283 [v1.118.1] [production]

Howewer, still not listening on 0.0.0.0. In kubernetes, the following workaround is working, howewer, I think that the app should bind to 0.0.0.0 as before.

        env:
        - name: HOST
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP

bo0tzz commented 1 month ago

How are you running Immich?

rkojedzinszky commented 1 month ago

@bo0tzz it is run inside Kubernetes, with the following Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: immich-server-api
spec:
  selector:
    matchLabels:
      app.kubernetes.io/instance: immich
      app.kubernetes.io/name: server-api
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: immich
        app.kubernetes.io/name: server-api
    spec:
      containers:
      - env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: immich-postgresql
        - name: IMMICH_WORKERS_INCLUDE
          value: api
        - name: NODE_OPTIONS
          value: --max-old-space-size=384
        - name: HOST
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        envFrom:
        - configMapRef:
            name: immich-server-environment
        image: ghcr.io/immich-app/immich-server:v1.118.1
        livenessProbe:
          failureThreshold: 30
          httpGet:
            path: /api/server/ping
            port: http
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: immich-server
        ports:
        - containerPort: 2283
          name: http
          protocol: TCP
        - containerPort: 8081
          name: metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /api/server/ping
            port: http
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 100m
            memory: 384Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - NET_RAW
        volumeMounts:
        - mountPath: /usr/src/app/upload
          name: library
      enableServiceLinks: false
      securityContext:
        runAsGroup: 8080
        runAsNonRoot: true
        runAsUser: 18760
      volumes:
      - name: library
        persistentVolumeClaim:
          claimName: immich

immich-server-environment ConfigMap contents:

apiVersion: v1
data:
  DB_DATABASE_NAME: immich
  DB_HOSTNAME: immich-postgresql
  DB_USERNAME: immich
  IMMICH_MACHINE_LEARNING_URL: http://immich-machine-learning:3003
  IMMICH_METRICS: "true"
  REDIS_HOSTNAME: immich-redis
kind: ConfigMap
metadata:
  name: immich-server-environment

bo0tzz commented 1 month ago

Is anything actually not working? My pod binds to http://[::1]:2283 and works just fine. It's pretty common for containers to listen on localhost & the external port bind then picks that up further.

rkojedzinszky commented 1 month ago

@bo0tzz Sorry for the complaint, it was my fault, everything works fine.

So I went through these steps when finally I opened this issue:

upgraded the deployment image from v1.117 to v1.118 (did not read the breaking changes section)
while waiting for the pod to be ready, I observed the port change, so I've updated readiness/liveness probe's ports also
Then I've observed the (misleading) log message containing address http://[::1]:2283. That is definitely a loopback address, and a loopback bound socket only can receive connections from localhost
still, the pod did not become ready, so my guess was that the http://[::1]:2283 is the reason. I would have been happy if it printed http://[::]:2283, that would mean a wildcard address, howewer, I could only achieve that by the workaround I've posted.
then, when with the workaround it still did not work, I've noticed the breaking changes, that api urls have moved which I used for readiness and liveness probes. With those fixed, the pod became ready.

So, after the last step, I did not revert the HOST environment setting, as it seemed to be working, and also, I was expecting an address in the log message like [::] or 0.0.0.0. After your last comment, I've removed the HOST var, and surprise(!), it still works, howewer, it displays [::1] as bound address. Then, I've checked the socket, and surely it is bound to [::], and not to [::1]. So, actually, the log message is really confusing, and I trusted that message, that's why I thought that that is the problem. Just to prove, now I'am running immich without HOST, as you are, receiving the same log message as you, but the application indeed listens on wildcard address:

# ps axw|grep "immich$"
 863873 ?        Sl     0:20 immich
# nsenter -n -t 863873 netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp6       0      0 :::2283                 :::*                    LISTEN      863918/immich-api   
tcp6       0      0 :::8081                 :::*                    LISTEN      863918/immich-api   
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path

Probably this was the same with earlier versions too.

So, again, I am sorry for opening this. Howewer, I think the log message should be fixed to not be confusing next time.

immich-app / immich