search

immich-app / immich

High performance self-hosted photo and video management solution.
https://immich.app
GNU Affero General Public License v3.0
45.19k stars 2.19k forks source link

[BUG] Error [ERR_HTTP_HEADERS_SENT]: Cannot remove headers after they are sent to the client #5744

Closed dsm1212 closed 9 months ago

dsm1212 commented 9 months ago

The bug

After updating to 1.91.0 I'm seeing these messages in the server log:

[Nest] 8 - 12/15/2023, 9:38:56 PM ERROR [ExceptionsHandler] Cannot remove headers after they are sent to the client Error [ERR_HTTP_HEADERS_SENT]: Cannot remove headers after they are sent to the client at ServerResponse.removeHeader (node:_http_outgoing:782:11) at ServerResponse.send (/usr/src/app/node_modules/express/lib/response.js:214:10) at ExpressAdapter.reply (/usr/src/app/node_modules/@nestjs/platform-express/adapters/express-adapter.js:31:29) at RouterResponseController.apply (/usr/src/app/node_modules/@nestjs/core/router/router-response-controller.js:15:36) at /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:175:48 at async /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:47:13 at async /usr/src/app/node_modules/@nestjs/core/router/router-proxy.js:9:17

The system is working fine and my read of the error is that it is benign but the code apparently is doing someting to the response after it has been sent. This is usually bug. unfortuntely I'm not getting enough of the call stack to know where to look in immich.

The OS that Immich Server is running on

Debian (docker)

Version of Immich Server

v1.91.0

Version of Immich Mobile App

N/A

Platform with the issue

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    hostname: immich-server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    network_mode: svcnet
    command: ["start.sh", "immich"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro 
      - /media/pictures/<redacted>:/media/pictures/<redacted>:ro
      - /apps/immich:/apps/immich
    user: "1030:1001"
#    ports:
#      - 2283:3001
    env_file:
      - immich.env
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    hostname: immich-microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    network_mode: svcnet
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: ["start.sh", "microservices"]
    #user: "1030:1001"
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /media/pictures/<redacted>:/media/pictures/<redacted>:ro
      - /apps/immich:/apps/immich
    env_file:
      - immich.env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    hostname: immich-machine-learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    network_mode: svcnet
    volumes:
      - model-cache:/cache
    env_file:
      - immich.env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    network_mode: svcnet
    volumes:
      - redisdata:/data 
    restart: always

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11
    network_mode: svcnet
    env_file:
      - immich.env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:
  redisdata:

Your .env content

# The location where your uploaded files are stored
UPLOAD_LOCATION=/unsafe/immich

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

IMMICH_CONFIG_FILE=/apps/immich/immich.config

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=<redacted>
DB_PASSWORD=<redacted>

TZ=America/New_York

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

As I browse in image the server logs these messages.

Additional information

I use swag. here is my proxy config. I already had the http settings you suggested in a shared proxy.cfg file.

server { listen 443 ssl; listen [::]:443 ssl;

server_name immich.*;

include /config/nginx/ssl.conf;

client_max_body_size 50000M;

location / {
    include /config/nginx/proxy.conf;
    include /config/nginx/resolver.conf;
    set $upstream_app immich_server;
    set $upstream_port 3001;
    set $upstream_proto http;
    proxy_pass $upstream_proto://$upstream_app:$upstream_port;
}

}

hobbsAU commented 9 months ago

I have the same issue v1.91.0, using traefik as a proxy to immich-server.

immich-immich-server-1            | [Nest] 7  - 12/16/2023, 7:16:13 AM     LOG [CommunicationRepository] Websocket Connect:    xxxxxxxxxxxxxxxxxxx
immich-immich-server-1            | [Nest] 7  - 12/16/2023, 7:16:13 AM   ERROR [ExceptionsHandler] Cannot remove headers after they are sent to the client
immich-immich-server-1            | Error [ERR_HTTP_HEADERS_SENT]: Cannot remove headers after they are sent to the client
immich-immich-server-1            |     at ServerResponse.removeHeader (node:_http_outgoing:782:11)
immich-immich-server-1            |     at ServerResponse.send (/usr/src/app/node_modules/express/lib/response.js:214:10)
immich-immich-server-1            |     at ExpressAdapter.reply (/usr/src/app/node_modules/@nestjs/platform-express/adapters/express-adapter.js:31:29)
immich-immich-server-1            |     at RouterResponseController.apply (/usr/src/app/node_modules/@nestjs/core/router/router-response-controller.js:15:36)
immich-immich-server-1            |     at /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:175:48
immich-immich-server-1            |     at async /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:47:13
immich-immich-server-1            |     at async /usr/src/app/node_modules/@nestjs/core/router/router-proxy.js:9:17
lslvxy commented 9 months ago

same +1 app auto backup also failed

Arbide commented 9 months ago

+1 here. Same error.

TomasValenta commented 9 months ago

Same here, it happens on page refresh and just shows a blank page.

traktuner commented 9 months ago

Same here. Refresh - blank page since 1.91.0.

[Nest] 653  - 12/16/2023, 3:22:03 PM    WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 653  - 12/16/2023, 3:22:30 PM   ERROR [ExceptionsHandler] Cannot remove headers after they are sent to the client

Error [ERR_HTTP_HEADERS_SENT]: Cannot remove headers after they are sent to the client

    at ServerResponse.removeHeader (node:_http_outgoing:782:11)

    at ServerResponse.send (/app/immich/server/node_modules/express/lib/response.js:214:10)

    at ExpressAdapter.reply (/app/immich/server/node_modules/@nestjs/platform-express/adapters/express-adapter.js:31:29)

    at RouterResponseController.apply (/app/immich/server/node_modules/@nestjs/core/router/router-response-controller.js:15:36)

    at /app/immich/server/node_modules/@nestjs/core/router/router-execution-context.js:175:48

    at async /app/immich/server/node_modules/@nestjs/core/router/router-execution-context.js:47:13

    at async /app/immich/server/node_modules/@nestjs/core/router/router-proxy.js:9:17
alextran1502 commented 9 months ago

@jrasm91 Any thoughts on this error Jason? I can't reproduce the blank page issue

waclaw66 commented 9 months ago

Who is not using Docker? I don't. Maybe it's related.

alextran1502 commented 9 months ago

@traktuner @TomasValenta Are you guys using docker/docker-compose to deploy Immich?

cookandy commented 9 months ago

Hi. I'm also having this issue, and am using https://github.com/imagegenius/docker-immich to deploy, along with the tensorchord/pgvecto-rs:pg15-v0.1.11 psql image.

nicokaiser commented 9 months ago

Same here. But no blank page, frontend seems to work correctly. Docker containers via Traefik.

traktuner commented 9 months ago

@traktuner @TomasValenta Are you guys using docker/docker-compose to deploy Immich?

Yes, docker-compose image @martabal AIO image

martabal commented 9 months ago

The blank page issue should been have solved with #5753.

Will be fixed for the AIO when https://github.com/imagegenius/docker-immich/pull/255 is merged

dsm1212 commented 9 months ago

I just took an immich update and this message is now gone. I can't get it to reappear. Normally I would resolve since I submitted it, but since there are so many others here I'll leave to whomever fixed it to resolve.

waclaw66 commented 9 months ago

That error message "Cannot remove headers after they are sent to the client" persists in 1.91.2. Blank page was fixed by https://github.com/immich-app/immich/pull/5753.

waclaw66 commented 9 months ago

Additionally seeing a lot of... [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses

dabenzel commented 9 months ago

I have this error too in 1.91.2

Logs:


immich_server            | Error [ERR_HTTP_HEADERS_SENT]: Cannot remove headers after they are sent to the client
immich_server            |     at ServerResponse.removeHeader (node:_http_outgoing:782:11)
immich_server            |     at ServerResponse.send (/usr/src/app/node_modules/express/lib/response.js:214:10)
immich_server            |     at ExpressAdapter.reply (/usr/src/app/node_modules/@nestjs/platform-express/adapters/express-adapter.js:31:29)
immich_server            |     at RouterResponseController.apply (/usr/src/app/node_modules/@nestjs/core/router/router-response-controller.js:15:36)
immich_server            |     at /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:175:48
immich_server            |     at async /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:47:13
immich_server            |     at async /usr/src/app/node_modules/@nestjs/core/router/router-proxy.js:9:17
i`
traktuner commented 9 months ago

1.91.2


[Nest] 591  - 12/17/2023, 1:06:05 PM   ERROR [ExceptionsHandler] Cannot remove headers after they are sent to the client
Error [ERR_HTTP_HEADERS_SENT]: Cannot remove headers after they are sent to the client
    at ServerResponse.removeHeader (node:_http_outgoing:782:11)
    at ServerResponse.send (/app/immich/server/node_modules/express/lib/response.js:214:10)
    at ExpressAdapter.reply (/app/immich/server/node_modules/@nestjs/platform-express/adapters/express-adapter.js:31:29)
    at RouterResponseController.apply (/app/immich/server/node_modules/@nestjs/core/router/router-response-controller.js:15:36)
    at /app/immich/server/node_modules/@nestjs/core/router/router-execution-context.js:175:48
    at async /app/immich/server/node_modules/@nestjs/core/router/router-execution-context.js:47:13
    at async /app/immich/server/node_modules/@nestjs/core/router/router-proxy.js:9:17
martabal commented 9 months ago

I only have this issue when using my reverse proxy

mvivaldi commented 9 months ago

same problem in 1.91.3, and now (I don't know if this is related) my android phone backup the photos but there is none in web interface (with 1.91.2 everything was working fine)

nvm: my phone changed the location (different timezone) and the photos was taken in the future... after one hour (difference of timezone) the photos showed in the web interface

cohen87 commented 9 months ago

Same error im experiencing when trying to perform a search in 1.91.2

ma0dan commented 9 months ago

@mvivaldi @lslvxy @martabal backup issue may relate with reverse-proxy's config like in nginx client_max_body_size

hafx commented 9 months ago

Same issue when I reload or scroll a page (not a specific page) but no blank page, frontend seems to work correctly. Docker (compose) containers via Nginx. I'm using this : 

client_max_body_size 50000M;

Version : 1.91.3

[Nest] 7  - 12/19/2023, 2:53:19 AM   ERROR [ExceptionsHandler] Cannot remove headers after they are sent to the client
Error [ERR_HTTP_HEADERS_SENT]: Cannot remove headers after they are sent to the client
    at ServerResponse.removeHeader (node:_http_outgoing:782:11)
    at ServerResponse.send (/usr/src/app/node_modules/express/lib/response.js:214:10)
    at ExpressAdapter.reply (/usr/src/app/node_modules/@nestjs/platform-express/adapters/express-adapter.js:31:29)
    at RouterResponseController.apply (/usr/src/app/node_modules/@nestjs/core/router/router-response-controller.js:15:36)
    at /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:175:48
    at async /usr/src/app/node_modules/@nestjs/core/router/router-execution-context.js:47:13
    at async /usr/src/app/node_modules/@nestjs/core/router/router-proxy.js:9:17

What am I supposed to do ? Thanks

dabenzel commented 9 months ago

What am I supposed to do ?

Update.

ricardomga commented 3 months ago

I have a similar log when i try to load some videos, when this happens the video seems to load very slow

[Nest] 17  - 06/14/2024, 4:36:17 PM   ERROR [Api:LoggerRepository~sb9vgm39] Unable to send file: Error
Error: write EPIPE
    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:95:16)
    at WriteWrap.callbackTrampoline (node:internal/async_hooks:130:17)
[Nest] 17  - 06/14/2024, 4:36:17 PM   ERROR [Api:Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
    at ServerResponse.setHeader (node:_http_outgoing:659:11)
    at ServerResponse.header (/usr/src/app/node_modules/express/lib/response.js:795:10)
    at sendFile (/usr/src/app/dist/utils/file.js:61:13)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async AssetMediaController.playAssetVideo (/usr/src/app/dist/controllers/asset-media.controller.js:60:9)~sb9vgm39] Failed to play asset video
[Nest] 17  - 06/14/2024, 4:36:17 PM   ERROR [Api:Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
    at ServerResponse.setHeader (node:_http_outgoing:659:11)
    at ServerResponse.header (/usr/src/app/node_modules/express/lib/response.js:795:10)
    at sendFile (/usr/src/app/dist/utils/file.js:61:13)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async AssetMediaController.playAssetVideo (/usr/src/app/dist/controllers/asset-media.controller.js:60:9)~sb9vgm39] Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
jrasm91 commented 3 months ago

Fixed again in #10289, which will be in the next release.