docker-zulip behind "docker Caddy" as reverse proxy has problems with CSS and scripts

[peracchi]

On some browsers (Brave/Edge/Chrome) I got:

Refused to execute script from '' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.

On Firefox, I got pages without CSS and missing scripts functionalities. But after do a CTRL-F5, CSS/scripts load.

Bellow my docker-compose.yml for Zulip:

version: "2"
services:

  zulip-db:
    image: "zulip/zulip-postgresql:14"
    container_name: zulip-db
    hostname: zulip-db
    restart: unless-stopped
    environment:
      POSTGRES_DB: "zulip"
      POSTGRES_USER: "zulip"
      POSTGRES_PASSWORD: "postgres-password"
    volumes:
      - "db:/var/lib/postgresql/data:rw"

  zulip-memcached:
    image: "memcached:alpine"
    container_name: zulip-memcached
    hostname: zulip-memcached
    restart: unless-stopped
    command:
      - "sh"
      - "-euc"
      - |
        echo 'mech_list: plain' > "$$SASL_CONF_PATH"
        echo "zulip@$$HOSTNAME:$$MEMCACHED_PASSWORD" > "$$MEMCACHED_SASL_PWDB"
        echo "zulip@localhost:$$MEMCACHED_PASSWORD" >> "$$MEMCACHED_SASL_PWDB"
        exec memcached -S
    environment:
      SASL_CONF_PATH: "/home/memcache/memcached.conf"
      MEMCACHED_SASL_PWDB: "/home/memcache/memcached-sasl-db"
      MEMCACHED_PASSWORD: "memcached-password"

  zulip-rabbitmq:
    image: "rabbitmq:3.13.1"
    container_name: zulip-rabbitmq
    hostname: zulip-rabbitmq
    restart: unless-stopped
    environment:
      RABBITMQ_DEFAULT_USER: "zulip"
      RABBITMQ_DEFAULT_PASS: "rabbitmq-password"
    volumes:
      - "rabbitmq:/var/lib/rabbitmq:rw"

  zulip-redis:
    image: "redis:alpine"
    container_name: zulip-redis
    hostname: zulip-redis
    restart: unless-stopped
    command:
      - "sh"
      - "-euc"
      - |
        echo "requirepass '$$REDIS_PASSWORD'" > /etc/redis.conf
        exec redis-server /etc/redis.conf
    environment:
      REDIS_PASSWORD: "redis-password"
    volumes:
      - "redis:/data:rw"

  zulip:
    image: "zulip/docker-zulip:8.3-1"
    container_name: zulip
    hostname: zulip
    restart: unless-stopped
    build:
      context: .
      args:
        ZULIP_GIT_URL: https://github.com/zulip/zulip.git
        ZULIP_GIT_REF: "8.3"
    #ports:
    #  - "80:80"
    #  - "443:443"
    environment:
      DB_HOST: "zulip-db"
      DB_HOST_PORT: "5432"
      DB_USER: "zulip"
      DISABLE_HTTPS: "True" # Caddy HTTPS Server will handle TLS and act as reverse proxy
      LOADBALANCER_IPS: "172.18.0.5" # Caddy HTTPS server container's IP address
      SSL_CERTIFICATE_GENERATION: "self-signed"
      SETTING_MEMCACHED_LOCATION: "zulip-memcached:11211"
      SETTING_RABBITMQ_HOST: "zulip-rabbitmq"
      SETTING_REDIS_HOST: "zulip-redis"
      SECRETS_rabbitmq_password: "rabbitmq-password"
      SECRETS_postgres_password: "postgres-password"
      SECRETS_memcached_password: "memcached-password"
      SECRETS_redis_password: "redis-password"
      SECRETS_secret_key: "secret_key"
      SETTING_EXTERNAL_HOST: "zulip.my.domain.com"
      SETTING_ZULIP_ADMINISTRATOR: "zulip.admin@my.domain.com"
      SETTING_EMAIL_HOST: "mail.my.domain.com"
      SETTING_EMAIL_HOST_USER: "zulip.admin@my.domain.com"
      SECRETS_EMAIL_PASSWORD: "zulip.admin-password"
      SETTING_EMAIL_PORT: "587"
      SETTING_NOREPLY_EMAIL_ADDRESS: "zulip.no-reply@my.domain.com"
      SETTING_ADD_TOKENS_TO_NOREPLY_ADDRESS: "False"
      SETTING_EMAIL_USE_SSL: "False"
      SETTING_EMAIL_USE_TLS: "True"
      ZULIP_AUTH_BACKENDS: "EmailAuthBackend"
    volumes:
      - "data:/data:rw"
    ulimits:
      nofile:
        soft: 1000000
        hard: 1048576

volumes:
  data:
  db:
  rabbitmq:
  redis:

networks:
  default:
    name: caddy_net
    external: true

On Caddy side, my Caddyfile needs only:

zulip.my.domain.com {
        reverse_proxy http://zulip:80
}

Some reference links:

• Stylesheet not loaded because of MIME type
• docker-zulip behind "docker Caddy" as reverse proxy
• Caddy HTTPS Server

[peracchi]

Appears to be a misconfiguration and/or bug in Caddy.

See reverse_proxy leads to duplicate Server headers

[francislavoie]

The issue you linked to seems to have no relevance to the problem you're having.

[mholt]

@peracchi Can you explain how the Server header is affecting CSS/scripts?

[peracchi]

Dear @francislavoie and @mholt , if you can, please take a look at

https://chat.zulip.org/#narrow/stream/31-production-help/topic/docker-zulip.20behind.20.22docker.20Caddy.22.20as.20reverse.20proxy

There is a lot of information there.

I'm still trying to understand what is happening. I even got a behavior on Firefox and another in Edge/Chrome.

I don't think it is a bug. Probably a misconfiguration I have done unintentionally.

[mholt]

Can the behavior of the server be reproduced with curl? If not, it's likely a browser bug.

[alexmv]

The bug was a misplaced header Content-Type text/html in the Caddy configuration, which resulted in duplicate Content-Type headers in the response.

[mholt]

It's because the header directive applies immediately, then the reverse proxy copies down the header from the backend, resulting in two.

Aside from what sounds like the proper fix already discovered, if the header directive were deferred:

header Content-Type text/html {
    defer
}

there would be no duplicate. Alternatively, utilizing the proxy's header manipulation config to apply changes to headers coming down from the backend without ambiguity:

reverse_proxy host:port {
    header_down Content-Type text/html
}

would also not result in a duplicate header.