bitwarden_rs behind traefik 2 issue

I'm not sure if this is a bug or I have something misconfigured but I used the http-to-https traefik example from the wiki and cannot get logged into bitwarden. The login screen appears and I can key in the username and password but it kicks back "An error has occured, wrong username or password"/

If I open the developer tools I can see the POST is 404'ing on the prelogin (https://bitwarden.mydomain.com/api/accounts/prelogin) and 400 on the token (https://bitwarden.mydomain.com/identity/connect/token).

If I drop the letencrypt cert and default to port 80 everything appears to work. I don't see anything obvious in my traefik or bitwarden logs. Any suggestions would be great. Thanks!

I have bitwarden_rs working with Traefik 2. Here are the labels from my docker-compose that are working for me. I'm no expert but I may be able to help if you post your configuration.

relevant part of my docker-compose.yml

labels:
      - "traefik.enable=true"

      - "traefik.http.routers.bitwarden.entrypoints=http"
      - "traefik.http.routers.bitwarden.rule=Host(`<hostname>`)"
      - "traefik.http.middlewares.bitwarden-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.bitwarden.middlewares=bitwarden-https-redirect"
      - "traefik.http.routers.bitwarden.service=bitwarden"
      - "traefik.http.services.bitwarden.loadbalancer.server.port=80"

      - "traefik.http.routers.bitwarden-secure.entrypoints=https"
      - "traefik.http.routers.bitwarden-secure.rule=Host(`<hostname>`)"
      - "traefik.http.routers.bitwarden-secure.tls=true"
      - "traefik.http.routers.bitwarden-secure.tls.certresolver=dns"
      - "traefik.http.routers.bitwarden-secure.service=bitwarden-secure"
      - "traefik.http.services.bitwarden-secure.loadbalancer.server.port=80"

      - "traefik.http.routers.bitwarden-ws.entrypoints=https"
      - "traefik.http.routers.bitwarden-ws.rule=Host(`<hostname>`) && Path(`/notifications/hub`)"
      - "traefik.http.middlewares.bitwarden-ws-strip.stripprefix.prefixes=/notifications/hub"
      - "traefik.http.routers.bitwarden-ws.middlewares=bitwarden-ws-strip"
      - "traefik.http.routers.bitwarden-ws.tls=true"
      - "traefik.http.routers.bitwarden-ws.tls.certresolver=dns"
      - "traefik.http.routers.bitwarden-ws.service=bitwarden-ws"
      - "traefik.http.services.bitwarden-ws.loadbalancer.server.port=3012"

      - "traefik.docker.network=proxy"

traefik-config.yml

http:
  middlewares:
    https-redirect:
      redirectScheme:
        scheme: https

    default-headers:
      headers:
        frameDeny: true
        sslRedirect: true
        browserXssFilter: true
        contentTypeNosniff: true
        forceSTSHeader: true
        stsIncludeSubdomains: true
        stsPreload: true
        stsSeconds: 31536000

I solved the issue but it was unrelated to my docker-compose file for bitwarden. The error turned out to be because I had the traefik dashboard enabled with the following rule which hijacks the bitwarden_rs api call...

"traefik.http.routers.api.rule=PathPrefix(`/dashboard/`) || PathPrefix(`/api`)"

before I close this issue (if rules allow) may I ask where you found the following line? I added it to my config l but it appears to operate without it as well...

"traefik.http.middlewares.bitwarden-ws-strip.stripprefix.prefixes=/notifications/hub"

I added it when I was testing my traefik 2 configuration. I found that without it, I would get a 404 error when I when to my /notifications/hub . With it I get an error about parsing my websocket key but that was validation to me that it was actually trying to communicate to that port via websockets.

I think the need to expose port 3012 for /notifications/hub are a workaround for limitations of the version of Rocket bitwarden_rs is using and websockets. The port 3012 listener I think is on the root and after the redirect, doesn't need the path information.

I'm not sure it's correct and would welcome feedback from the community.

Glad to hear your instance is working.

Thanks for elaborating! I'm not sure I would have discovered that on my own. I've got all my passwords imported and working now. Thanks again!

I solved the issue but it was unrelated to my docker-compose file for bitwarden. The error turned out to be because I had the traefik dashboard enabled with the following rule which hijacks the bitwarden_rs api call...
"traefik.http.routers.api.rule=PathPrefix(`/dashboard/`) || PathPrefix(`/api`)"
before I close this issue (if rules allow) may I ask where you found the following line? I added it to my config l but it appears to operate without it as well...
"traefik.http.middlewares.bitwarden-ws-strip.stripprefix.prefixes=/notifications/hub"

Hey there on Traefik 2.1 I see I'm facing the same issue of traefik's api conflicting with reverse proxying into bitwarden. I rolled back to 2.0.

How did you solve this ?

On 2.0 with everything working I see my loadbalancer is going to http://172.18.0.7:80. This doesn't seem right ? Shouldn't it be going to :3012 ?

Sorry I wasn't through in my resolution. To properly fix the dashboard and API try:

"Host(`traefik.domain.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"

The default rules are just grabbing anything that ends in /api

In regards to your loadbalancer, I'm not sure I understand. You should have your loadbalancer port set to :3012 in the yml file so traefik will redirect it to http/https 80 and 443 respectively. Please let me know if you need further help, I'll be happy to assist.

I haven't upgrade to Traefik 2.1 yet as it's in release candidate status. When it becomes a "release" I'll update my config. They're very good with their release candidates but not perfect so you may be experiencing a bug.

Sorry I wasn't through in my resolution. To properly fix the dashboard and API try:
"Host(`traefik.domain.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
The default rules are just grabbing anything that ends in /api

In regards to your loadbalancer, I'm not sure I understand. You should have your loadbalancer port set to :3012 in the yml file so traefik will redirect it to http/https 80 and 443 respectively. Please let me know if you need further help, I'll be happy to assist.

So I ended up having a configuration error on my docker-compose.yml. I was using traefik.port=3012 as opposed to "traefik.http.services.bitwarden-ws.loadbalancer.server.port=3012"

What's odd though is my docker-compose looks nothing like @scosno and when I change the port I get "WebSocket Protocol Error: Unable to parse WebSocket key."

Here's my relevant parts of my compose file.

      - "traefik.enable=true"
      - "traefik.http.routers.bitwarden.rule=Host(`bitwarden.domain.com`)"
      - "traefik.http.routers.bitwarden.entrypoints=websecure"
      - "traefik.http.routers.bitwarden.tls.certresolver=mydnschallenge"
      - "traefik.http.services.bitwarden.loadbalancer.server.port=3012"
      - "traefik.docker.network=traefik_proxy"

I'll give my Traefik config as well.

  traefik:
    image: "traefik:v2.0"
    container_name: "traefik"
    restart: always
    command:
      #- "--log.level=DEBUG"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.network=traefik_proxy"
      - "--providers.file.directory=/etc/traefik"
      - "--providers.file.filename=config.toml"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.minecraft.address=:25565"
      - "--certificatesresolvers.mydnschallenge.acme.dnschallenge=true"
      - "--certificatesresolvers.mydnschallenge.acme.dnschallenge.provider=cloudflare"
      #- "--certificatesresolvers.mydnschallenge.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.mydnschallenge.acme.email=OMIT"
      - "--certificatesresolvers.mydnschallenge.acme.storage=/letsencrypt/acme.json"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    environment:
      - "CF_API_EMAIL=OMIT"
      - "CF_API_KEY=OMIT"
    networks:
      traefik_proxy:
    volumes:
      - "$Docker:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "$Docker:/etc/traefik"

@grymlan Should I be adding "Host(`traefik.domain.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" to my Traefik's compose file ? I currently don't have traffic exposed outside the LAN.

Since with my current configuration I can only use bitwarden on port 80 I would assume it would be best to get that fixed and utilize the WS port 3012 ?

I think I would have to use a configuration like @scosno ?

It's also odd that the LB is going to port 80 since Traefik is using 80. I don't have 80 exposed in my bitwarden compose file though I do with 3012 and websocket enabled. Both are using the external network traefik_proxy

It's also odd that the LB is going to port 80 since Traefik is using 80. I don't have 80 exposed in my bitwarden compose file though I do with 3012 and websocket enabled. Both are using the external network traefik_proxy

According to your configuration, you are exposing - "traefik.http.routers.bitwarden.entrypoints=websecure" which would be port 443 or https.

I'll paste my docker-compose.yml and then try to explain whats going on.

      - "traefik.enable=true"
      - "traefik.http.routers.bitwarden.entrypoints=web"
      - "traefik.http.routers.bitwarden.rule=Host(`bw.mydomain.com`)"
      - "traefik.http.middlewares.bitwarden-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.bitwarden.middlewares=bitwarden-https-redirect"
      - "traefik.http.routers.bitwarden.service=bitwarden"
      - "traefik.http.services.bitwarden.loadbalancer.server.port=80"

      - "traefik.http.routers.bitwarden-secure.entrypoints=websecure"
      - "traefik.http.routers.bitwarden-secure.rule=Host(`bw.mydomain.com`)"
      - "traefik.http.routers.bitwarden-secure.tls=true"
      - "traefik.http.routers.bitwarden-secure.tls.certresolver=mycert"
      - "traefik.http.routers.bitwarden-secure.service=bitwarden-secure"
      - "traefik.http.services.bitwarden-secure.loadbalancer.server.port=80"

      - "traefik.http.routers.bitwarden-ws.entrypoints=websecure"
      - "traefik.http.routers.bitwarden-ws.rule=Host(`bw.mydomain.com`) && Path(`/notifications/hub`)"
      - "traefik.http.middlewares.bitwarden-ws-strip.stripprefix.prefixes=/notifications/hub"
      - "traefik.http.routers.bitwarden-ws.middlewares=bitwarden-ws-strip"
      - "traefik.http.routers.bitwarden-ws.tls=true"
      - "traefik.http.routers.bitwarden-ws.tls.certresolver=mycert"
      - "traefik.http.routers.bitwarden-ws.service=bitwarden-ws"
      - "traefik.http.services.bitwarden-ws.loadbalancer.server.port=3012"

      - "traefik.docker.network=traefik_proxy"
      - "traefik.http.routers.bitwarden-ui-https.tls.certresolver=mycert"

This config will redirect http to https and keep everything secure. It also includes the fix for websockets on port 3012. The important thing to remember is the loadbalancer ports are interfacing with your docker application. Traefik entrypoints are rerouting and actually exposing those services on web/websecure as denoted in your traefik config.

      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"

@grymlan Should I be adding "Host(traefik.domain.com) && (PathPrefix(/api) || PathPrefix(/dashboard))" to my Traefik's compose file ? I currently don't have traffic exposed outside the LAN.

Not unless you plan to expose or access your traefik dashboard. If you decide to do this latter I would use traefik middleware to add some basic authetication or you risk enumerating your entire infrastructure.

It's also odd that the LB is going to port 80 since Traefik is using 80. I don't have 80 exposed in my bitwarden compose file though I do with 3012 and websocket enabled. Both are using the external network traefik_proxy

According to your configuration, you are exposing - "traefik.http.routers.bitwarden.entrypoints=websecure" which would be port 443 or https.

What I'm saying is Traefik has port 80 being exposed on my host. I guess since I didn't expose it on Bitwarden that's how I'm not getting a port conflict.

I'll paste my docker-compose.yml and then try to explain whats going on.
      - "traefik.enable=true"
      - "traefik.http.routers.bitwarden.entrypoints=web"
      - "traefik.http.routers.bitwarden.rule=Host(`bw.mydomain.com`)"
      - "traefik.http.middlewares.bitwarden-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.bitwarden.middlewares=bitwarden-https-redirect"
      - "traefik.http.routers.bitwarden.service=bitwarden"
      - "traefik.http.services.bitwarden.loadbalancer.server.port=80"

      - "traefik.http.routers.bitwarden-secure.entrypoints=websecure"
      - "traefik.http.routers.bitwarden-secure.rule=Host(`bw.mydomain.com`)"
      - "traefik.http.routers.bitwarden-secure.tls=true"
      - "traefik.http.routers.bitwarden-secure.tls.certresolver=mycert"
      - "traefik.http.routers.bitwarden-secure.service=bitwarden-secure"
      - "traefik.http.services.bitwarden-secure.loadbalancer.server.port=80"

      - "traefik.http.routers.bitwarden-ws.entrypoints=websecure"
      - "traefik.http.routers.bitwarden-ws.rule=Host(`bw.mydomain.com`) && Path(`/notifications/hub`)"
      - "traefik.http.middlewares.bitwarden-ws-strip.stripprefix.prefixes=/notifications/hub"
      - "traefik.http.routers.bitwarden-ws.middlewares=bitwarden-ws-strip"
      - "traefik.http.routers.bitwarden-ws.tls=true"
      - "traefik.http.routers.bitwarden-ws.tls.certresolver=mycert"
      - "traefik.http.routers.bitwarden-ws.service=bitwarden-ws"
      - "traefik.http.services.bitwarden-ws.loadbalancer.server.port=3012"

      - "traefik.docker.network=traefik_proxy"
      - "traefik.http.routers.bitwarden-ui-https.tls.certresolver=mycert"
This config will redirect http to https and keep everything secure. It also includes the fix for websockets on port 3012. The important thing to remember is the loadbalancer ports are interfacing with your docker application. Traefik entrypoints are rerouting and actually exposing those services on web/websecure as denoted in your traefik config.
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
@grymlan Should I be adding "Host(traefik.domain.com) && (PathPrefix(/api) || PathPrefix(/dashboard))" to my Traefik's compose file ? I currently don't have traffic exposed outside the LAN.

Not unless you plan to expose or access your traefik dashboard. If you decide to do this latter I would use traefik middleware to add some basic authetication or you risk enumerating your entire infrastructure.

I do not plan on exposing my Traefik dashboard at all so no auth is necessary. Though I'm confused on what to add to traefik to correct it conflicting with Bitwarden. Since I didn't give it any host declaration.

What is this fix you're talking about on the Websockets ? Sorry I'm a little confused on this.

    command:
      #- "--log.level=DEBUG"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.network=traefik_proxy"
      - "--providers.file.directory=/etc/traefik"
      - "--providers.file.filename=config.toml"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.minecraft.address=:25565"
      - "--certificatesresolvers.mydnschallenge.acme.dnschallenge=true"
      - "--certificatesresolvers.mydnschallenge.acme.dnschallenge.provider=cloudflare"
      #- "--certificatesresolvers.mydnschallenge.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.mydnschallenge.acme.email=OMIT"
      - "--certificatesresolvers.mydnschallenge.acme.storage=/letsencrypt/acme.json"

I can try out your configuration and see if that works, but it's odd I was able to get it to work just fine with just a few lines. On traefik 2.0, but not 2.1

      - "traefik.enable=true"
      - "traefik.http.routers.bitwarden.rule=Host(`bitwarden.domain.com`)"
      - "traefik.http.routers.bitwarden.entrypoints=websecure"
      - "traefik.http.routers.bitwarden.tls.certresolver=mydnschallenge"
      - "traefik.http.services.bitwarden.loadbalancer.server.port=3012"
      - "traefik.docker.network=traefik_proxy"

Though my services are going to port 80. So does this mean Bitwarden is serving it's webserver on port 80 but using the websocket on 3012 ?

My mobile apps and extensions currently work with the configuration so what would be the point of the Websocket. My connections to the FQDN are indeed over HTTPS/443

Thank you !

I do not plan on exposing my Traefik dashboard at all so no auth is necessary. Though I'm confused on what to add to traefik to correct it conflicting with Bitwarden. Since I didn't give it any host declaration.

Oh, I misunderstood. I thought you had that declaration in there. Looking back I can see I was wrong. I'm not seeing a conflict at this point but I'm a bit perplexed as to why your screenshot does reflect bitwarden being served on port 80 given your traefik labels.

What is this fix you're talking about on the Websockets ? Sorry I'm a little confused on this.

My understanding is the websockets manage the notifications and background sync for mobile clients. With your 3012 being served on your loadbalancer perhaps its working...

Though my services are going to port 80. So does this mean Bitwarden is serving it's webserver on port 80 but using the websocket on 3012 ?

I'm not sure where port 80 is coming into play with the configurations you posted thus far. Would you mind posting a copy of your full docker compose file? And perhaps the config.toml as well? Your screenshot doesn't reflect the traefik labels which is perplexing me... Also, what's handling your requests to the traefik dashboard without a prefix deceleration somewhere? Odd... I'm suspecting the toml file has the key.

Sorry for the delayed response. I'm glad you're willing to take a look !

I do not plan on exposing my Traefik dashboard at all so no auth is necessary. Though I'm confused on what to add to traefik to correct it conflicting with Bitwarden. Since I didn't give it any host declaration.

Oh, I misunderstood. I thought you had that declaration in there. Looking back I can see I was wrong. I'm not seeing a conflict at this point but I'm a bit perplexed as to why your screenshot does reflect bitwarden being served on port 80 given your traefik labels.

What's going on here is definitely odd. What my assumption is the websockets along with the web serving are both being served on 443. Although Bitwarden gets clunky somtimes for the most part everything works as expected and I have been using this setup with Traefik 2 for a few months.

What is this fix you're talking about on the Websockets ? Sorry I'm a little confused on this.

My understanding is the websockets manage the notifications and background sync for mobile clients. With your 3012 being served on your loadbalancer perhaps its working...

Though my services are going to port 80. So does this mean Bitwarden is serving it's webserver on port 80 but using the websocket on 3012 ?

I'm not sure where port 80 is coming into play with the configurations you posted thus far. Would you mind posting a copy of your full docker compose file? And perhaps the config.toml as well? Your screenshot doesn't reflect the traefik labels which is perplexing me... Also, what's handling your requests to the traefik dashboard without a prefix deceleration somewhere? Odd... I'm suspecting the toml file has the key.

Sure ! I really appreciate another set of eyes. I don't beleive my config.toml is in conflict though lets see.

`config.toml`

[providers]
  [providers.file]
    filename = "config.toml"
    watch = true

`docker-compose.yml` containing Traefik

version: "3.3"

services:

  traefik:
    image: "traefik:v2.0"
    container_name: "traefik"
    restart: always
    command:
      #- "--log.level=DEBUG"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.network=traefik_proxy"
      - "--providers.file.directory=/etc/traefik"
      - "--providers.file.filename=config.toml"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.minecraft.address=:25565"
      - "--certificatesresolvers.mydnschallenge.acme.dnschallenge=true"
      - "--certificatesresolvers.mydnschallenge.acme.dnschallenge.provider=cloudflare"
      #- "--certificatesresolvers.mydnschallenge.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.mydnschallenge.acme.email="
      - "--certificatesresolvers.mydnschallenge.acme.storage=/letsencrypt/acme.json"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    environment:
      - "CF_API_EMAIL="
      - "CF_API_KEY="
    networks:
      traefik_proxy:
    volumes:
      - "$Docker/traefik/.letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "$Docker/traefik:/etc/traefik"

networks:
  traefik_proxy:
    external:
      name: traefik_proxy

`docker-compose.yml` containing Bitwarden

version: "3.3"

services:
  bitwarden:
    image: bitwardenrs/server
    restart: unless-stopped
    volumes:
      - $Docker/bitwarden:/data
    environment:
      WEBSOCKET_ENABLED: "true" # Required to use websockets
      SIGNUPS_ALLOWED: "true" # set to false to disable signups
    networks:
      - "traefik_proxy"
    ports:
      - "3012:3012"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.bitwarden.rule=Host(`bitwarden.domain.com`)"
      - "traefik.http.routers.bitwarden.entrypoints=websecure"
      - "traefik.http.routers.bitwarden.tls.certresolver=mydnschallenge"
      - "traefik.http.services.bitwarden.loadbalancer.server.port=80"
      - "traefik.docker.network=traefik_proxy"
networks:
  traefik_proxy:
    external:
      name: traefik_proxy

Huh, I guess at some point I indeed made it port 80. Super strange. I went back and check some previous docker-compose.yml backups and noticed I wasn't declaring a port and letting Traefik auto discover. I was also exposing 3012.

When I change traefik.http.services.bitwarden.loadbalancer.server.port=80 visiting Bitwarden from the FQDN I now get:

WebSocket Protocol Error: Unable to parse WebSocket key.

From the web browser.

What's the best place to go from here ? Using bitwarden to port 80 doesn't seem to work properly on on Traefik 2.1. I would assume it's best to use a configuration like yours considering the conflict with

"traefik.http.routers.api.rule=PathPrefix(`/dashboard/`) || PathPrefix(`/api`)"

dani-garcia / vaultwarden