Basic Auth - Critical worker timeout error

[iFargle]

[2023-03-25 12:03:28 +0700] [1] [INFO] Starting gunicorn 20.1.0
[2023-03-25 12:03:28 +0700] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2023-03-25 12:03:28 +0700] [1] [INFO] Using worker: sync
[2023-03-25 12:03:28 +0700] [8] [INFO] Booting worker with pid: 8
[2023-03-25 12:03:30,351] INFO in server: Headscale-WebUI Version:  v0.5.6 / main
[2023-03-25 12:03:30,351] INFO in server: LOG LEVEL SET TO DEBUG
[2023-03-25 12:03:30,351] INFO in server: DEBUG STATE:  True
[2023-03-25 12:03:30,352] INFO in server: Loading basic auth libraries and configuring app...
[2023-03-25 12:06:38 +0700] [1] [CRITICAL] WORKER TIMEOUT (pid:8)
[2023-03-25 12:06:38 +0700] [8] [INFO] Worker exiting (pid: 8)
[2023-03-25 12:06:38 +0700] [9] [INFO] Booting worker with pid: 9
[2023-03-25 12:06:39,526] INFO in server: Headscale-WebUI Version:  v0.5.6 / main
[2023-03-25 12:06:39,526] INFO in server: LOG LEVEL SET TO DEBUG
[2023-03-25 12:06:39,527] INFO in server: DEBUG STATE:  True
[2023-03-25 12:06:39,527] INFO in server: Loading basic auth libraries and configuring app...

Just tried today and keep getting bad gateway when trying to access the webui. Everytime I tried to access the URL, I always got worker exiting in the log.

Any idea what's wrong?

Originally posted by @inlophe in https://github.com/iFargle/headscale-webui/issues/60#issuecomment-1483728482

[iFargle]

First question: Are you using Podman or Docker? Can you paste your docker-compose.yml?

[iFargle]

Also: What is the full path of the URL you are trying to access? SCRIPT_NAME is the base path for your application, so if you're hosting on http://localhost/ and your SCRIPT_NAME is set to /admin (which I think I have the docker-compose.yml as /admin), you would need to access it from http://localhost/admin

[inlophe]

I'm using docker. This is my compose

version: '3.9'

services:
  headscale:
    image: headscale/headscale:latest
    pull_policy: always
    container_name: headscale
    restart: unless-stopped
    command: headscale serve
    networks:
      - proxy
    volumes:
      - /opt/container/headscale/config:/etc/headscale
      - /opt/container/headscale/data:/var/lib/headscale
    labels:
      - traefik.enable=true
      - traefik.http.routers.headscale.rule=PathPrefix(`/`) && Host(`[mydomain]`) # you might want to add: && Host(`your.domain.name`)"
      - traefik.http.services.headscale.loadbalancer.server.port=8080

  headscale-webui:
    image: ghcr.io/ifargle/headscale-webui:latest
    container_name: headscale-webui
    environment:
      - TZ=Asia/Jakarta
      - COLOR=red                              # Use the base colors (ie, no darken-3, etc) - 
      - HS_SERVER=https://[mydomain]    # Reachable endpoint for your Headscale server
      - DOMAIN_NAME=https://[mydomain]  # The base domain name for this container.
      - SCRIPT_NAME=/admin                     # This is your applications base path (wsgi requires the name "SCRIPT_NAME")
      - KEY="[xxxsecretxxx]"             # Generate with "openssl rand -base64 32" - used to encrypt your key on disk.
      - AUTH_TYPE=basic                         # AUTH_TYPE is either Basic or OIDC.  Empty for no authentication
      - LOG_LEVEL=DEBUG                         # Log level.  "DEBUG", "ERROR", "WARNING", or "INFO".  Default "INFO"
      # ENV for Basic Auth (Used only if AUTH_TYPE is "Basic").  Can be omitted if you aren't using Basic Auth
      - BASIC_AUTH_USER=[someuser]                   # Used for basic auth
      - BASIC_AUTH_PASS=[somepassword]                   # Used for basic auth
      # ENV for OIDC (Used only if AUTH_TYPE is "OIDC").  Can be omitted if you aren't using OIDC
      #- OIDC_AUTH_URL=https://auth.$DOMAIN/.well-known/openid-configuration # URL for your OIDC issuer's well-known endpoint
      #- OIDC_CLIENT_ID=headscale-webui         # Your OIDC Issuer's Client ID for Headscale-WebUI
      #- OIDC_CLIENT_SECRET=YourSecretHere      # Your OIDC Issuer's Secret Key for Headscale-WebUI
    volumes:
      - /opt/container/headscale/ui/data:/data                         # Headscale-WebUI's storage.  Make sure ./volume is readable by UID 1000 (chown 1000:1000 ./volume)
      - /opt/container/headscale/config:/etc/headscale:ro # Headscale's config storage location.  Used to read your Headscale config.
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.http.routers.headscale-webui.entrypoints=https
      - traefik.http.routers.headscale-webui.rule=Host(`[mydomain]`) && (PathPrefix(`/admin/`) || PathPrefix(`/admin`))
      - traefik.http.services.headscale-webui.loadbalancer.server.port=5000

networks:
  proxy:
    external: true

I'm using subdomain the same subdomain for headscale and webui with the difference adding /admin after the subdomain (https://net.[mydomain]/admin

EDIT: When I access the URL, it prompted me for the basic auth, after entering the correct credentials it just keep loading for a minute or two until it gives me bad gateway

[iFargle]

Are you trying to load /overview?

[iFargle]

no scratch that... Hmmm... I can't see anything wrong with this config. Let me do some tests.

[iFargle]

can you run with no auth and see if it still times out?

[inlophe]

Ok, I'll try it

[inlophe]

Before that, one quick question. I need to mount headscale config folder to webui container right? Then the permission inside the webui container will be like this:

~ $ ls -l /etc/headscale/
total 80
-rw-r--r--    1 app      app          11501 Mar 25 09:54 config.yaml
-rw-r--r--    1 root     root         57344 Mar 25 10:53 db.sqlite
-rw-------    1 root     root            72 Mar 25 10:29 noise_private.key
-rw-------    1 root     root            72 Mar 25 10:29 private.key

db and key owned by root. Is this correct?

[iFargle]

yeah that looks fine. headscale-webui only accesses the config.yaml, the other files don't matter. And your permissions look good -- just need read only

[inlophe]

Removing the auth doesn't work. Still getting WORKER timeout in log, and bad gateway on the browser

[iFargle]

Give ghcr.io/ifargle/headscale-webui:testing a shot. I've made a few changes, though I'm unsure if I've made changes that would impact this error. Still worth a shot though.

It's under heavy testing so if you pull now and it works don't pull again until v0.6.0 is pushed to main, it may break lol

[iFargle]

oh also worth asking: ARM or x86? Testing is only available as x86

[inlophe]

Mine's x86, Debian 11

EDIT: still not working with testing. I'll try to spin up fresh server to test further

[inlophe]

Welp, turns out my nftables config is the one causing the issue. I'll try to look deeper into what's causing it.

[inlophe]

Weird, my nftables input chain default to drop and that chain is the one causing it. Isn't docker container suppose to use it's own docker chain? (at least that's how it is for me until now)

[iFargle]

That's interesting.... Glad you figured it out. The application shouldn't be doing anything funky with Docker at all -- At least as far as I know.

Let me know if you have any more issues. Thanks!

[inlophe]

I think I found the problem.

Mar 25 10:24:04 auxiliary kernel: [8744276.793412] Dropped: IN=br-87c38579dda4 OUT= PHYSIN=vetha087dad MAC=02:42:f9:92:64:15:02:42:ac:12:00:03:08:00 SRC=172.18.0.3 DST=[public IP] LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=8897 DF PROTO=TCP SPT=48850 DPT=443 WINDOW=64240 RES=0x00 SYN URGP=0

Found this after trying to log the dropped packet of the firewall. I think this is webui trying to connect to headscale (judging by the destination port - 443) using domain/public ip? That's why the firewall drop it, because it is not using docker internal network (I think) and coming in to the bridge interface (IN=br-87c38579dda4) instead of the host network interface (the manual nftables rules only allow connection to certain listed port from the host network interface).

Is there particular reason why the webui does not probe the headscale using the internal docker network (using the usual service name and port)?

[iFargle]

That's the HS_SERVER variable. You can use the internal IP:port there :)

On March 25, 2023 4:26:19 PM UTC, inlophe @.***> wrote:

I think I found the problem.

Mar 25 10:24:04 auxiliary kernel: [8744276.793412] Dropped: IN=br-87c38579dda4 OUT= PHYSIN=vetha087dad MAC=02:42:f9:92:64:15:02:42:ac:12:00:03:08:00 SRC=172.18.0.3 DST=[public IP] LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=8897 DF PROTO=TCP SPT=48850 DPT=443 WINDOW=64240 RES=0x00 SYN URGP=0

Found this after trying to log the dropped packet of the firewall. I think this is webui trying to connect to headscale (judging by the destination port - 443) using domain/public ip? That's why the firewall drop it, because it is not using docker internal network (I think) and coming in to the bridge interface (IN=br-87c38579dda4) instead of the host network interface (the manual nftables rules only allow connection to certain listed port from the host network interface).

Is there particular reason why the webui does not probe the headscale using the internal docker network (using the usual service name and port)?

-- Reply to this email directly or view it on GitHub: https://github.com/iFargle/headscale-webui/issues/61#issuecomment-1483863237 You are receiving this because you modified the open/close state.

Message ID: @.> -- Albert J. Copeland @. PGP Fingerprint:  64F6C4EB46C4543A