Support custom tailscale socket path

[OverHash]

Welcome!

• [X] Yes, I've searched similar issues on GitHub and didn't find any.
• [X] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you expect to see?

Currently Traefik does not support setting a custom socket path to the tailscaled process.

On Unix systems, Traefik will assume that the path is at /var/run/tailscale/tailscaled.sock (see https://github.com/tailscale/tscert/blob/28a91b69a0467442178b62e2cfb9ab272ed3b64c/internal/paths/paths.go#L35-L37).

Some systems don't have the socket path at this location, and so it is necessary to specify a custom path to the socket.

tscert supports using the TS_SOCKET env variable to specify this location, but this does not currently work with traefik v3 / traefik master.

[kevinpollet]

Hello @OverHash, we opened https://github.com/traefik/traefik/pull/10668, could you test it to confirm it fixes the issue?

[OverHash]

Will test it out today!

[OverHash]

@kevinpollet can confirm that Traefik now refers to the correct socket path.

Doesn't fix my issue, unfortunately (for some reason the file is not recognized), but this is an improvement nonetheless.

Thanks for the swift work!

My issue is specifically in Docker logs:

router-traefik-1  | 2024-04-29T22:32:48Z ERR Unable to fetch certificate for domain "traefik-dashboard.[REDACTED].ts.net" error="Get \"http://local-tailscaled.sock/localapi/v0/cert/traefik-dashboard.[REDACTED].ts.net?type=pair\": dial unix \"/volume1/@appdata/Tailscale/tailscaled.sock\": connect: no such file or directory" providerName=myresolver.tailscale

but then this file does exist:

ls -alh /volume1/@appdata/Tailscale/tailscaled.sock
srw-rw-rw- 1 tailscale tailscale 0 Apr 16 13:25 /volume1/@appdata/Tailscale/tailscaled.sock

weird!

[emilevauge]

Hey @OverHash, Can you confirm you have mounted the socket inside the container? Do you run this command ls -alh /volume1/@appdata/Tailscale/tailscaled.sock from the container or the host?

[OverHash]

Hey @emilevauge I appreciate you sense checking me. I tried using the forums for my issue, but it seems there's not much activity that goes on there. Let me know if there is a better way for me to share my problem. I suspect not many people use Traefik v3 w/ Tailscale.

I do indeed mount the socket inside the container. My docker-compose.yml looks like

version: '3'
services:
  traefik:
    image: traefik:v3.0.0
    restart: unless-stopped
    volumes:
      - /var/packages/Tailscale/var/tailscaled.sock:/var/packages/Tailscale/var/tailscaled.sock:ro
    environment:
     - TS_SOCKET="/var/packages/Tailscale/var/tailscaled.sock"

an interesting point I had not noticed before: there is both a var/packages/Tailscale/var/tailscaled.sock and /volume1/@appdata/Tailscale/tailscaled.sock file. I elected to use the former since that's what running tailscale on my host machine shows (this is a Synology, if you could not tell already):

$ tailscale
[...]

FLAGS
  --socket string
        path to tailscaled socket (default /var/packages/Tailscale/var/tailscaled.sock)

which exists on both my host and (theoretically, on the mounted container):

user@host:/volume1/path/to/router$ ls -alh /var/packages/Tailscale/var/tailscaled.sock
srw-rw-rw- 1 tailscale tailscale 0 Apr 16 13:25 /var/packages/Tailscale/var/tailscaled.sock

unfortunately I can't figure out how to bash into the traefik docker container, as there is no /bin/bash, so I can't confirm that the traefik container also sees the sock file. I am still seeing:

router-traefik-1  | 2024-05-02T13:22:24Z ERR Unable to fetch certificate for domain "traefik-dashboard.[REDACTED].ts.net" error="Get \"http://local-tailscaled.sock/localapi/v0/cert/traefik-dashboard.[REDACTED].ts.net?type=pair\": dial unix \"/var/packages/Tailscale/var/tailscaled.sock\": connect: no such file or directory" providerName=myresolver.tailscale

Is there some networking I'm missing here? there's no mention of funny networking stuff I would need to do on the docs. I'm guessing this problem may not exist if I ran Tailscale on the same Docker network as Traefik (rather than on the host machine, as I currently do). Not quite sure how to proceed with figuring out this issue.

Thanks for your time!

[ch9hn]

We have exactly the same problem, but more specified for Kubernetes. Traefik doesn't find the Tailscale Socket:

2024-06-04T12:37:04Z ERR github.com/traefik/traefik/v3/pkg/provider/tailscale/provider.go:250 > Unable to fetch certificate for domain "xxxx.ts.net" error="Get \"http://local-tailscaled.sock/localapi/v0/cert/xxxxx.ts.net?type=pair\": dial unix /var/run/tailscale/tailscaled.sock: connect: no such file or directory" providerName=ts.tailscale

[wblew]

Random thought here... is your Traefik container user either root, or a tailscale operator?

[OverHash]

I tried to simplify the problem by moving Tailscale to my docker-compose.yml, however it seems the issue persists:

Relevant docker-compose.yml

My `docker-compose.yml`: ```yml version: '3' services: tailscale: image: tailscale/tailscale:latest container_name: ts-router hostname: tailscale environment: - TS_ACCEPT_DNS=true - TS_STATE_DIR=/var/lib/tailscale env_file: - .env ports: - 81:80 - 8080:8080 volumes: - ${PWD}/tailscale/state:/var/lib/tailscale - /dev/net/tun:/dev/net/tun cap_add: - net_admin - sys_module restart: unless-stopped traefik: image: traefik:v3.0.1 restart: unless-stopped networks: - proxy volumes: # let traefik subscribe to Docker events - /var/run/docker.sock:/var/run/docker.sock:ro - ./data/config.yml:/config.yml:ro - ./data/traefik.yml:/traefik.yml:ro labels: - "traefik.enable=true" - "traefik.http.routers.traefik.entrypoints=web" - "traefik.http.services.traefik.loadbalancer.server.port=8080" - "traefik.http.middlewares.traefik-auth.basicauth.users=USER:BASIC_AUTH_PASSWORD" - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https" - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https" - "traefik.http.routers.traefik.middlewares=traefik-https-redirect" - "traefik.http.routers.traefik-secure.entrypoints=websecure" - "traefik.http.routers.traefik-secure.rule=Host(`traefik-dashboard.local.example.com`)" - "traefik.http.routers.traefik-secure.middlewares=traefik-auth" - "traefik.http.routers.traefik-secure.tls=true" - "traefik.http.routers.traefik-secure.tls.certresolver=myresolver" - "traefik.http.routers.traefik-secure.tls.domains[0].main=traefik-dashboard.[REDACTED].ts.net" - "traefik.http.routers.traefik-secure.service=api@internal" depends_on: - tailscale network_mode: service:tailscale networks: proxy: external: true ``` and I then have a `data/traefik.yml` file with the following: ```yml providers: docker: endpoint: "unix:///var/run/docker.sock" exposedByDefault: false file: filename: /config.yml # empty file here certificatesResolvers: myresolver: tailscale: {} ```

when I review the logs of the docker container, I see that the tailscale component has successfully authenticated and is streaming traffic. However, traefik is not happy and does not use Tailscale to acquire the HTTPS cert:

router-traefik-1  | 2024-06-10T05:17:34Z ERR Unable to fetch certificate for domain "traefik-dashboard.[REDACTED].ts.net" error="Get \"http://local-tailscaled.sock/localapi/v0/cert/traefik-dashboard.[REDACTED].ts.net?type=pair\": dial unix /var/run/tailscale/tailscaled.sock: connect: no such file or directory" providerName=myresolver.tailscale

My suspicion is that I have to actually pass through the .sock file from the tailscale container to the traefik container, however this does not align with the tailscale guide on the matter, nor the traefik documentation.

[wblew]

My suspicion is that I have to actually pass through the .sock file from the tailscale container to the traefik container, however this does not align with the tailscale guide on the matter, nor the traefik documentation.

Last night, I spent some time looking into Traefik's tailscale certificate provider. It's using a tailscale cert golang library. I don't find it needing the tailscaled socket surprising. Documentation? meh, It's frequently out of date.

[OverHash]

I see that even a few months ago users have been having this issue, with reports of the same .sock problem. I'm out of moves now, so hoping a new stone to turn shows up.

I just tried making a volume in my docker-compose.yml and mounting that volume to /var/run/tailscale in both the tailscale and traefik container, but it does not resolve the issue.

Regardless, happy to hear another persons input on this matter :)

[OverHash]

After reviewing #9772, I managed to get somewhere by using my original configuration I posted in this issue (where my host has tailscale installed, and I am trying to pass it through to the traefik in a Docker container), alongside

security_opt:
     - label:disable

in the traefik container docker-compose.yml configuration.

The original issue of that thread persists: you get a unexpected output: no delimiter unless you configure traefik to acquire tailscale certs from the name of the host Tailscale machine. It would be lovely if there was a way to allow for traefik to get custom DNS names under your tailnet.

Maintainers, the original response in that thread was to post a new issue on the tscert library about this issue. I see nobody has done so. Should I open a tracking issue under traefik as well regarding this problem of unexpected output: no delimiter?

[lumnikemel]

@OverHash What did you end up doing? No matter what I do, I keep getting the unexpected output: no delimiter error.