Daytona server doesn't start when the local machine already has a tailnet client connecting to a self-hosted Headscale control server

[shuguet]

Describe the bug

I run a personal Headscale server, and have tailscale deployed on every single one of my machines. That means that all my machines have a customized login/control server variable for the Tailscale client, especially on Windows, where this requires a registry key change, pointing to my self-hosted Headscale instance.

When attempting to run daytona server on my Windows-based laptop, I ran into this issue in the logs:

[...]
2024/03/12 09:51:10 LocalBackend state is NeedsLogin; running StartLoginInteractive...
2024/03/12 09:51:10 StartLoginInteractive: url=false
2024/03/12 09:51:10 control: client.Login(false, 10)
2024/03/12 09:51:10 control: [v1] mapRoutine: context done.
2024/03/12 09:51:10 control: [v1] mapRoutine: state:new
2024/03/12 09:51:10 control: [v1] authRoutine: context done.
2024/03/12 09:51:10 control: [v1] authRoutine: state:new; wantLoggedIn=true
2024/03/12 09:51:10 control: [v1] direct.TryLogin(token=false, flags=10)
2024/03/12 09:51:10 control: LoginInteractive -> regen=true
2024/03/12 09:51:10 control: doLogin(regen=true, hasUrl=false)
2024/03/12 09:51:10 control: control server key from https://<my.headscale.server>: ts2021=[NI15q], legacy=[mVUHl]
2024/03/12 09:51:10 control: Generating a new nodekey.
2024/03/12 09:51:10 control: RegisterReq: onode= node=[FMVB/] fup=false nks=false
2024/03/12 09:51:10 control: [v1] creating new noise client
2024/03/12 09:51:10 control: [v1] TryLogin: register request: http 401: {"User":{"ID":0,"DisplayName":"","ProfilePicURL":"","Domain":"","Logins":null,"Created":"0001-01-01T00:00:00Z"},"Login":{"ID":0,"Provider":"","LoginName":"","DisplayName":"","ProfilePicURL":"","Domain
2024/03/12 09:51:10 control: [v1] sendStatus: authRoutine-report: state:authenticating
2024/03/12 09:51:10 control: authRoutine: [v1] backoff: 13 msec
2024/03/12 09:51:10 Received error: register request: http 401: {"User":{"ID":0,"DisplayName":"","ProfilePicURL":"","Domain":"","Logins":null,"Created":"0001-01-01T00:00:00Z"},"Login":{"ID":0,"Provider":"","LoginName":"","DisplayName":"","ProfilePicURL":"","Domain
2024/03/12 09:51:10 control: [v1] authRoutine: state:authenticating; wantLoggedIn=true
2024/03/12 09:51:10 control: [v1] direct.TryLogin(token=false, flags=10)
2024/03/12 09:51:10 control: LoginInteractive -> regen=true
2024/03/12 09:51:10 control: doLogin(regen=true, hasUrl=false)
2024/03/12 09:51:10 control: Generating a new nodekey.
2024/03/12 09:51:10 control: RegisterReq: onode= node=[v3eTY] fup=false nks=false
2024/03/12 09:51:10 control: [v1] TryLogin: register request: http 401: {"User":{"ID":0,"DisplayName":"","ProfilePicURL":"","Domain":"","Logins":null,"Created":"0001-01-01T00:00:00Z"},"Login":{"ID":0,"Provider":"","LoginName":"","DisplayName":"","ProfilePicURL":"","Domain
2024/03/12 09:51:10 control: [v1] sendStatus: authRoutine-report: state:authenticating
2024/03/12 09:51:10 control: authRoutine: [v1] backoff: 37 msec
[...]

And then it keeps repeating/backing off for longer and longer, never moving. I looked around, but I don't see any obvious way to either register daytona with my existing headscale deployment (preferred), or force the client that the daytona server spins up to connect to the specific local headscale server that also gets spun up in C:\Users\<login>\AppData\Roaming\tsnet-daytona

To Reproduce Steps to reproduce the behavior:

1. Install tailscale on a Windows machine (not tested on other OSes)
2. Deploy headscale on a seperate system
3. Properly connect the tailscale client on the Windows system to the Headscale server (Registery keys and all)
4. Launch daytona server on the Windows machine
5. See error

Expected behavior Either Daytona should support registering the daytona server with the existing custom Headscale server OR Daytona should re-use the existing tailscale connection from the local machine to the Headscale server (preferred, if possible)

Screenshots Logs provided above instead

Desktop (please complete the following information):

• OS: Windows 11, Version 23H2 (OS Build: 22631.3155)
• Browser: Brave (irrelevant here)
• Daytona Version: v0.2.0

Additional context None that I think I relevant here, but feel free to ask

[shuguet]

Originally posted/discussed on Slack here: https://daytonacommunity.slack.com/archives/C067LG7678W/p1710234126460089

[Tpuljak]

@shuguet thanks for such a detailed issue. This is something that we would love to support and we definitely will try to.

[daytonaBot]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 14 days). It will be closed if no further activity occurs. Thanks for the feedback.

[shuguet]

The issue is still relevant :) (Unsure if there is a /command to use for the bot to remove the tag, can't seem to be able to do it myself).

[Tpuljak]

@shuguet you're right, that was a mistake by the bot.

We are aware this is still an issue, we have @gdraganic working on this and some other tailscale improvements.

[shuguet]

@shuguet you're right, that was a mistake by the bot.

I didn't think it was a mistake by the bot, most likely it is doing it's job of marking issues that have not received any update in 2 weeks as stale :) Generally, with most GitHub bots I'm familiar with, there's either a command like (/remove-lifecycle stale with Prow for example), or simply adding a comment cancel out the stale label/status. Looks like yours require a manual intervention of an authorized user to remove the stale label (since I could not remove the label myself).

We are aware this is still an issue, we have @gdraganic working on this and some other tailscale improvements.

Thank you 🙏 Let me know if there's anything you need/I can help with.

[Tpuljak]

@shuguet apologies for not picking this up sooner. We tested creating and ssh-ing into a workspace while connected to an external headscale server and everything seems to be working okay.

I'll close this issue for now and open a new one that will allow users to configure their own headscale server instance to connect to.