search

juanfont / headscale

An open source, self-hosted implementation of the Tailscale control server
BSD 3-Clause "New" or "Revised" License
23.58k stars 1.3k forks source link

[Bug] failed to bind to TCP address: listen tcp :50443: bind: address already in use #2218

Open xopherKPMG opened 3 weeks ago

xopherKPMG commented 3 weeks ago

Is this a support request?

Is there an existing issue for this?

Current Behavior

I am trying to start a new headscale instance. I have done this several times in the past and thought it would be straightforward. However no matter what I do, I run into the following error:

$ sudo headscale serve
2024-10-31T12:50:15Z WRN 
WARN: The "dns.use_username_in_magic_dns" configuration key is deprecated and has been removed. Please see the changelog for more details.

2024-10-31T12:50:15Z INF Opening database database=sqlite3 path=/var/lib/headscale/db.sqlite
2024-10-31T12:50:16Z INF Setting up a DERPMap update worker frequency=86400000
2024-10-31T12:50:16Z INF Enabling remote gRPC at :50443
2024-10-31T12:50:16Z FTL ../runner/work/headscale/headscale/cmd/headscale/cli/serve.go:29 > Headscale ran into an error and had to shut down. error="failed to bind to TCP address: listen tcp :50443: bind: address already in use"

Expected Behavior

I expect headscale to run as normal, so that I am able to connect new nodes to it

Steps To Reproduce

  1. Follow the instructions from Headscale's documentation https://headscale.net/setup/install/official/#using-packages-for-debianubuntu-recommended
  2. sudo headscale server

headscale will look for a configuration file named config.yaml (or config.json) in the following order:

#

- /etc/headscale

- ~/.headscale

- current working directory

The url clients will connect to.

Typically this will be a domain like:

#

https://myheadscale.example.com:443

# server_url: https://********:443

Address to listen to / bind to on the server

#

For production:

listen_addr: 0.0.0.0:8080

listen_addr: 127.0.0.1:8080

Address to listen to /metrics, you may want

to keep this endpoint private to your internal

network

# metrics_listen_addr: 127.0.0.1:9090

Address to listen for gRPC.

gRPC is used for controlling a headscale server

remotely with the CLI

Note: Remote access only works if you have

valid certificates.

#

For production:

grpc_listen_addr: 0.0.0.0:50443

grpc_listen_addr: 127.0.0.1:50443

Allow the gRPC admin interface to run in INSECURE

mode. This is not recommended as the traffic will

be unencrypted. Only enable if you know what you

are doing.

grpc_allow_insecure: false

The Noise section includes specific configuration for the

TS2021 Noise protocol

noise:

The Noise private key is used to encrypt the

traffic between headscale and Tailscale clients when

using the new Noise-based protocol.

private_key_path: /var/lib/headscale/noise_private.key

List of IP prefixes to allocate tailaddresses from.

Each prefix consists of either an IPv4 or IPv6 address,

and the associated prefix length, delimited by a slash.

It must be within IP ranges supported by the Tailscale

client - i.e., subnets of 100.64.0.0/10 and fd7a:115c:a1e0::/48.

See below:

IPv6: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#LL81C52-L81C71

IPv4: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#L33

Any other range is NOT supported, and it will cause unexpected issues.

prefixes: v6: fd7a:115c:a1e0::/48 v4: 100.64.0.0/10

Strategy used for allocation of IPs to nodes, available options:

- sequential (default): assigns the next free IP from the previous given IP.

- random: assigns the next free IP from a pseudo-random IP generator (crypto/rand).

allocation: sequential

DERP is a relay system that Tailscale uses when a direct

connection cannot be established.

https://tailscale.com/blog/how-tailscale-works/#encrypted-tcp-relays-derp

#

headscale needs a list of DERP servers that can be presented

to the clients.

derp: server:

If enabled, runs the embedded DERP server and merges it into the rest of the DERP config

# The Headscale server_url defined above MUST be using https, DERP requires TLS to be in place
enabled: false

# Region ID to use for the embedded DERP server.
# The local DERP prevails if the region ID collides with other region ID coming from
# the regular DERP config.
region_id: 999

# Region code and name are displayed in the Tailscale UI to identify a DERP region
region_code: "headscale"
region_name: "Headscale Embedded DERP"

# Listens over UDP at the configured address for STUN connections - to help with NAT traversal.
# When the embedded DERP server is enabled stun_listen_addr MUST be defined.
#
# For more details on how this works, check this great article: https://tailscale.com/blog/how-tailscale-works/
stun_listen_addr: "0.0.0.0:3478"

# Private key used to encrypt the traffic between headscale DERP
# and Tailscale clients.
# The private key file will be autogenerated if it's missing.
#
private_key_path: /var/lib/headscale/derp_server_private.key

# This flag can be used, so the DERP map entry for the embedded DERP server is not written automatically,
# it enables the creation of your very own DERP map entry using a locally available file with the parameter DERP.paths
# If you enable the DERP server and set this to false, it is required to add the DERP server to the DERP map using DERP.paths
automatically_add_embedded_derp_region: true

# For better connection stability (especially when using an Exit-Node and DNS is not working),
# it is possible to optionally add the public IPv4 and IPv6 address to the Derp-Map using:
ipv4: 1.2.3.4
ipv6: 2001:db8::1

List of externally available DERP maps encoded in JSON

urls:

Disables the automatic check for headscale updates on startup

disable_check_updates: false

Time before an inactive ephemeral node is deleted?

ephemeral_node_inactivity_timeout: 30m

database:

Database type. Available options: sqlite, postgres

Please note that using Postgres is highly discouraged as it is only supported for legacy reasons.

All new development, testing and optimisations are done with SQLite in mind.

type: sqlite

Enable debug mode. This setting requires the log.level to be set to "debug" or "trace".

debug: false

GORM configuration settings.

gorm:

Enable prepared statements.

prepare_stmt: true

# Enable parameterized queries.
parameterized_queries: true

# Skip logging "record not found" errors.
skip_err_record_not_found: true

# Threshold for slow queries in milliseconds.
slow_threshold: 1000

SQLite config

sqlite: path: /var/lib/headscale/db.sqlite

# Enable WAL mode for SQLite. This is recommended for production environments.
# https://www.sqlite.org/wal.html
write_ahead_log: true

Postgres config

Please note that using Postgres is highly discouraged as it is only supported for legacy reasons.

See database.type for more information.

postgres:

If using a Unix socket to connect to Postgres, set the socket path in the 'host' field and leave 'port' blank.

host: localhost

port: 5432

name: headscale

user: foo

pass: bar

max_open_conns: 10

max_idle_conns: 10

conn_max_idle_time_secs: 3600

If other 'sslmode' is required instead of 'require(true)' and 'disabled(false)', set the 'sslmode' you need

in the 'ssl' field. Refers to https://www.postgresql.org/docs/current/libpq-ssl.html Table 34.1.

ssl: false

TLS configuration

#

Let's encrypt / ACME

#

headscale supports automatically requesting and setting up

TLS for a domain with Let's Encrypt.

#

URL to ACME directory

acme_url: https://acme-v02.api.letsencrypt.org/directory

Email to register with ACME provider

acme_email: ""

Domain name to request a TLS certificate for:

tls_letsencrypt_hostname: "*****"

Path to store certificates and metadata needed by

letsencrypt

For production:

tls_letsencrypt_cache_dir: /var/lib/headscale/cache

Type of ACME challenge to use, currently supported types:

HTTP-01 or TLS-ALPN-01

See docs/tls.md for more information

tls_letsencrypt_challenge_type: HTTP-01

When HTTP-01 challenge is chosen, letsencrypt must set up a

verification endpoint, and it will be listening on:

:http = port 80

tls_letsencrypt_listen: ":http"

Use already defined certificates:

tls_cert_path: "" tls_key_path: ""

log:

Output formatting for logs: text or json

format: text level: info

Policy

headscale supports Tailscale's ACL policies.

Please have a look to their KB to better

understand the concepts: https://tailscale.com/kb/1018/acls/

policy:

The mode can be "file" or "database" that defines

where the ACL policies are stored and read from.

mode: file

If the mode is set to "file", the path to a

HuJSON file containing ACL policies.

path: ""

DNS

#

headscale supports Tailscale's DNS configuration and MagicDNS.

Please have a look to their KB to better understand the concepts:

#

- https://tailscale.com/kb/1054/dns/

- https://tailscale.com/kb/1081/magicdns/

- https://tailscale.com/blog/2021-09-private-dns-with-magicdns/

#

Please note that for the DNS configuration to have any effect,

clients must have the --accept-dns=true option enabled. This is the

default for the Tailscale client. This option is enabled by default

in the Tailscale client.

#

Setting any of the configuration and --accept-dns=true on the

clients will integrate with the DNS manager on the client or

overwrite /etc/resolv.conf.

https://tailscale.com/kb/1235/resolv-conf

#

If you want stop Headscale from managing the DNS configuration

all the fields under dns should be set to empty values.

dns:

Whether to use MagicDNS.

Only works if there is at least a nameserver defined.

magic_dns: true

Defines the base domain to create the hostnames for MagicDNS.

This domain must be different from the server_url domain.

base_domain must be a FQDN, without the trailing dot.

The FQDN of the hosts will be

hostname.base_domain (e.g., myhost.example.com).

base_domain: redteam.com

List of DNS servers to expose to clients.

nameservers: global:

Unix socket used for the CLI to connect without authentication

Note: for production you will want to set this to something like:

unix_socket: /var/run/headscale/headscale.sock unix_socket_permission: "0770" #

headscale supports experimental OpenID connect support,

it is still being tested and might have some bugs, please

help us test it.

OpenID Connect

oidc:

only_start_if_oidc_is_available: true

issuer: "https://your-oidc.issuer.com/path"

client_id: "your-oidc-client-id"

client_secret: "your-oidc-client-secret"

Alternatively, set client_secret_path to read the secret from the file.

It resolves environment variables, making integration to systemd's

LoadCredential straightforward:

client_secret_path: "${CREDENTIALS_DIRECTORY}/oidc_client_secret"

client_secret and client_secret_path are mutually exclusive.

#

The amount of time from a node is authenticated with OpenID until it

expires and needs to reauthenticate.

Setting the value to "0" will mean no expiry.

expiry: 180d

#

Use the expiry from the token received from OpenID when the user logged

in, this will typically lead to frequent need to reauthenticate and should

only been enabled if you know what you are doing.

Note: enabling this will cause oidc.expiry to be ignored.

use_expiry_from_token: false

#

Customize the scopes used in the OIDC flow, defaults to "openid", "profile" and "email" and add custom query

parameters to the Authorize Endpoint request. Scopes default to "openid", "profile" and "email".

#

scope: ["openid", "profile", "email", "custom"]

extra_params:

domain_hint: example.com

#

List allowed principal domains and/or users. If an authenticated user's domain is not in this list, the

authentication request will be rejected.

#

allowed_domains:

- example.com

Note: Groups from keycloak have a leading '/'

allowed_groups:

- /headscale

allowed_users:

- alice@example.com

#

If strip_email_domain is set to true, the domain part of the username email address will be removed.

This will transform first-name.last-name@example.com to the user first-name.last-name

If strip_email_domain is set to false the domain part will NOT be removed resulting to the following

user: first-name.last-name.example.com

#

strip_email_domain: true

Logtail configuration

Logtail is Tailscales logging and auditing infrastructure, it allows the control panel

to instruct tailscale nodes to log their activity to a remote server.

logtail:

Enable logtail for this headscales clients.

As there is currently no support for overriding the log server in headscale, this is

disabled by default. Enabling this will make your clients send logs to Tailscale Inc.

enabled: false

Enabling this option makes devices prefer a random port for WireGuard traffic over the

default static port 41641. This option is intended as a workaround for some buggy

firewall devices. See https://tailscale.com/kb/1181/firewalls/ for more information.

randomize_client_port: false

Environment

- OS:Ubuntu 24.04.1 
- Headscale version:v0.23.0

$ sudo ss -tulnp|grep headscale
tcp   LISTEN 0      128             127.0.0.1:9090       0.0.0.0:*    users:(("headscale",pid=1861,fd=16))                   
tcp   LISTEN 0      128                     *:8080             *:*    users:(("headscale",pid=1861,fd=15))                   
tcp   LISTEN 0      128                     *:50443            *:*    users:(("headscale",pid=1861,fd=13))                   
tcp   LISTEN 0      128                     *:80               *:*    users:(("headscale",pid=1861,fd=14))      

NSG is allowing inbound traffic to ports 22, 80, 8080, 443, and 50443

Runtime environment

Anything else?

No response

nblock commented 3 weeks ago

Picked up your config and adjusted the "****" parts and paths to keys/db.

Started with (works):

sudo ./releases/0.23.0/headscale_0.23.0_linux_amd64 -c config.yaml serve
2024-10-31T14:44:25+01:00 WRN 
WARN: The "dns.use_username_in_magic_dns" configuration key is deprecated and has been removed. Please see the changelog for more details.

2024-10-31T14:44:25+01:00 INF Opening database database=sqlite3 path=db.sqlite
2024-10-31T14:44:25+01:00 INF Setting up a DERPMap update worker frequency=86400000
2024-10-31T14:44:25+01:00 INF Enabling remote gRPC at 0.0.0.0:50443
2024-10-31T14:44:25+01:00 INF listening and serving gRPC on: 0.0.0.0:50443
2024-10-31T14:44:25+01:00 INF listening and serving HTTP on: 0.0.0.0:8080
2024-10-31T14:44:25+01:00 INF listening and serving debug and metrics on: 127.0.0.1:9090

Are you really sure that no other instance is running (maybe as systemd service, maybe a container?)? I get the same error when trying to start headscale a second time in another terminal.

It seems you are running another config than the one you posted, have a look at the line:

2024-10-31T12:50:16Z INF Enabling remote gRPC at :50443

With your config it should say: Enabling remote gRPC at 0.0.0.0:50443

khairul169 commented 3 weeks ago

From your sudo ss -tulnp | grep headscale output, it seems that you already have an instance of Headscale running in the background (PID: 1861). You may want to try killing it.

If you’re running Headscale in a container, please check your list of running processes with docker ps | grep headscale and stop it if necessary.