search

jonathanspw / headscale-rpm

4 stars 4 forks source link

headscale doesn't stop gracefully #2

Open dustymabe opened 1 year ago

dustymabe commented 1 year ago

If I systemctl stop headscale it doesn't appear to stop gracefully.

This may be because I'm running it in a container. Is anyone else able to reproduce?

This is with headscale-0.22.3-1.aarch64 from the copr repo.


[root@c5b9776eab18 /]# systemctl stop headscale
Jul 06 17:23:43 c5b9776eab18 headscale[26]: 2023-07-06T17:23:43-04:00 INF Client is ready to access the tailnet handler=PollNetMap machine=hattopvm noise=true
Jul 06 17:23:43 c5b9776eab18 headscale[26]: 2023-07-06T17:23:43-04:00 INF Sending initial map handler=PollNetMap machine=hattopvm noise=true
Jul 06 17:23:43 c5b9776eab18 headscale[26]: 2023-07-06T17:23:43-04:00 INF Notifying peers handler=PollNetMap machine=hattopvm noise=true
Jul 06 17:23:46 c5b9776eab18 headscale[26]: 2023-07-06T17:23:46-04:00 INF The client has closed the connection handler=PollNetMapStream machine=hattopvm
Jul 06 17:23:46 c5b9776eab18 headscale[26]: 2023-07-06T17:23:46-04:00 INF Client is ready to access the tailnet handler=PollNetMap machine=hattopvm noise=true
Jul 06 17:23:46 c5b9776eab18 headscale[26]: 2023-07-06T17:23:46-04:00 INF Sending initial map handler=PollNetMap machine=hattopvm noise=true
Jul 06 17:23:46 c5b9776eab18 headscale[26]: 2023-07-06T17:23:46-04:00 INF Notifying peers handler=PollNetMap machine=hattopvm noise=true
Jul 06 17:27:38 c5b9776eab18 headscale[26]: 2023-07-06T17:27:38-04:00 INF Received signal to stop, shutting down gracefully signal=terminated
Jul 06 17:27:38 c5b9776eab18 headscale[26]: 2023-07-06T17:27:38-04:00 INF The long-poll handler is shutting down handler=PollNetMapStream machine=hattopvm noise=true
Jul 06 17:27:38 c5b9776eab18 headscale[26]: 2023-07-06T17:27:38-04:00 INF The long-poll handler is shutting down handler=PollNetMapStream machine=vanilla-f37 noise=true
Jul 06 17:27:38 c5b9776eab18 systemd[1]: Stopping headscale.service - headscale coordination server for Tailscale...
Jul 06 17:27:38 c5b9776eab18 headscale[26]: 2023-07-06T17:27:38-04:00 INF Headscale stopped
Jul 06 17:27:42 c5b9776eab18 headscale[26]: 2023-07-06T17:27:42-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:42 c5b9776eab18 headscale[26]: 2023-07-06T17:27:42-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:27:42 c5b9776eab18 headscale[26]: 2023-07-06T17:27:42-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:43 c5b9776eab18 headscale[26]: 2023-07-06T17:27:43-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:27:45 c5b9776eab18 headscale[26]: 2023-07-06T17:27:45-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:27:47 c5b9776eab18 headscale[26]: 2023-07-06T17:27:47-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:47 c5b9776eab18 headscale[26]: 2023-07-06T17:27:47-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:47 c5b9776eab18 headscale[26]: 2023-07-06T17:27:47-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:27:51 c5b9776eab18 headscale[26]: 2023-07-06T17:27:51-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:27:52 c5b9776eab18 headscale[26]: 2023-07-06T17:27:52-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:52 c5b9776eab18 headscale[26]: 2023-07-06T17:27:52-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:52 c5b9776eab18 headscale[26]: 2023-07-06T17:27:52-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:27:56 c5b9776eab18 headscale[26]: 2023-07-06T17:27:56-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:27:57 c5b9776eab18 headscale[26]: 2023-07-06T17:27:57-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:57 c5b9776eab18 headscale[26]: 2023-07-06T17:27:57-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:27:57 c5b9776eab18 headscale[26]: 2023-07-06T17:27:57-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:28:02 c5b9776eab18 headscale[26]: 2023-07-06T17:28:02-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:02 c5b9776eab18 headscale[26]: 2023-07-06T17:28:02-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:02 c5b9776eab18 headscale[26]: 2023-07-06T17:28:02-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:28:03 c5b9776eab18 headscale[26]: 2023-07-06T17:28:03-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:28:07 c5b9776eab18 headscale[26]: 2023-07-06T17:28:07-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:07 c5b9776eab18 headscale[26]: 2023-07-06T17:28:07-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:07 c5b9776eab18 headscale[26]: 2023-07-06T17:28:07-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:28:07 c5b9776eab18 headscale[26]: 2023-07-06T17:28:07-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:28:12 c5b9776eab18 headscale[26]: 2023-07-06T17:28:12-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:12 c5b9776eab18 headscale[26]: 2023-07-06T17:28:12-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:12 c5b9776eab18 headscale[26]: 2023-07-06T17:28:12-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:28:16 c5b9776eab18 headscale[26]: 2023-07-06T17:28:16-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:28:17 c5b9776eab18 headscale[26]: 2023-07-06T17:28:17-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:17 c5b9776eab18 headscale[26]: 2023-07-06T17:28:17-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:17 c5b9776eab18 headscale[26]: 2023-07-06T17:28:17-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:28:19 c5b9776eab18 headscale[26]: 2023-07-06T17:28:19-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:28:22 c5b9776eab18 headscale[26]: 2023-07-06T17:28:22-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:22 c5b9776eab18 headscale[26]: 2023-07-06T17:28:22-04:00 ERR Error listing users error="sql: database is closed"
Jul 06 17:28:22 c5b9776eab18 headscale[26]: 2023-07-06T17:28:22-04:00 ERR error getting routes error="sql: database is closed"
Jul 06 17:28:23 c5b9776eab18 headscale[26]: 2023-07-06T17:28:23-04:00 ERR Failed to fetch machine from the database with node key: 
Jul 06 17:28:24 c5b9776eab18 systemd[1]: headscale.service: State 'stop-sigterm' timed out. Aborting.
Jul 06 17:28:24 c5b9776eab18 systemd[1]: headscale.service: Killing process 26 (headscale) with signal SIGABRT.
Jul 06 17:28:24 c5b9776eab18 headscale[26]: SIGABRT: abort
Jul 06 17:28:24 c5b9776eab18 headscale[26]: PC=0xaaaae1d80110 m=2 sigcode=0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 0 [idle]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.futex()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/sys_linux_arm64.s:663 +0x20 fp=0xffff81659710 sp=0xffff81659710 pc=0xaaaae1d80110
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.futexsleep(0x2ae5452697b88?, 0xb83c3?, 0xffff81659798?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/os_linux.go:75 +0x90 fp=0xffff81659760 sp=0xffff81659710 pc=0xaaaae1d44c30
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.notetsleep_internal(0xaaaae368e3c0, 0x9635ee87)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/lock_futex.go:202 +0xcc fp=0xffff816597a0 sp=0xffff81659760 pc=0xaaaae1d1b89c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.notetsleep(0x2ae5452697797?, 0xffff816597f0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/lock_futex.go:225 +0x34 fp=0xffff816597c0 sp=0xffff816597a0 pc=0xaaaae1d1b9b4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.sysmon()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:5333 +0x220 fp=0xffff81659830 sp=0xffff816597c0 pc=0xaaaae1d56cb0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.mstart1()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:1498 +0x88 fp=0xffff81659860 sp=0xffff81659830 pc=0xaaaae1d4da28
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.mstart0()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:1455 +0x70 fp=0xffff81659890 sp=0xffff81659860 pc=0xaaaae1d4d980
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.mstart()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:128 +0x10 fp=0xffff816598a0 sp=0xffff81659890 pc=0xaaaae1d7c5f0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 1 [semacquire, 45 minutes]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0xaaaae3698ca0?, 0xaaaae1d1d16c?, 0x60?, 0x0?, 0x18?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000859640 sp=0x4000859620 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goparkunlock(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:387
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.semacquire1(0x4000706190, 0x80?, 0x1, 0x0, 0xd0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/sema.go:160 +0x21c fp=0x40008596a0 sp=0x4000859640 pc=0xaaaae1d5cf6c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: sync.runtime_Semacquire(0x4000ad1718?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/sema.go:62 +0x2c fp=0x40008596e0 sp=0x40008596a0 pc=0xaaaae1d7a72c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: sync.(*WaitGroup).Wait(0x4000706188)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/sync/waitgroup.go:116 +0x78 fp=0x4000859700 sp=0x40008596e0 pc=0xaaaae1d8ad68
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/sync/errgroup.(*Group).Wait(0x4000706180)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/sync/errgroup/errgroup.go:53 +0x2c fp=0x4000859720 sp=0x4000859700 pc=0xaaaae2332a9c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).Serve(0x40000cacc0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:833 +0x1410 fp=0x4000859ac0 sp=0x4000859720 pc=0xaaaae27fff20
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/cmd/headscale/cli.glob..func26(0xaaaae35989a0?, {0xaaaae28e1118?, 0x0?, 0x0?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/cmd/headscale/cli/server.go:24 +0x74 fp=0x4000859b00 sp=0x4000859ac0 pc=0xaaaae28a6424
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/spf13/cobra.(*Command).execute(0xaaaae35989a0, {0xaaaae36c3f00, 0x0, 0x0})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/spf13/cobra/command.go:944 +0x5b0 fp=0x4000859c40 sp=0x4000859b00 pc=0xaaaae2856910
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/spf13/cobra.(*Command).ExecuteC(0xaaaae3597580)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/spf13/cobra/command.go:1068 +0x35c fp=0x4000859d00 sp=0x4000859c40 pc=0xaaaae285707c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/spf13/cobra.(*Command).Execute(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/spf13/cobra/command.go:992
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/cmd/headscale/cli.Execute()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/cmd/headscale/cli/root.go:101 +0x28 fp=0x4000859d50 sp=0x4000859d00 pc=0xaaaae28a4a28
Jul 06 17:28:24 c5b9776eab18 headscale[26]: main.main()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/cmd/headscale/headscale.go:56 +0x3b0 fp=0x4000859f70 sp=0x4000859d50 pc=0xaaaae28d0960
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.main()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:250 +0x248 fp=0x4000859fd0 sp=0x4000859f70 pc=0xaaaae1d4ad18
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000859fd0 sp=0x4000859fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 2 [force gc (idle), 3 minutes]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x2ae44060a3878?, 0x0?, 0x0?, 0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x400005cfa0 sp=0x400005cf80 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goparkunlock(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:387
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.forcegchelper()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:305 +0xb8 fp=0x400005cfd0 sp=0x400005cfa0 pc=0xaaaae1d4af88
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400005cfd0 sp=0x400005cfd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by runtime.init.7
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:293 +0x24
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 3 [GC sweep wait]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x400005d760 sp=0x400005d740 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goparkunlock(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:387
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.bgsweep(0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgcsweep.go:319 +0x110 fp=0x400005d7b0 sp=0x400005d760 pc=0xaaaae1d35740
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gcenable.func1()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:178 +0x28 fp=0x400005d7d0 sp=0x400005d7b0 pc=0xaaaae1d2a268
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400005d7d0 sp=0x400005d7d0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by runtime.gcenable
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:178 +0x74
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 4 [GC scavenge wait]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x2ae449c5114f5?, 0x1087cc?, 0x0?, 0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x400005df50 sp=0x400005df30 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goparkunlock(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:387
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.(*scavengerState).park(0xaaaae368cec0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgcscavenge.go:400 +0x5c fp=0x400005df80 sp=0x400005df50 pc=0xaaaae1d335cc
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.bgscavenge(0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgcscavenge.go:633 +0xac fp=0x400005dfb0 sp=0x400005df80 pc=0xaaaae1d33b8c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gcenable.func2()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:179 +0x28 fp=0x400005dfd0 sp=0x400005dfb0 pc=0xaaaae1d2a208
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400005dfd0 sp=0x400005dfd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by runtime.gcenable
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:179 +0xb8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 5 [finalizer wait, 43 minutes]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x0?, 0x4000710108?, 0x50?, 0x82?, 0x1000000010?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000636d80 sp=0x4000636d60 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.runfinq()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mfinal.go:193 +0x10c fp=0x4000636fd0 sp=0x4000636d80 pc=0xaaaae1d292fc
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000636fd0 sp=0x4000636fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by runtime.createfing
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mfinal.go:163 +0x84
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 19 [GC worker (idle)]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x2ae44063d3359?, 0x4000091128?, 0x98?, 0xe7?, 0xaaaae2302828?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x400005e740 sp=0x400005e720 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gcBgMarkWorker()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:1275 +0xec fp=0x400005e7d0 sp=0x400005e740 pc=0xaaaae1d2c1ac
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400005e7d0 sp=0x400005e7d0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by runtime.gcBgMarkStartWorkers
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:1199 +0x28
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 20 [GC worker (idle)]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x2ae44063d36e8?, 0x0?, 0x0?, 0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000058740 sp=0x4000058720 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gcBgMarkWorker()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:1275 +0xec fp=0x40000587d0 sp=0x4000058740 pc=0xaaaae1d2c1ac
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40000587d0 sp=0x40000587d0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by runtime.gcBgMarkStartWorkers
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/mgc.go:1199 +0x28
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 26 [select, 7 minutes]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x400006df88?, 0x2?, 0x80?, 0x1?, 0x400006df74?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x400006de10 sp=0x400006ddf0 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.selectgo(0x400006df88, 0x400006df70, 0x4000343f78?, 0x0, 0x4000343f98?, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/select.go:327 +0x690 fp=0x400006df30 sp=0x400006de10 pc=0xaaaae1d5c040
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/patrickmn/go-cache.(*janitor).Run(0x40002117b0, 0x4000220840?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/patrickmn/go-cache/cache.go:1079 +0x78 fp=0x400006dfb0 sp=0x400006df30 pc=0xaaaae22e42f8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/patrickmn/go-cache.runJanitor.func1()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0x2c fp=0x400006dfd0 sp=0x400006dfb0 pc=0xaaaae22e44fc
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400006dfd0 sp=0x400006dfd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by github.com/patrickmn/go-cache.runJanitor
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0x100
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 10 [chan receive]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x400007eba0?, 0x4?, 0xc0?, 0xbb?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000addea0 sp=0x4000adde80 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv(0x40006aed20, 0x4000addf90, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:583 +0x45c fp=0x4000addf30 sp=0x4000addea0 pc=0xaaaae1d1667c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv2(0x40000cacc0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:447 +0x14 fp=0x4000addf60 sp=0x4000addf30 pc=0xaaaae1d16204
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).expireEphemeralNodes(0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:217 +0x60 fp=0x4000addfb0 sp=0x4000addf60 pc=0xaaaae27fc8e0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).Serve.func14()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:555 +0x2c fp=0x4000addfd0 sp=0x4000addfb0 pc=0xaaaae28008fc
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000addfd0 sp=0x4000addfd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by github.com/juanfont/headscale/hscontrol.(*Headscale).Serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:555 +0x29c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 9 [select, 45 minutes]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x40005b7f08?, 0x2?, 0x40?, 0x60?, 0x40005b7ef4?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000635d40 sp=0x4000635d20 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.selectgo(0x4000635f08, 0x40005b7ef0, 0x22?, 0x0, 0x0?, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/select.go:327 +0x690 fp=0x4000635e60 sp=0x4000635d40 pc=0xaaaae1d5c040
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).scheduledDERPMapUpdateWorker(0x40000cacc0, 0x4000648120)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/derp.go:143 +0xd0 fp=0x4000635fb0 sp=0x4000635e60 pc=0xaaaae2808980
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).Serve.func13()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:552 +0x2c fp=0x4000635fd0 sp=0x4000635fb0 pc=0xaaaae280095c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000635fd0 sp=0x4000635fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by github.com/juanfont/headscale/hscontrol.(*Headscale).Serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:552 +0x258
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 53 [chan receive]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0xaaaae2d81420?, 0x4000040170?, 0x68?, 0x1d?, 0xaaaae1f0e0e4?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000841d30 sp=0x4000841d10 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv(0x40006af200, 0x4000841f28, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:583 +0x45c fp=0x4000841dc0 sp=0x4000841d30 pc=0xaaaae1d1667c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv1(0x4000903500?, 0xaaaae28f244d?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:442 +0x14 fp=0x4000841df0 sp=0x4000841dc0 pc=0xaaaae1d161e4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).Serve.func6(0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:753 +0xa8 fp=0x4000841f40 sp=0x4000841df0 pc=0xaaaae2800128
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).Serve.func7()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:828 +0x28 fp=0x4000841f60 sp=0x4000841f40 pc=0xaaaae2800058
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/sync/errgroup.(*Group).Go.func1()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x5c fp=0x4000841fd0 sp=0x4000841f60 pc=0xaaaae2332c0c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000841fd0 sp=0x4000841fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by golang.org/x/sync/errgroup.(*Group).Go
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/sync/errgroup/errgroup.go:72 +0xa0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 11 [chan receive]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x4000af02a0?, 0x4?, 0x40?, 0xbd?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000c09ea0 sp=0x4000c09e80 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv(0x40002201e0, 0x4000c09f90, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:583 +0x45c fp=0x4000c09f30 sp=0x4000c09ea0 pc=0xaaaae1d1667c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv2(0x40000cacc0?, 0x40004a1459?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:447 +0x14 fp=0x4000c09f60 sp=0x4000c09f30 pc=0xaaaae1d16204
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).expireExpiredMachines(0x3b?, 0x40004a159e?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:226 +0x60 fp=0x4000c09fb0 sp=0x4000c09f60 pc=0xaaaae27fc970
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).Serve.func15()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:556 +0x2c fp=0x4000c09fd0 sp=0x4000c09fb0 pc=0xaaaae280089c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000c09fd0 sp=0x4000c09fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by github.com/juanfont/headscale/hscontrol.(*Headscale).Serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:556 +0x2e0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 12 [chan receive]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x4000220180?, 0xaaaae1d16204?, 0xe0?, 0xed?, 0xaaaae1d7b95c?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x40011bbe90 sp=0x40011bbe70 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv(0x4000220120, 0x40011bbf90, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:583 +0x45c fp=0x40011bbf20 sp=0x40011bbe90 pc=0xaaaae1d1667c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.chanrecv2(0x40000cacc0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/chan.go:447 +0x14 fp=0x40011bbf50 sp=0x40011bbf20 pc=0xaaaae1d16204
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).failoverSubnetRoutes(0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:233 +0x50 fp=0x40011bbfb0 sp=0x40011bbf50 pc=0xaaaae27fc9f0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).Serve.func16()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:558 +0x2c fp=0x40011bbfd0 sp=0x40011bbfb0 pc=0xaaaae280083c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40011bbfd0 sp=0x40011bbfd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by github.com/juanfont/headscale/hscontrol.(*Headscale).Serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:558 +0x324
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 42 [IO wait, 21 minutes]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000071b90 sp=0x4000071b70 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.netpollblock(0x0?, 0x0?, 0x0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/netpoll.go:527 +0x158 fp=0x4000071bd0 sp=0x4000071b90 pc=0xaaaae1d43ce8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.runtime_pollWait(0xffff7edd1088, 0x72)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/netpoll.go:306 +0xa0 fp=0x4000071c00 sp=0x4000071bd0 pc=0xaaaae1d78c40
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*pollDesc).wait(0x40002e8a00?, 0x0?, 0x0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x4000071c30 sp=0x4000071c00 pc=0xaaaae1db64f8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*pollDesc).waitRead(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*FD).Accept(0x40002e8a00)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_unix.go:614 +0x250 fp=0x4000071ce0 sp=0x4000071c30 pc=0xaaaae1dbafa0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*netFD).accept(0x40002e8a00)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/fd_unix.go:172 +0x28 fp=0x4000071da0 sp=0x4000071ce0 pc=0xaaaae1f9bd58
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*TCPListener).accept(0x4000202e28)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/tcpsock_posix.go:148 +0x28 fp=0x4000071dd0 sp=0x4000071da0 pc=0xaaaae1fb3c18
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*TCPListener).Accept(0x4000202e28)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/tcpsock.go:297 +0x2c fp=0x4000071e10 sp=0x4000071dd0 pc=0xaaaae1fb2d6c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.(*onceCloseListener).Accept(0x40008838c0?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         <autogenerated>:1 +0x30 fp=0x4000071e30 sp=0x4000071e10 pc=0xaaaae20b2be0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.(*Server).Serve(0x400012e0f0, {0xaaaae2d802a0, 0x4000202e28})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:3059 +0x304 fp=0x4000071f60 sp=0x4000071e30 pc=0xaaaae208e4a4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.(*Server).ListenAndServe(0x400012e0f0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:2988 +0x84 fp=0x4000071f90 sp=0x4000071f60 pc=0xaaaae208e164
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).getTLSSettings.func1()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:873 +0x20 fp=0x4000071fd0 sp=0x4000071f90 pc=0xaaaae28012d0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000071fd0 sp=0x4000071fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by github.com/juanfont/headscale/hscontrol.(*Headscale).getTLSSettings
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/app.go:872 +0x340
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 54 [select]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x4000ae13d8?, 0x5?, 0x0?, 0x0?, 0x4000ae0f9e?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000ae0df0 sp=0x4000ae0dd0 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.selectgo(0x4000ae13d8, 0x4000ae0f94, 0xaaaae36c3f00?, 0x0, 0x0?, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/select.go:327 +0x690 fp=0x4000ae0f10 sp=0x4000ae0df0 pc=0xaaaae1d5c040
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*serverConn).serve(0x4000261200)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:938 +0x718 fp=0x4000ae1430 sp=0x4000ae0f10 pc=0xaaaae2122f38
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*Server).ServeConn(0x40006ff130, {0xaaaae2d86af0?, 0x400012e4b0}, 0x4000ae1840)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:531 +0x9fc fp=0x4000ae1700 sp=0x4000ae1430 pc=0xaaaae212165c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).NoiseUpgradeHandler(0x40000cacc0, {0xaaaae2d804b0, 0x400025ea80}, 0x40000cd500)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/noise.go:114 +0x4f8 fp=0x4000ae1920 sp=0x4000ae1700 pc=0xaaaae2816408
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).NoiseUpgradeHandler-fm({0xaaaae2d804b0?, 0x400025ea80?}, 0x4000714780?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         <autogenerated>:1 +0x40 fp=0x4000ae1950 sp=0x4000ae1920 pc=0xaaaae283b1b0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.HandlerFunc.ServeHTTP(0x40000cd400?, {0xaaaae2d804b0?, 0x400025ea80?}, 0xaaaae1db69d8?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:2122 +0x38 fp=0x4000ae1980 sp=0x4000ae1950 pc=0xaaaae208b338
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/gorilla/mux.(*Router).ServeHTTP(0x4000ac4180, {0xaaaae2d804b0, 0x400025ea80}, 0x40000cd100)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/gorilla/mux/mux.go:210 +0x19c fp=0x4000ae1ab0 sp=0x4000ae1980 pc=0xaaaae228d33c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.serverHandler.ServeHTTP({0x400056bda0?}, {0xaaaae2d804b0, 0x400025ea80}, 0x40000cd100)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:2936 +0x2d8 fp=0x4000ae1b60 sp=0x4000ae1ab0 pc=0xaaaae208dfd8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.(*conn).serve(0x40004125a0, {0xaaaae2d81490, 0x400064a2a0})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:1995 +0x560 fp=0x4000ae1fa0 sp=0x4000ae1b60 pc=0xaaaae208a120
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.(*Server).Serve.func3()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:3089 +0x30 fp=0x4000ae1fd0 sp=0x4000ae1fa0 pc=0xaaaae208e800
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000ae1fd0 sp=0x4000ae1fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by net/http.(*Server).Serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:3089 +0x520
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 49 [select, 45 minutes, locked to thread]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x40004f57a0?, 0x2?, 0x78?, 0x56?, 0x40004f579c?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x40004f5620 sp=0x40004f5600 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.selectgo(0x40004f57a0, 0x40004f5798, 0x0?, 0x0, 0x1?, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/select.go:327 +0x690 fp=0x40004f5740 sp=0x40004f5620 pc=0xaaaae1d5c040
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.ensureSigM.func1()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/signal_unix.go:1000 +0x1bc fp=0x40004f57d0 sp=0x40004f5740 pc=0xaaaae1d7536c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40004f57d0 sp=0x40004f57d0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by runtime.ensureSigM
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/signal_unix.go:983 +0xf4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 66 [syscall]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.notetsleepg(0xffffffffffffffff?, 0x40004f5f20?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/lock_futex.go:236 +0x34 fp=0x40004f5f90 sp=0x40004f5f50 pc=0xaaaae1d1ba24
Jul 06 17:28:24 c5b9776eab18 headscale[26]: os/signal.signal_recv()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/sigqueue.go:152 +0x30 fp=0x40004f5fb0 sp=0x40004f5f90 pc=0xaaaae1d7af90
Jul 06 17:28:24 c5b9776eab18 headscale[26]: os/signal.loop()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x1c fp=0x40004f5fd0 sp=0x40004f5fb0 pc=0xaaaae229858c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x40004f5fd0 sp=0x40004f5fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by os/signal.Notify.func1.1
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/os/signal/signal.go:151 +0x2c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 71 [IO wait]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x0?, 0x3b9?, 0xc8?, 0x25?, 0xaaaae1d25b74?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000632540 sp=0x4000632520 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.netpollblock(0x0?, 0xffffffff?, 0xff?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/netpoll.go:527 +0x158 fp=0x4000632580 sp=0x4000632540 pc=0xaaaae1d43ce8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.runtime_pollWait(0xffff7edd0f98, 0x72)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/netpoll.go:306 +0xa0 fp=0x40006325b0 sp=0x4000632580 pc=0xaaaae1d78c40
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*pollDesc).wait(0x40002e9600?, 0x40005c8000?, 0x0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x40006325e0 sp=0x40006325b0 pc=0xaaaae1db64f8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*pollDesc).waitRead(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*FD).Read(0x40002e9600, {0x40005c8000, 0x900, 0x900})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x200 fp=0x4000632680 sp=0x40006325e0 pc=0xaaaae1db7860
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*netFD).Read(0x40002e9600, {0x40005c8000?, 0x4000632838?, 0xaaaae1ff1f1c?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/fd_posix.go:55 +0x28 fp=0x40006326d0 sp=0x4000632680 pc=0xaaaae1f9a0b8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*conn).Read(0x4000010990, {0x40005c8000?, 0x4000632758?, 0xaaaae1d1cf20?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/net.go:183 +0x34 fp=0x4000632720 sp=0x40006326d0 pc=0xaaaae1faaf54
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*TCPConn).Read(0x4000632798?, {0x40005c8000?, 0x4000bfddd0?, 0x18?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         <autogenerated>:1 +0x2c fp=0x4000632750 sp=0x4000632720 pc=0xaaaae1fbdbec
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*atLeastReader).Read(0x4000bfddd0, {0x40005c8000?, 0x4000bfddd0?, 0x0?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:788 +0x40 fp=0x40006327a0 sp=0x4000632750 pc=0xaaaae1ff3700
Jul 06 17:28:24 c5b9776eab18 headscale[26]: bytes.(*Buffer).ReadFrom(0x4000416990, {0xaaaae2d73ec0, 0x4000bfddd0})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/bytes/buffer.go:202 +0x90 fp=0x4000632800 sp=0x40006327a0 pc=0xaaaae1df2980
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).readFromUntil(0x4000416700, {0xaaaae2d76ea0?, 0x4000010990}, 0x8c7?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:810 +0xd4 fp=0x4000632840 sp=0x4000632800 pc=0xaaaae1ff38f4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).readRecordOrCCS(0x4000416700, 0x0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:617 +0xd8 fp=0x4000632bb0 sp=0x4000632840 pc=0xaaaae1ff1748
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).readRecord(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:583
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).Read(0x4000416700, {0x400012e560, 0x3, 0xaaaae27a2788?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:1316 +0x178 fp=0x4000632c20 sp=0x4000632bb0 pc=0xaaaae1ff6698
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/net/netutil.(*drainBufConn).Read(0x4000632c78?, {0x400012e560?, 0x400012e520?, 0x4000418000?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/net/netutil/netutil.go:87 +0x94 fp=0x4000632c50 sp=0x4000632c20 pc=0xaaaae278d854
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlhttp.(*corkConn).Read(0x4000632ca8?, {0x400012e560?, 0x0?, 0x5?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlhttp/server.go:193 +0x5c fp=0x4000632c80 sp=0x4000632c50 pc=0xaaaae27c923c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlbase.(*Conn).readNLocked(0x400012e4b0, 0x3)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlbase/conn.go:115 +0xe0 fp=0x4000632ce0 sp=0x4000632c80 pc=0xaaaae27a2070
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlbase.(*Conn).decryptOneLocked(0x400012e4b0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlbase/conn.go:223 +0x1b0 fp=0x4000632d20 sp=0x4000632ce0 pc=0xaaaae27a2730
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlbase.(*Conn).Read(0x400012e4b0, {0x400025ec80, 0x9, 0x0?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlbase/conn.go:253 +0x13c fp=0x4000632d90 sp=0x4000632d20 pc=0xaaaae27a295c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: io.ReadAtLeast({0xffff7e940318, 0x400012e4b0}, {0x400025ec80, 0x9, 0x9}, 0x9)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/io/io.go:332 +0xa0 fp=0x4000632de0 sp=0x4000632d90 pc=0xaaaae1db0f50
Jul 06 17:28:24 c5b9776eab18 headscale[26]: io.ReadFull(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/io/io.go:351
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.readFrameHeader({0x400025ec80?, 0x9?, 0x400056b050?}, {0xffff7e940318?, 0x400012e4b0?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/frame.go:237 +0x58 fp=0x4000632e30 sp=0x4000632de0 pc=0xaaaae21179b8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*Framer).ReadFrame(0x400025ec40)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/frame.go:498 +0x84 fp=0x4000632ee0 sp=0x4000632e30 pc=0xaaaae21181b4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*serverConn).readFrames(0x4000261200)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:818 +0x88 fp=0x4000632fb0 sp=0x4000632ee0 pc=0xaaaae21222a8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*serverConn).serve.func7()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:930 +0x28 fp=0x4000632fd0 sp=0x4000632fb0 pc=0xaaaae21235e8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000632fd0 sp=0x4000632fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by golang.org/x/net/http2.(*serverConn).serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:930 +0x4a4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 324 [select]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x400085d3d8?, 0x5?, 0x0?, 0x0?, 0x400085cf9e?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x400085cdf0 sp=0x400085cdd0 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.selectgo(0x400085d3d8, 0x400085cf94, 0xaaaae36c3f00?, 0x0, 0x0?, 0x1)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/select.go:327 +0x690 fp=0x400085cf10 sp=0x400085cdf0 pc=0xaaaae1d5c040
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*serverConn).serve(0x40008d1b00)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:938 +0x718 fp=0x400085d430 sp=0x400085cf10 pc=0xaaaae2122f38
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*Server).ServeConn(0x4000ced810, {0xaaaae2d86af0?, 0x400012fc20}, 0x400085d840)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:531 +0x9fc fp=0x400085d700 sp=0x400085d430 pc=0xaaaae212165c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).NoiseUpgradeHandler(0x40000cacc0, {0xaaaae2d804b0, 0x400025e540}, 0x40008bf600)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/hscontrol/noise.go:114 +0x4f8 fp=0x400085d920 sp=0x400085d700 pc=0xaaaae2816408
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/juanfont/headscale/hscontrol.(*Headscale).NoiseUpgradeHandler-fm({0xaaaae2d804b0?, 0x400025e540?}, 0x4000714780?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         <autogenerated>:1 +0x40 fp=0x400085d950 sp=0x400085d920 pc=0xaaaae283b1b0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.HandlerFunc.ServeHTTP(0x40008bf500?, {0xaaaae2d804b0?, 0x400025e540?}, 0xaaaae1db69d8?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:2122 +0x38 fp=0x400085d980 sp=0x400085d950 pc=0xaaaae208b338
Jul 06 17:28:24 c5b9776eab18 headscale[26]: github.com/gorilla/mux.(*Router).ServeHTTP(0x4000ac4180, {0xaaaae2d804b0, 0x400025e540}, 0x40008bf200)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/github.com/gorilla/mux/mux.go:210 +0x19c fp=0x400085dab0 sp=0x400085d980 pc=0xaaaae228d33c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.serverHandler.ServeHTTP({0x4000ce80f0?}, {0xaaaae2d804b0, 0x400025e540}, 0x40008bf200)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:2936 +0x2d8 fp=0x400085db60 sp=0x400085dab0 pc=0xaaaae208dfd8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.(*conn).serve(0x4000cccbd0, {0xaaaae2d81490, 0x400064a2a0})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:1995 +0x560 fp=0x400085dfa0 sp=0x400085db60 pc=0xaaaae208a120
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net/http.(*Server).Serve.func3()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:3089 +0x30 fp=0x400085dfd0 sp=0x400085dfa0 pc=0xaaaae208e800
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x400085dfd0 sp=0x400085dfd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by net/http.(*Server).Serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/http/server.go:3089 +0x520
Jul 06 17:28:24 c5b9776eab18 headscale[26]: goroutine 328 [IO wait, 5 minutes]:
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.gopark(0x0?, 0x1ed?, 0xc8?, 0x85?, 0xaaaae1d25b74?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/proc.go:381 +0xe4 fp=0x4000978540 sp=0x4000978520 pc=0xaaaae1d4b144
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.netpollblock(0x0?, 0xffffffff?, 0xff?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/netpoll.go:527 +0x158 fp=0x4000978580 sp=0x4000978540 pc=0xaaaae1d43ce8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.runtime_pollWait(0xffff7edd1718, 0x72)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/netpoll.go:306 +0xa0 fp=0x40009785b0 sp=0x4000978580 pc=0xaaaae1d78c40
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*pollDesc).wait(0x4000898180?, 0x4000c7d000?, 0x0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x28 fp=0x40009785e0 sp=0x40009785b0 pc=0xaaaae1db64f8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*pollDesc).waitRead(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
Jul 06 17:28:24 c5b9776eab18 headscale[26]: internal/poll.(*FD).Read(0x4000898180, {0x4000c7d000, 0x1000, 0x1000})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/internal/poll/fd_unix.go:167 +0x200 fp=0x4000978680 sp=0x40009785e0 pc=0xaaaae1db7860
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*netFD).Read(0x4000898180, {0x4000c7d000?, 0x4000978838?, 0xaaaae1ff1f1c?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/fd_posix.go:55 +0x28 fp=0x40009786d0 sp=0x4000978680 pc=0xaaaae1f9a0b8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*conn).Read(0x4000010058, {0x4000c7d000?, 0x40009787b8?, 0xaaaae1df250c?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/net/net.go:183 +0x34 fp=0x4000978720 sp=0x40009786d0 pc=0xaaaae1faaf54
Jul 06 17:28:24 c5b9776eab18 headscale[26]: net.(*TCPConn).Read(0x4000978798?, {0x4000c7d000?, 0x40006ed2d8?, 0x18?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         <autogenerated>:1 +0x2c fp=0x4000978750 sp=0x4000978720 pc=0xaaaae1fbdbec
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*atLeastReader).Read(0x40006ed2d8, {0x4000c7d000?, 0x40006ed2d8?, 0x0?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:788 +0x40 fp=0x40009787a0 sp=0x4000978750 pc=0xaaaae1ff3700
Jul 06 17:28:24 c5b9776eab18 headscale[26]: bytes.(*Buffer).ReadFrom(0x4000c25410, {0xaaaae2d73ec0, 0x40006ed2d8})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/bytes/buffer.go:202 +0x90 fp=0x4000978800 sp=0x40009787a0 pc=0xaaaae1df2980
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).readFromUntil(0x4000c25180, {0xaaaae2d76ea0?, 0x4000010058}, 0x1000?)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:810 +0xd4 fp=0x4000978840 sp=0x4000978800 pc=0xaaaae1ff38f4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).readRecordOrCCS(0x4000c25180, 0x0)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:617 +0xd8 fp=0x4000978bb0 sp=0x4000978840 pc=0xaaaae1ff1748
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).readRecord(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:583
Jul 06 17:28:24 c5b9776eab18 headscale[26]: crypto/tls.(*Conn).Read(0x4000c25180, {0x400012fcd0, 0x3, 0xaaaae27a2788?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/crypto/tls/conn.go:1316 +0x178 fp=0x4000978c20 sp=0x4000978bb0 pc=0xaaaae1ff6698
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/net/netutil.(*drainBufConn).Read(0x4000978c78?, {0x400012fcd0?, 0x400012fc90?, 0x40000ff000?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/net/netutil/netutil.go:87 +0x94 fp=0x4000978c50 sp=0x4000978c20 pc=0xaaaae278d854
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlhttp.(*corkConn).Read(0x4000978ca8?, {0x400012fcd0?, 0x0?, 0x5?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlhttp/server.go:193 +0x5c fp=0x4000978c80 sp=0x4000978c50 pc=0xaaaae27c923c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlbase.(*Conn).readNLocked(0x400012fc20, 0x3)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlbase/conn.go:115 +0xe0 fp=0x4000978ce0 sp=0x4000978c80 pc=0xaaaae27a2070
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlbase.(*Conn).decryptOneLocked(0x400012fc20)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlbase/conn.go:223 +0x1b0 fp=0x4000978d20 sp=0x4000978ce0 pc=0xaaaae27a2730
Jul 06 17:28:24 c5b9776eab18 headscale[26]: tailscale.com/control/controlbase.(*Conn).Read(0x400012fc20, {0x400025e740, 0x9, 0x0?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/tailscale.com/control/controlbase/conn.go:253 +0x13c fp=0x4000978d90 sp=0x4000978d20 pc=0xaaaae27a295c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: io.ReadAtLeast({0xffff7e940318, 0x400012fc20}, {0x400025e740, 0x9, 0x9}, 0x9)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/io/io.go:332 +0xa0 fp=0x4000978de0 sp=0x4000978d90 pc=0xaaaae1db0f50
Jul 06 17:28:24 c5b9776eab18 headscale[26]: io.ReadFull(...)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/io/io.go:351
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.readFrameHeader({0x400025e740?, 0x9?, 0x400056bdd0?}, {0xffff7e940318?, 0x400012fc20?})
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/frame.go:237 +0x58 fp=0x4000978e30 sp=0x4000978de0 pc=0xaaaae21179b8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*Framer).ReadFrame(0x400025e700)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/frame.go:498 +0x84 fp=0x4000978ee0 sp=0x4000978e30 pc=0xaaaae21181b4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*serverConn).readFrames(0x40008d1b00)
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:818 +0x88 fp=0x4000978fb0 sp=0x4000978ee0 pc=0xaaaae21222a8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: golang.org/x/net/http2.(*serverConn).serve.func7()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:930 +0x28 fp=0x4000978fd0 sp=0x4000978fb0 pc=0xaaaae21235e8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: runtime.goexit()
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /usr/lib/golang/src/runtime/asm_arm64.s:1172 +0x4 fp=0x4000978fd0 sp=0x4000978fd0 pc=0xaaaae1d7ec44
Jul 06 17:28:24 c5b9776eab18 headscale[26]: created by golang.org/x/net/http2.(*serverConn).serve
Jul 06 17:28:24 c5b9776eab18 headscale[26]:         /builddir/build/BUILD/headscale-0.22.3/vendor/golang.org/x/net/http2/server.go:930 +0x4a4
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r0      0xfffffffffffffffc
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r1      0x80
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r2      0x0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r3      0xffff81659748
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r4      0x0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r5      0x0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r6      0xb83c3
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r7      0x2cddc5c
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r8      0x62
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r9      0x3d633814959ed
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r10     0xffffffffffffff
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r11     0x53a596422cfe
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r12     0xaaaae36c395e
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r13     0xffffe1d1b760
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r14     0x7
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r15     0x88
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r16     0xffff80e5a018
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r17     0xffff81658fc0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r18     0x6474e552
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r19     0xaaaae1d7c5e0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r20     0xffff81659730
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r21     0x4000060400
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r22     0x1
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r23     0x80ea40
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r24     0xffffe1d1b4c7
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r25     0xffff80e4b000
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r26     0xaaaae2d703c8
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r27     0xaaaae35a4000
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r28     0x40000029c0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: r29     0xffff81659708
Jul 06 17:28:24 c5b9776eab18 headscale[26]: lr      0xaaaae1d44c30
Jul 06 17:28:24 c5b9776eab18 headscale[26]: sp      0xffff81659710
Jul 06 17:28:24 c5b9776eab18 headscale[26]: pc      0xaaaae1d80110
Jul 06 17:28:24 c5b9776eab18 headscale[26]: fault   0x0
Jul 06 17:28:24 c5b9776eab18 headscale[26]: -----
[root@c5b9776eab18 /]#
[root@c5b9776eab18 /]# Jul 06 17:28:29 c5b9776eab18 systemd[1]: headscale.service: Main process exited, code=dumped, status=6/ABRT
Jul 06 17:28:29 c5b9776eab18 systemd[1]: headscale.service: Failed with result 'timeout'.
Jul 06 17:28:29 c5b9776eab18 systemd[1]: Stopped headscale.service - headscale coordination server for Tailscale.
Jul 06 17:28:29 c5b9776eab18 systemd[1]: headscale.service: Consumed 1.862s CPU time.
jonathanspw commented 1 year ago

I've not run into this running it in full VMs. Does it happen only with the RPM or also when installed manually in the container?

dustymabe commented 1 year ago

Haven't had a chance to run it in a VM. Will post here if I get a chance to try it this weekend.

ajayramaswamy commented 12 months ago

This patch fixes it for me https://github.com/juanfont/headscale/pull/1480

codyro commented 12 months ago

This patch is against a different version of app.go than the one in the vendored source/tagged v0.22.3 (latest), and the differences around this change are somewhat significant.

I've gone ahead and made a modified patch (adding a return after the close() function; (here) to see if this resolves the issue for you (I'm doubtful, but it's worth a shot).

Could you try this build, and if it resolves the issue, I'll merge it into this repo. Otherwise, we'll wait until the next tagged release (unless @jonathanspw has a better idea).

https://copr.fedorainfracloud.org/coprs/codyrobertson/headscale/build/6252827/

jonathanspw commented 9 months ago

@dustymabe any report on if @codyro 's patch above fixed your issue?