search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.69k stars 1.77k forks source link

Agents connected using websocket upgrades don't reconnect successfully after interruption #39184

Open webvictim opened 8 months ago

webvictim commented 8 months ago

Expected behaviour: Agents connected to a Teleport proxy using websocket upgrades (rather than Teleport's standard connection mechanism, or using ALPN Upgrade through layer 7 LBs) should seamlessly reconnect when the tunnel between the agent and proxy is torn down due to interruption or network topology changes.

Current behaviour: Agents seem to get stuck attempting to connect using the "regular" tunnel method which results in a ssh: overflow reading connection string error.

I am not 100% sure whether this affects all ALPN upgrades or just the websocket versions.

Related: https://github.com/gravitational/teleport/pull/36343

Logs:

# active connection working fine
Mar 10 13:58:41 server teleport[3501761]: 2024-03-10T13:58:41-03:00 DEBU [PROXY:AGE] Ping -> 1.2.3.4:443. leaseID:4 target:teleport.example.com:443 localCluster: targetCluster:teleport.webvict.im reversetunnel/agent.go:561
Mar 10 13:58:41 server teleport[3501761]: 2024-03-10T13:58:41-03:00 DEBU [PROXY:AGE] Ping -> 1.2.3.4:443. leaseID:4 target:teleport.example.com:443 localCluster: targetCluster:teleport.webvict.im reversetunnel/agent.go:561
Mar 10 13:58:41 server teleport[3501761]: 2024-03-10T13:58:41-03:00 DEBU [PROXY:AGE] Received discovery request: [d1236170-0a87-4727-a151-8b7bc2441087] leaseID:4 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:652
Mar 10 14:00:41 server teleport[3501761]: 2024-03-10T14:00:41-03:00 DEBU [PROXY:AGE] Received discovery request: [d1236170-0a87-4727-a151-8b7bc2441087] leaseID:4 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:652
Mar 10 14:00:41 server teleport[3501761]: 2024-03-10T14:00:41-03:00 DEBU [PROXY:AGE] Received discovery request: [d1236170-0a87-4727-a151-8b7bc2441087] leaseID:4 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:652
Mar 10 14:00:47 server teleport[3501761]: 2024-03-10T14:00:47-03:00 DEBU             "teleport-system" group not found, not deleting users srv/usermgmt.go:402
Mar 10 14:01:58 server teleport[3501761]: 2024-03-10T14:01:58-03:00 DEBU [PROXY:AGE] Received discovery request: [d1236170-0a87-4727-a151-8b7bc2441087] leaseID:4 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:652
Mar 10 14:01:58 server teleport[3501761]: 2024-03-10T14:01:58-03:00 DEBU [PROXY:AGE] Received discovery request: [d1236170-0a87-4727-a151-8b7bc2441087] leaseID:4 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:652
Mar 10 14:02:41 server teleport[3501761]: 2024-03-10T14:02:41-03:00 DEBU [PROXY:AGE] Received discovery request: [d1236170-0a87-4727-a151-8b7bc2441087] leaseID:4 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:652
Mar 10 14:02:41 server teleport[3501761]: 2024-03-10T14:02:41-03:00 DEBU [PROXY:AGE] Received discovery request: [d1236170-0a87-4727-a151-8b7bc2441087] leaseID:4 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:652

# connection interrupted due to upstream connectivity drop
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 INFO [PROXY:AGE] Connection closed, returning leaseID:4 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:642
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [PROXY:AGE] Failed to handle drainable channels. error:[
Mar 10 14:03:27 server teleport[3501761]: ERROR REPORT:
Mar 10 14:03:27 server teleport[3501761]: Original Error: *trace.ConnectionProblemError heartbeat: connection closed
Mar 10 14:03:27 server teleport[3501761]: Stack Trace:
Mar 10 14:03:27 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:548 github.com/gravitational/teleport/lib/reversetunnel.(*agent).handleDrainChannels
Mar 10 14:03:27 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:338 github.com/gravitational/teleport/lib/reversetunnel.(*agent).Start.func3
Mar 10 14:03:27 server teleport[3501761]:         runtime/asm_amd64.s:1650 runtime.goexit
Mar 10 14:03:27 server teleport[3501761]: User Message: heartbeat: connection closed] leaseID:4 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:339
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [PROXY:AGE] Failed to handle global requests. error:[
Mar 10 14:03:27 server teleport[3501761]: ERROR REPORT:
Mar 10 14:03:27 server teleport[3501761]: Original Error: *errors.errorString global request channel is closing
Mar 10 14:03:27 server teleport[3501761]: Stack Trace:
Mar 10 14:03:27 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:456 github.com/gravitational/teleport/lib/reversetunnel.(*agent).handleGlobalRequests
Mar 10 14:03:27 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:327 github.com/gravitational/teleport/lib/reversetunnel.(*agent).Start.func2
Mar 10 14:03:27 server teleport[3501761]:         runtime/asm_amd64.s:1650 runtime.goexit
Mar 10 14:03:27 server teleport[3501761]: User Message: global request channel is closing] leaseID:4 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:328
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [PROXY:AGE] Changing state connected -> closed. leaseID:4 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:287
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 WARN [PROXY:AGE] Failed to close discovery channel: EOF leaseID:4 target:teleport.example.com:443 localCluster: targetCluster:teleport.webvict.im reversetunnel/agent.go:631
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423

# connection attempts to reestablish
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [PROXY:AGE] Runtime config: restrict_connection_count: false connection_count: 1 localCluster: targetCluster:teleport.webvict.im reversetunnel/agentpool.go:340
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU             Attempting GET teleport.example.com:443/webapi/find webclient/webclient.go:129
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [PROXY:AGE] Starting agent {teleport.example.com:443 tcp } leaseID:5 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:299
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [PROXY:AGE] Changing state initial -> connecting. leaseID:5 target:teleport.example.com:443 localCluster: targetCluster:teleport.webvict.im reversetunnel/agent.go:287
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:197
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [PROXY:AGE] Changing state connecting -> closed. leaseID:5 target:teleport.example.com:443 localCluster: targetCluster:teleport.webvict.im reversetunnel/agent.go:287
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [PROXY:AGE] Failed to connect agent. error:[
Mar 10 14:03:27 server teleport[3501761]: ERROR REPORT:
Mar 10 14:03:27 server teleport[3501761]: Original Error: *fmt.wrapError ssh: handshake failed: ssh: overflow reading version string
Mar 10 14:03:27 server teleport[3501761]: Stack Trace:
Mar 10 14:03:27 server teleport[3501761]:         github.com/gravitational/teleport/api@v0.0.0/observability/tracing/ssh/ssh.go:158 github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConn
Mar 10 14:03:27 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agent_dialer.go:97 github.com/gravitational/teleport/lib/reversetunnel.(*agentDialer).DialContext
Mar 10 14:03:27 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:364 github.com/gravitational/teleport/lib/reversetunnel.(*agent).connect
Mar 10 14:03:27 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:319 github.com/gravitational/teleport/lib/reversetunnel.(*agent).Start
Mar 10 14:03:27 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:321 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).connectAgent
Mar 10 14:03:27 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:270 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).run
Mar 10 14:03:27 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:257 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start.func1
Mar 10 14:03:27 server teleport[3501761]:         runtime/asm_amd64.s:1650 runtime.goexit
Mar 10 14:03:27 server teleport[3501761]: User Message: ssh: handshake failed: ssh: overflow reading version string] localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:279
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [PROXY:AGE] Runtime config: restrict_connection_count: false connection_count: 1 localCluster: targetCluster:teleport.webvict.im reversetunnel/agentpool.go:340
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [PROXY:AGE] Starting agent {teleport.example.com:443 tcp } leaseID:6 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:299
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [PROXY:AGE] Changing state initial -> connecting. leaseID:6 target:teleport.example.com:443 localCluster: targetCluster:teleport.webvict.im reversetunnel/agent.go:287
Mar 10 14:03:27 server teleport[3501761]: 2024-03-10T14:03:27-03:00 DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:197
Mar 10 14:03:28 server teleport[3501761]: 2024-03-10T14:03:28-03:00 DEBU [PROXY:AGE] Changing state connecting -> closed. leaseID:6 target:teleport.example.com:443 localCluster: targetCluster:teleport.webvict.im reversetunnel/agent.go:287
Mar 10 14:03:28 server teleport[3501761]: 2024-03-10T14:03:28-03:00 DEBU [PROXY:AGE] Failed to connect agent. error:[
Mar 10 14:03:28 server teleport[3501761]: ERROR REPORT:
Mar 10 14:03:28 server teleport[3501761]: Original Error: *fmt.wrapError ssh: handshake failed: ssh: overflow reading version string
Mar 10 14:03:28 server teleport[3501761]: Stack Trace:
Mar 10 14:03:28 server teleport[3501761]:         github.com/gravitational/teleport/api@v0.0.0/observability/tracing/ssh/ssh.go:158 github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConn
Mar 10 14:03:28 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agent_dialer.go:97 github.com/gravitational/teleport/lib/reversetunnel.(*agentDialer).DialContext
Mar 10 14:03:28 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:364 github.com/gravitational/teleport/lib/reversetunnel.(*agent).connect
Mar 10 14:03:28 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:319 github.com/gravitational/teleport/lib/reversetunnel.(*agent).Start
Mar 10 14:03:28 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:321 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).connectAgent
Mar 10 14:03:28 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:270 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).run
Mar 10 14:03:28 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:257 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start.func1
Mar 10 14:03:28 server teleport[3501761]:         runtime/asm_amd64.s:1650 runtime.goexit
Mar 10 14:03:28 server teleport[3501761]: User Message: ssh: handshake failed: ssh: overflow reading version string] localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:279
Mar 10 14:03:28 server teleport[3501761]: 2024-03-10T14:03:28-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 10 14:03:28 server teleport[3501761]: 2024-03-10T14:03:28-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 10 14:03:28 server teleport[3501761]: 2024-03-10T14:03:28-03:00 DEBU [PROXY:AGE] Runtime config: restrict_connection_count: false connection_count: 1 localCluster: targetCluster:teleport.webvict.im reversetunnel/agentpool.go:340
Mar 10 14:03:28 server teleport[3501761]: 2024-03-10T14:03:28-03:00 DEBU [PROXY:AGE] Starting agent {teleport.example.com:443 tcp } leaseID:7 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:299
Mar 10 14:03:28 server teleport[3501761]: 2024-03-10T14:03:28-03:00 DEBU [PROXY:AGE] Changing state initial -> connecting. leaseID:7 target:teleport.example.com:443 localCluster: targetCluster:teleport.webvict.im reversetunnel/agent.go:287
Mar 10 14:03:28 server teleport[3501761]: 2024-03-10T14:03:28-03:00 DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:197
Mar 10 14:03:28 server teleport[3501761]: 2024-03-10T14:03:28-03:00 DEBU [PROXY:AGE] Changing state connecting -> closed. leaseID:7 target:teleport.example.com:443 localCluster: targetCluster:teleport.webvict.im reversetunnel/agent.go:287
Mar 10 14:03:28 server teleport[3501761]: 2024-03-10T14:03:28-03:00 DEBU [PROXY:AGE] Failed to connect agent. error:[
Mar 10 14:03:28 server teleport[3501761]: ERROR REPORT:
Mar 10 14:03:28 server teleport[3501761]: Original Error: *fmt.wrapError ssh: handshake failed: ssh: overflow reading version string
Mar 10 14:03:28 server teleport[3501761]: Stack Trace:
Mar 10 14:03:28 server teleport[3501761]:         github.com/gravitational/teleport/api@v0.0.0/observability/tracing/ssh/ssh.go:158 github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConn
Mar 10 14:03:28 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agent_dialer.go:97 github.com/gravitational/teleport/lib/reversetunnel.(*agentDialer).DialContext
Mar 10 14:03:28 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:364 github.com/gravitational/teleport/lib/reversetunnel.(*agent).connect
Mar 10 14:03:28 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:319 github.com/gravitational/teleport/lib/reversetunnel.(*agent).Start
Mar 10 14:03:28 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:321 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).connectAgent
Mar 10 14:03:28 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:270 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).run
Mar 10 14:03:28 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:257 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start.func1
Mar 10 14:03:28 server teleport[3501761]:         runtime/asm_amd64.s:1650 runtime.goexit
Mar 10 14:03:28 server teleport[3501761]: User Message: ssh: handshake failed: ssh: overflow reading version string] localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:279
Mar 10 14:03:28 server teleport[3501761]: 2024-03-10T14:03:28-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 10 14:03:28 server teleport[3501761]: 2024-03-10T14:03:28-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 10 14:03:30 server teleport[3501761]: 2024-03-10T14:03:30-03:00 DEBU [PROXY:AGE] Runtime config: restrict_connection_count: false connection_count: 1 localCluster: targetCluster:teleport.webvict.im reversetunnel/agentpool.go:340
Mar 10 14:03:30 server teleport[3501761]: 2024-03-10T14:03:30-03:00 DEBU [PROXY:AGE] Starting agent {teleport.example.com:443 tcp } leaseID:8 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:299
Mar 10 14:03:30 server teleport[3501761]: 2024-03-10T14:03:30-03:00 DEBU [PROXY:AGE] Changing state initial -> connecting. leaseID:8 target:teleport.example.com:443 localCluster: targetCluster:teleport.webvict.im reversetunnel/agent.go:287
Mar 10 14:03:30 server teleport[3501761]: 2024-03-10T14:03:30-03:00 DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:197
Mar 10 14:03:30 server teleport[3501761]: 2024-03-10T14:03:30-03:00 DEBU [PROXY:AGE] Changing state connecting -> closed. leaseID:8 target:teleport.example.com:443 localCluster: targetCluster:teleport.webvict.im reversetunnel/agent.go:287
Mar 10 14:03:30 server teleport[3501761]: 2024-03-10T14:03:30-03:00 DEBU [PROXY:AGE] Failed to connect agent. error:[
Mar 10 14:03:30 server teleport[3501761]: ERROR REPORT:
Mar 10 14:03:30 server teleport[3501761]: Original Error: *fmt.wrapError ssh: handshake failed: ssh: overflow reading version string
Mar 10 14:03:30 server teleport[3501761]: Stack Trace:
Mar 10 14:03:30 server teleport[3501761]:         github.com/gravitational/teleport/api@v0.0.0/observability/tracing/ssh/ssh.go:158 github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConn
Mar 10 14:03:30 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agent_dialer.go:97 github.com/gravitational/teleport/lib/reversetunnel.(*agentDialer).DialContext
Mar 10 14:03:30 server teleport[3501761]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:364 github.com/gravitational/teleport/lib/reversetunnel.(*agent).connect
# logs continue like this until agent is restarted

Bug details:

webvictim commented 8 months ago

Frustratingly, the diagnostic endpoints don't report that the tunnel isn't connected correctly when in this state either:

[root@hades ~]# curl http://127.0.0.1:3434/readyz
{"status":"ok"}
[root@hades ~]# curl http://127.0.0.1:3434/healthz
{"status":"ok"}
webvictim commented 8 months ago

Some additional log context confirming that the agent is still just stuck in a loop hours later:

Mar 11 11:32:12 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent_dialer.go:97 github.com/gravitational/teleport/lib/reversetunnel.(*agentDialer).DialContext
Mar 11 11:32:12 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:364 github.com/gravitational/teleport/lib/reversetunnel.(*agent).connect
Mar 11 11:32:12 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:319 github.com/gravitational/teleport/lib/reversetunnel.(*agent).Start
Mar 11 11:32:12 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:321 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).connectAgent
Mar 11 11:32:12 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:270 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).run
Mar 11 11:32:12 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:257 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start.func1
Mar 11 11:32:12 hades teleport[2120659]:         runtime/asm_amd64.s:1650 runtime.goexit
Mar 11 11:32:12 hades teleport[2120659]: User Message: ssh: handshake failed: ssh: overflow reading version string] localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:279
Mar 11 11:32:12 hades teleport[2120659]: 2024-03-11T11:32:12-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 11 11:32:12 hades teleport[2120659]: 2024-03-11T11:32:12-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 11 11:32:17 hades teleport[2120659]: 2024-03-11T11:32:17-03:00 DEBU [PROXY:AGE] Runtime config: restrict_connection_count: false connection_count: 1 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:340
Mar 11 11:32:17 hades teleport[2120659]: 2024-03-11T11:32:17-03:00 DEBU             Attempting GET teleport.example.com:443/webapi/find webclient/webclient.go:129
Mar 11 11:32:17 hades teleport[2120659]: 2024-03-11T11:32:17-03:00 DEBU [PROXY:AGE] Starting agent {teleport.example.com:443 tcp } leaseID:2386 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:299
Mar 11 11:32:17 hades teleport[2120659]: 2024-03-11T11:32:17-03:00 DEBU [PROXY:AGE] Changing state initial -> connecting. leaseID:2386 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:287
Mar 11 11:32:17 hades teleport[2120659]: 2024-03-11T11:32:17-03:00 DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:197
Mar 11 11:32:17 hades teleport[2120659]: 2024-03-11T11:32:17-03:00 DEBU [PROXY:AGE] Changing state connecting -> closed. leaseID:2386 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:287
Mar 11 11:32:17 hades teleport[2120659]: 2024-03-11T11:32:17-03:00 DEBU [PROXY:AGE] Failed to connect agent. error:[
Mar 11 11:32:17 hades teleport[2120659]: ERROR REPORT:
Mar 11 11:32:17 hades teleport[2120659]: Original Error: *fmt.wrapError ssh: handshake failed: ssh: overflow reading version string
Mar 11 11:32:17 hades teleport[2120659]: Stack Trace:
Mar 11 11:32:17 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/observability/tracing/ssh/ssh.go:158 github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConn
Mar 11 11:32:17 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent_dialer.go:97 github.com/gravitational/teleport/lib/reversetunnel.(*agentDialer).DialContext
Mar 11 11:32:17 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:364 github.com/gravitational/teleport/lib/reversetunnel.(*agent).connect
Mar 11 11:32:17 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:319 github.com/gravitational/teleport/lib/reversetunnel.(*agent).Start
Mar 11 11:32:17 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:321 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).connectAgent
Mar 11 11:32:17 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:270 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).run
Mar 11 11:32:17 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:257 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start.func1
Mar 11 11:32:17 hades teleport[2120659]:         runtime/asm_amd64.s:1650 runtime.goexit
Mar 11 11:32:17 hades teleport[2120659]: User Message: ssh: handshake failed: ssh: overflow reading version string] localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:279
Mar 11 11:32:17 hades teleport[2120659]: 2024-03-11T11:32:17-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 11 11:32:17 hades teleport[2120659]: 2024-03-11T11:32:17-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 11 11:32:18 hades teleport[2120659]: 2024-03-11T11:32:18-03:00 DEBU [PROXY:AGE] Runtime config: restrict_connection_count: false connection_count: 1 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:340
Mar 11 11:32:18 hades teleport[2120659]: 2024-03-11T11:32:18-03:00 DEBU [PROXY:AGE] Starting agent {teleport.example.com:443 tcp } leaseID:12144 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:299
Mar 11 11:32:18 hades teleport[2120659]: 2024-03-11T11:32:18-03:00 DEBU [PROXY:AGE] Changing state initial -> connecting. leaseID:12144 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:287
Mar 11 11:32:18 hades teleport[2120659]: 2024-03-11T11:32:18-03:00 DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:197
Mar 11 11:32:18 hades teleport[2120659]: 2024-03-11T11:32:18-03:00 DEBU [PROXY:AGE] Changing state connecting -> closed. leaseID:12144 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:287
Mar 11 11:32:18 hades teleport[2120659]: 2024-03-11T11:32:18-03:00 DEBU [PROXY:AGE] Failed to connect agent. error:[
Mar 11 11:32:18 hades teleport[2120659]: ERROR REPORT:
Mar 11 11:32:18 hades teleport[2120659]: Original Error: *fmt.wrapError ssh: handshake failed: ssh: overflow reading version string
Mar 11 11:32:18 hades teleport[2120659]: Stack Trace:
Mar 11 11:32:18 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/observability/tracing/ssh/ssh.go:158 github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConn
Mar 11 11:32:18 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent_dialer.go:97 github.com/gravitational/teleport/lib/reversetunnel.(*agentDialer).DialContext
Mar 11 11:32:18 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:364 github.com/gravitational/teleport/lib/reversetunnel.(*agent).connect
Mar 11 11:32:18 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:319 github.com/gravitational/teleport/lib/reversetunnel.(*agent).Start
Mar 11 11:32:18 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:321 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).connectAgent
Mar 11 11:32:18 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:270 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).run
Mar 11 11:32:18 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:257 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start.func1
Mar 11 11:32:18 hades teleport[2120659]:         runtime/asm_amd64.s:1650 runtime.goexit
Mar 11 11:32:18 hades teleport[2120659]: User Message: ssh: handshake failed: ssh: overflow reading version string] localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:279
Mar 11 11:32:18 hades teleport[2120659]: 2024-03-11T11:32:18-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 11 11:32:18 hades teleport[2120659]: 2024-03-11T11:32:18-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 11 11:32:25 hades teleport[2120659]: 2024-03-11T11:32:25-03:00 DEBU [PROXY:AGE] Runtime config: restrict_connection_count: false connection_count: 1 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:340
Mar 11 11:32:25 hades teleport[2120659]: 2024-03-11T11:32:25-03:00 DEBU             Attempting GET teleport.example.com:443/webapi/find webclient/webclient.go:129
Mar 11 11:32:25 hades teleport[2120659]: 2024-03-11T11:32:25-03:00 DEBU [PROXY:AGE] Runtime config: restrict_connection_count: false connection_count: 1 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:340
Mar 11 11:32:26 hades teleport[2120659]: 2024-03-11T11:32:26-03:00 DEBU [PROXY:AGE] Starting agent {teleport.example.com:443 tcp } leaseID:2387 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:299
Mar 11 11:32:26 hades teleport[2120659]: 2024-03-11T11:32:26-03:00 DEBU [PROXY:AGE] Changing state initial -> connecting. leaseID:2387 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:287
Mar 11 11:32:26 hades teleport[2120659]: 2024-03-11T11:32:26-03:00 DEBU [PROXY:AGE] Starting agent {teleport.example.com:443 tcp } leaseID:12145 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:299
Mar 11 11:32:26 hades teleport[2120659]: 2024-03-11T11:32:26-03:00 DEBU [PROXY:AGE] Changing state initial -> connecting. leaseID:12145 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:287
Mar 11 11:32:26 hades teleport[2120659]: 2024-03-11T11:32:26-03:00 DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:197
Mar 11 11:32:26 hades teleport[2120659]: 2024-03-11T11:32:26-03:00 DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:197
Mar 11 11:32:26 hades teleport[2120659]: 2024-03-11T11:32:26-03:00 DEBU [PROXY:AGE] Changing state connecting -> closed. leaseID:2387 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:287
Mar 11 11:32:26 hades teleport[2120659]: 2024-03-11T11:32:26-03:00 DEBU [PROXY:AGE] Failed to connect agent. error:[
Mar 11 11:32:26 hades teleport[2120659]: ERROR REPORT:
Mar 11 11:32:26 hades teleport[2120659]: Original Error: *fmt.wrapError ssh: handshake failed: ssh: overflow reading version string
Mar 11 11:32:26 hades teleport[2120659]: Stack Trace:
Mar 11 11:32:26 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/observability/tracing/ssh/ssh.go:158 github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConn
Mar 11 11:32:26 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent_dialer.go:97 github.com/gravitational/teleport/lib/reversetunnel.(*agentDialer).DialContext
Mar 11 11:32:26 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:364 github.com/gravitational/teleport/lib/reversetunnel.(*agent).connect
Mar 11 11:32:26 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:319 github.com/gravitational/teleport/lib/reversetunnel.(*agent).Start
Mar 11 11:32:26 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:321 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).connectAgent
Mar 11 11:32:26 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:270 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).run
Mar 11 11:32:26 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:257 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start.func1
Mar 11 11:32:26 hades teleport[2120659]:         runtime/asm_amd64.s:1650 runtime.goexit
Mar 11 11:32:26 hades teleport[2120659]: User Message: ssh: handshake failed: ssh: overflow reading version string] localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:279
Mar 11 11:32:26 hades teleport[2120659]: 2024-03-11T11:32:26-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 11 11:32:26 hades teleport[2120659]: 2024-03-11T11:32:26-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 11 11:32:26 hades teleport[2120659]: 2024-03-11T11:32:26-03:00 DEBU [PROXY:AGE] Changing state connecting -> closed. leaseID:12145 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:287
Mar 11 11:32:26 hades teleport[2120659]: 2024-03-11T11:32:26-03:00 DEBU [PROXY:AGE] Failed to connect agent. error:[
Mar 11 11:32:26 hades teleport[2120659]: ERROR REPORT:
Mar 11 11:32:26 hades teleport[2120659]: Original Error: *fmt.wrapError ssh: handshake failed: ssh: overflow reading version string
Mar 11 11:32:26 hades teleport[2120659]: Stack Trace:
Mar 11 11:32:26 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/observability/tracing/ssh/ssh.go:158 github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConn
Mar 11 11:32:26 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent_dialer.go:97 github.com/gravitational/teleport/lib/reversetunnel.(*agentDialer).DialContext
Mar 11 11:32:26 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:364 github.com/gravitational/teleport/lib/reversetunnel.(*agent).connect
Mar 11 11:32:26 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:319 github.com/gravitational/teleport/lib/reversetunnel.(*agent).Start
Mar 11 11:32:26 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:321 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).connectAgent
Mar 11 11:32:26 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:270 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).run
Mar 11 11:32:26 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:257 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start.func1
Mar 11 11:32:26 hades teleport[2120659]:         runtime/asm_amd64.s:1650 runtime.goexit
Mar 11 11:32:26 hades teleport[2120659]: User Message: ssh: handshake failed: ssh: overflow reading version string] localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:279
Mar 11 11:32:26 hades teleport[2120659]: 2024-03-11T11:32:26-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 11 11:32:26 hades teleport[2120659]: 2024-03-11T11:32:26-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 11 11:32:30 hades teleport[2120659]: 2024-03-11T11:32:30-03:00 DEBU [PROXY:AGE] Runtime config: restrict_connection_count: false connection_count: 1 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:340
Mar 11 11:32:30 hades teleport[2120659]: 2024-03-11T11:32:30-03:00 DEBU             Attempting GET teleport.example.com:443/webapi/find webclient/webclient.go:129
Mar 11 11:32:30 hades teleport[2120659]: 2024-03-11T11:32:30-03:00 DEBU [PROXY:AGE] Starting agent {teleport.example.com:443 tcp } leaseID:12146 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:299
Mar 11 11:32:30 hades teleport[2120659]: 2024-03-11T11:32:30-03:00 DEBU [PROXY:AGE] Changing state initial -> connecting. leaseID:12146 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:287
Mar 11 11:32:30 hades teleport[2120659]: 2024-03-11T11:32:30-03:00 DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:197
Mar 11 11:32:30 hades teleport[2120659]: 2024-03-11T11:32:30-03:00 DEBU [PROXY:AGE] Changing state connecting -> closed. leaseID:12146 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:287
Mar 11 11:32:30 hades teleport[2120659]: 2024-03-11T11:32:30-03:00 DEBU [PROXY:AGE] Failed to connect agent. error:[
Mar 11 11:32:30 hades teleport[2120659]: ERROR REPORT:
Mar 11 11:32:30 hades teleport[2120659]: Original Error: *fmt.wrapError ssh: handshake failed: ssh: overflow reading version string
Mar 11 11:32:30 hades teleport[2120659]: Stack Trace:
Mar 11 11:32:30 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/observability/tracing/ssh/ssh.go:158 github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConn
Mar 11 11:32:30 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent_dialer.go:97 github.com/gravitational/teleport/lib/reversetunnel.(*agentDialer).DialContext
Mar 11 11:32:30 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:364 github.com/gravitational/teleport/lib/reversetunnel.(*agent).connect
Mar 11 11:32:30 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:319 github.com/gravitational/teleport/lib/reversetunnel.(*agent).Start
Mar 11 11:32:30 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:321 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).connectAgent
Mar 11 11:32:30 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:270 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).run
Mar 11 11:32:30 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:257 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start.func1
Mar 11 11:32:30 hades teleport[2120659]:         runtime/asm_amd64.s:1650 runtime.goexit
Mar 11 11:32:30 hades teleport[2120659]: User Message: ssh: handshake failed: ssh: overflow reading version string] localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:279
Mar 11 11:32:30 hades teleport[2120659]: 2024-03-11T11:32:30-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 11 11:32:30 hades teleport[2120659]: 2024-03-11T11:32:30-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 11 11:32:34 hades teleport[2120659]: 2024-03-11T11:32:34-03:00 DEBU [PROXY:AGE] Runtime config: restrict_connection_count: false connection_count: 1 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:340
Mar 11 11:32:34 hades teleport[2120659]: 2024-03-11T11:32:34-03:00 DEBU             Attempting GET teleport.example.com:443/webapi/find webclient/webclient.go:129
Mar 11 11:32:34 hades teleport[2120659]: 2024-03-11T11:32:34-03:00 DEBU [PROXY:AGE] Starting agent {teleport.example.com:443 tcp } leaseID:2388 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:299
Mar 11 11:32:34 hades teleport[2120659]: 2024-03-11T11:32:34-03:00 DEBU [PROXY:AGE] Changing state initial -> connecting. leaseID:2388 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:287
Mar 11 11:32:34 hades teleport[2120659]: 2024-03-11T11:32:34-03:00 DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:197
Mar 11 11:32:34 hades teleport[2120659]: 2024-03-11T11:32:34-03:00 DEBU [PROXY:AGE] Changing state connecting -> closed. leaseID:2388 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:287
Mar 11 11:32:34 hades teleport[2120659]: 2024-03-11T11:32:34-03:00 DEBU [PROXY:AGE] Failed to connect agent. error:[
Mar 11 11:32:34 hades teleport[2120659]: ERROR REPORT:
Mar 11 11:32:34 hades teleport[2120659]: Original Error: *fmt.wrapError ssh: handshake failed: ssh: overflow reading version string
Mar 11 11:32:34 hades teleport[2120659]: Stack Trace:
Mar 11 11:32:34 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/observability/tracing/ssh/ssh.go:158 github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConn
Mar 11 11:32:34 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent_dialer.go:97 github.com/gravitational/teleport/lib/reversetunnel.(*agentDialer).DialContext
Mar 11 11:32:34 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:364 github.com/gravitational/teleport/lib/reversetunnel.(*agent).connect
Mar 11 11:32:34 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:319 github.com/gravitational/teleport/lib/reversetunnel.(*agent).Start
Mar 11 11:32:34 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:321 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).connectAgent
Mar 11 11:32:34 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:270 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).run
Mar 11 11:32:34 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:257 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start.func1
Mar 11 11:32:34 hades teleport[2120659]:         runtime/asm_amd64.s:1650 runtime.goexit
Mar 11 11:32:34 hades teleport[2120659]: User Message: ssh: handshake failed: ssh: overflow reading version string] localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:279
Mar 11 11:32:34 hades teleport[2120659]: 2024-03-11T11:32:34-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 11 11:32:34 hades teleport[2120659]: 2024-03-11T11:32:34-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 11 11:32:37 hades teleport[2120659]: 2024-03-11T11:32:37-03:00 DEBU [PROXY:AGE] Runtime config: restrict_connection_count: false connection_count: 1 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:340
Mar 11 11:32:37 hades teleport[2120659]: 2024-03-11T11:32:37-03:00 DEBU             Attempting GET teleport.example.com:443/webapi/find webclient/webclient.go:129
Mar 11 11:32:38 hades teleport[2120659]: 2024-03-11T11:32:38-03:00 DEBU [PROXY:AGE] Starting agent {teleport.example.com:443 tcp } leaseID:12147 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:299
Mar 11 11:32:38 hades teleport[2120659]: 2024-03-11T11:32:38-03:00 DEBU [PROXY:AGE] Changing state initial -> connecting. leaseID:12147 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:287
Mar 11 11:32:38 hades teleport[2120659]: 2024-03-11T11:32:38-03:00 DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:197
Mar 11 11:32:38 hades teleport[2120659]: 2024-03-11T11:32:38-03:00 DEBU [PROXY:AGE] Changing state connecting -> closed. leaseID:12147 target:teleport.example.com:443 localCluster: targetCluster:teleport.example.com reversetunnel/agent.go:287
Mar 11 11:32:38 hades teleport[2120659]: 2024-03-11T11:32:38-03:00 DEBU [PROXY:AGE] Failed to connect agent. error:[
Mar 11 11:32:38 hades teleport[2120659]: ERROR REPORT:
Mar 11 11:32:38 hades teleport[2120659]: Original Error: *fmt.wrapError ssh: handshake failed: ssh: overflow reading version string
Mar 11 11:32:38 hades teleport[2120659]: Stack Trace:
Mar 11 11:32:38 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/observability/tracing/ssh/ssh.go:158 github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConn
Mar 11 11:32:38 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent_dialer.go:97 github.com/gravitational/teleport/lib/reversetunnel.(*agentDialer).DialContext
Mar 11 11:32:38 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:364 github.com/gravitational/teleport/lib/reversetunnel.(*agent).connect
Mar 11 11:32:38 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agent.go:319 github.com/gravitational/teleport/lib/reversetunnel.(*agent).Start
Mar 11 11:32:38 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:321 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).connectAgent
Mar 11 11:32:38 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:270 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).run
Mar 11 11:32:38 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:257 github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start.func1
Mar 11 11:32:38 hades teleport[2120659]:         runtime/asm_amd64.s:1650 runtime.goexit
Mar 11 11:32:38 hades teleport[2120659]: User Message: ssh: handshake failed: ssh: overflow reading version string] localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:279
Mar 11 11:32:38 hades teleport[2120659]: 2024-03-11T11:32:38-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
Mar 11 11:32:38 hades teleport[2120659]: 2024-03-11T11:32:38-03:00 DEBU [PROXY:AGE] Active agent count: 0 localCluster: targetCluster:teleport.example.com reversetunnel/agentpool.go:423
webvictim commented 8 months ago

Dump from kill -USR1 $(pidof teleport):

Mar 11 11:37:04 hades teleport[2120659]: Runtime stats
Mar 11 11:37:04 hades teleport[2120659]: goroutines: 140
Mar 11 11:37:04 hades teleport[2120659]: OS threads: 24
Mar 11 11:37:04 hades teleport[2120659]: GOMAXPROCS: 8
Mar 11 11:37:04 hades teleport[2120659]: num CPU: 8
Mar 11 11:37:04 hades teleport[2120659]: Memory stats
Mar 11 11:37:04 hades teleport[2120659]: alloc: 49049040
Mar 11 11:37:04 hades teleport[2120659]: total-alloc: 29254741128
Mar 11 11:37:04 hades teleport[2120659]: sys: 82698520
Mar 11 11:37:04 hades teleport[2120659]: lookups: 0
Mar 11 11:37:04 hades teleport[2120659]: mallocs: 489299070
Mar 11 11:37:04 hades teleport[2120659]: frees: 489074792
Mar 11 11:37:04 hades teleport[2120659]: heap-alloc: 49049040
Mar 11 11:37:04 hades teleport[2120659]: heap-sys: 68714496
Mar 11 11:37:04 hades teleport[2120659]: heap-idle: 14499840
Mar 11 11:37:04 hades teleport[2120659]: heap-in-use: 54214656
Mar 11 11:37:04 hades teleport[2120659]: heap-released: 9453568
Mar 11 11:37:04 hades teleport[2120659]: heap-objects: 224278
Mar 11 11:37:04 hades teleport[2120659]: stack-in-use: 2588672
Mar 11 11:37:04 hades teleport[2120659]: stack-sys: 2588672
Mar 11 11:37:04 hades teleport[2120659]: stack-mspan-inuse: 756672
Mar 11 11:37:04 hades teleport[2120659]: stack-mspan-sys: 847392
Mar 11 11:37:04 hades teleport[2120659]: stack-mcache-inuse: 9600
Mar 11 11:37:04 hades teleport[2120659]: stack-mcache-sys: 15600
Mar 11 11:37:04 hades teleport[2120659]: other-sys: 1773113
Mar 11 11:37:04 hades teleport[2120659]: gc-sys: 6520648
Mar 11 11:37:04 hades teleport[2120659]: next-gc: when heap-alloc >= 58536760
Mar 11 11:37:04 hades teleport[2120659]: last-gc: 2024-03-11 11:35:20.926868905 -0300 ADT
Mar 11 11:37:04 hades teleport[2120659]: gc-pause-total: 445.70447ms
Mar 11 11:37:04 hades teleport[2120659]: gc-pause: 160037
Mar 11 11:37:04 hades teleport[2120659]: num-gc: 4634
Mar 11 11:37:04 hades teleport[2120659]: enable-gc: true
Mar 11 11:37:04 hades teleport[2120659]: debug-gc: false
Mar 11 11:37:04 hades teleport[2120659]: Goroutines
Mar 11 11:37:04 hades teleport[2120659]: goroutine 1 [running, locked to thread]:
Mar 11 11:37:04 hades teleport[2120659]: runtime/pprof.writeGoroutineStacks({0x55ebbf6f29e8, 0xc0000ea050})
Mar 11 11:37:04 hades teleport[2120659]:         runtime/pprof/pprof.go:703 +0x6a
Mar 11 11:37:04 hades teleport[2120659]: runtime/pprof.writeGoroutine({0x55ebbf6f29e8?, 0xc0000ea050?}, 0xb?)
Mar 11 11:37:04 hades teleport[2120659]:         runtime/pprof/pprof.go:692 +0x25
Mar 11 11:37:04 hades teleport[2120659]: runtime/pprof.(*Profile).WriteTo(0x55ebbbca126e?, {0x55ebbf6f29e8?, 0xc0000ea050?}, 0x0?)
Mar 11 11:37:04 hades teleport[2120659]:         runtime/pprof/pprof.go:329 +0x146
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.goroutineDump({0x55ebbf6f29e8, 0xc0000ea050})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/info.go:47 +0x3d
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.writeDebugInfo({0x55ebbf6f29e8, 0xc0000ea050})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/info.go:43 +0xa9
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*TeleportProcess).WaitForSignals(0xc0015dc000, {0x55ebbf7afb00, 0x55ebc4503a00})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/signals.go:119 +0xa6d
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.waitAndReload({_, _}, {{0x55ebbbc240a8, 0x2}, {0xc0011aed08, 0x11}, {0xc001b80e10, 0x5}, {0x55ebbbc40667, 0x5}, ...}, ...)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/service.go:728 +0x6c
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.Run({_, _}, {{0x55ebbbc240a8, 0x2}, {0xc0011aed08, 0x11}, {0xc001b80e10, 0x5}, {0x55ebbbc40667, 0x5}, ...}, ...)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/service.go:715 +0x245
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/tool/teleport/common.OnStart({{0x0, 0x0}, {0x0, 0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}, ...}, ...)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/tool/teleport/common/teleport.go:638 +0x1fa
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/tool/teleport/common.Run({{0xc0000b61d0?, 0x55ebc4228348?, 0x55ebc4228348?}, 0xa0?, 0x61?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/tool/teleport/common/teleport.go:547 +0xb685
Mar 11 11:37:04 hades teleport[2120659]: main.main()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/tool/teleport/main.go:33 +0x4a
Mar 11 11:37:04 hades teleport[2120659]: goroutine 9 [select]:
Mar 11 11:37:04 hades teleport[2120659]: go.opencensus.io/stats/view.(*worker).start(0xc0011ca680)
Mar 11 11:37:04 hades teleport[2120659]:         go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f
Mar 11 11:37:04 hades teleport[2120659]: created by go.opencensus.io/stats/view.init.0 in goroutine 1
Mar 11 11:37:04 hades teleport[2120659]:         go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159568 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).forwardKeepAlives(0xc0021d9b00)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:67 +0xb1
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 561
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:46 +0x16f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 93 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).fanOut(0xc001e20460)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:515 +0xb1
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/service.NewSupervisor in goroutine 1
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:207 +0x45b
Mar 11 11:37:04 hades teleport[2120659]: goroutine 94 [select, 9272 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: database/sql.(*DB).connectionOpener(0xc001e29790, {0x55ebbf7afcc0, 0xc001be5360})
Mar 11 11:37:04 hades teleport[2120659]:         database/sql/sql.go:1218 +0x87
Mar 11 11:37:04 hades teleport[2120659]: created by database/sql.OpenDB in goroutine 1
Mar 11 11:37:04 hades teleport[2120659]:         database/sql/sql.go:791 +0x165
Mar 11 11:37:04 hades teleport[2120659]: goroutine 106 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc001e5a970, {0x55ebbf7afcc0, 0xc001be4d70})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 159
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129
Mar 11 11:37:04 hades teleport[2120659]: goroutine 107 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc001e5a9e0, {0x55ebbf7afcc0, 0xc001be4dc0})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 159
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129
Mar 11 11:37:04 hades teleport[2120659]: goroutine 130 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/backend/lite.(*Backend).runPeriodicOperations(0xc001bc04d0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/backend/lite/periodic.go:39 +0xf2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/backend/lite.NewWithConfig in goroutine 1
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/backend/lite/lite.go:265 +0x6c9
Mar 11 11:37:04 hades teleport[2120659]: goroutine 116 [syscall]:
Mar 11 11:37:04 hades teleport[2120659]: os/signal.signal_recv()
Mar 11 11:37:04 hades teleport[2120659]:         runtime/sigqueue.go:152 +0x29
Mar 11 11:37:04 hades teleport[2120659]: os/signal.loop()
Mar 11 11:37:04 hades teleport[2120659]:         os/signal/signal_unix.go:23 +0x13
Mar 11 11:37:04 hades teleport[2120659]: created by os/signal.Notify.func1.1 in goroutine 1
Mar 11 11:37:04 hades teleport[2120659]:         os/signal/signal.go:151 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158927 [select]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0015a2fa0, 0x1)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:418 +0x113
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00016cf50)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:552 +0x86
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:452 +0x85
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 3158903
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:450 +0x246b
Mar 11 11:37:04 hades teleport[2120659]: goroutine 145 [select, 9272 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).WaitForEvent(0xc001e20460, {0x55ebbf7afb00, 0x55ebc4503a00}, {0x55ebbbcd7328, 0xc})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:483 +0x3c5
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*TeleportProcess).initUploaderService.(*TeleportProcess).OnExit.func6()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/service.go:2249 +0x5a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalService).Serve(0xc000fac930?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:587 +0x12
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:312 +0x2a2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve in goroutine 166
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:301 +0xa5
Mar 11 11:37:04 hades teleport[2120659]: goroutine 563 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv.(*Heartbeat).Run(0xc00222a7e0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/heartbeat.go:287 +0x1b0
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/srv/app.(*Server).startHeartbeat in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/server.go:475 +0x273
Mar 11 11:37:04 hades teleport[2120659]: goroutine 178 [select, 25 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/utils/interval.(*Interval).run(0xc001e8a300, {0x55ebbf78b300?, 0xc001e8c0a0})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/interval/interval.go:160 +0x1c5
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/utils/interval.New in goroutine 154
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/interval/interval.go:105 +0x225
Mar 11 11:37:04 hades teleport[2120659]: goroutine 146 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/inventory.(*downstreamHandle).handleStream(0xc001590500, {0x55ebbf7c0ef0?, 0xc002586ac0}, {{0x55ebbbc57616, 0x6}, {0xc001d9e120, 0x24}, {0xc00158ab00, 0x2, 0x2}, ...})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/inventory/inventory.go:237 +0x707
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/inventory.(*downstreamHandle).tryRun(0xc001590500, 0x0?, {{0x55ebbbc57616, 0x6}, {0xc001d9e120, 0x24}, {0xc00158ab00, 0x2, 0x2}, {0xc001b80e10, ...}, ...})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/inventory/inventory.go:196 +0x126
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/inventory.(*downstreamHandle).run(0xc001590500, 0x0?, {{0x55ebbbc57616, 0x6}, {0xc001d9e120, 0x24}, {0xc00158ab00, 0x2, 0x2}, {0xc001b80e10, ...}, ...})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/inventory/inventory.go:171 +0x88
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/inventory.NewDownstreamHandle in goroutine 1
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/inventory/inventory.go:92 +0x199
Mar 11 11:37:04 hades teleport[2120659]: goroutine 147 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/inventory.(*downstreamHandle).autoEmitMetadata(0xc001590500)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/inventory/inventory.go:160 +0x33f
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/inventory.NewDownstreamHandle in goroutine 1
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/inventory/inventory.go:93 +0x1d6
Mar 11 11:37:04 hades teleport[2120659]: goroutine 144 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/events.(*UploadCompleter).Serve(0xc001fc2800, {0x55ebbf7afcc0, 0xc001be5270})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/events/complete.go:148 +0x22a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*TeleportProcess).initUploaderService.func3()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/service.go:2996 +0x52
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalService).Serve(0xc000faa310?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:587 +0x12
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:312 +0x2a2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve in goroutine 166
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:301 +0xa5
Mar 11 11:37:04 hades teleport[2120659]: goroutine 154 [select, 25 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/versioncontrol/upgradewindow.(*Exporter[...]).run(0x55ebbf7c4ce0, {0x55ebbf7afcc0, 0xc0015a2870})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/versioncontrol/upgradewindow/upgradewindow.go:223 +0x4fa
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/versioncontrol/upgradewindow.(*Exporter[...]).Run(...)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/versioncontrol/upgradewindow/upgradewindow.go:191
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalService).Serve(0xc001ea2000?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:587 +0x12
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:312 +0x2a2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve in goroutine 1
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:301 +0xa5
Mar 11 11:37:04 hades teleport[2120659]: goroutine 155 [select, 9272 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).WaitForEvent(0xc001e20460, {0x55ebbf7afb00, 0x55ebc4503a00}, {0x55ebbbcd7328, 0xc})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:483 +0x3c5
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.NewTeleport.(*TeleportProcess).OnExit.func6()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/service.go:2249 +0x5a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalService).Serve(0xc00143e700?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:587 +0x12
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:312 +0x2a2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve in goroutine 1
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:301 +0xa5
Mar 11 11:37:04 hades teleport[2120659]: goroutine 156 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*TeleportProcess).initDiagnosticService.func2()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/service.go:3151 +0x18d
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalService).Serve(0xc001f86000?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:587 +0x12
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:312 +0x2a2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve in goroutine 1
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:301 +0xa5
Mar 11 11:37:04 hades teleport[2120659]: goroutine 157 [IO wait, 3 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.runtime_pollWait(0x7f6e643e4f68, 0x72)
Mar 11 11:37:04 hades teleport[2120659]:         runtime/netpoll.go:343 +0x85
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.(*pollDesc).wait(0xc00159c200?, 0x4?, 0x0)
Mar 11 11:37:04 hades teleport[2120659]:         internal/poll/fd_poll_runtime.go:84 +0x27
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.(*pollDesc).waitRead(...)
Mar 11 11:37:04 hades teleport[2120659]:         internal/poll/fd_poll_runtime.go:89
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.(*FD).Accept(0xc00159c200)
Mar 11 11:37:04 hades teleport[2120659]:         internal/poll/fd_unix.go:611 +0x2ac
Mar 11 11:37:04 hades teleport[2120659]: net.(*netFD).accept(0xc00159c200)
Mar 11 11:37:04 hades teleport[2120659]:         net/fd_unix.go:172 +0x29
Mar 11 11:37:04 hades teleport[2120659]: net.(*TCPListener).accept(0xc00158ad60)
Mar 11 11:37:04 hades teleport[2120659]:         net/tcpsock_posix.go:152 +0x1e
Mar 11 11:37:04 hades teleport[2120659]: net.(*TCPListener).Accept(0xc00158ad60)
Mar 11 11:37:04 hades teleport[2120659]:         net/tcpsock.go:315 +0x30
Mar 11 11:37:04 hades teleport[2120659]: net/http.(*Server).Serve(0xc0003c6960, {0x55ebbf789bb0, 0xc00158ad60})
Mar 11 11:37:04 hades teleport[2120659]:         net/http/server.go:3056 +0x364
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*TeleportProcess).initDiagnosticService.func4()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/service.go:3202 +0x2c
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalService).Serve(0xc00143e8c0?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:587 +0x12
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:312 +0x2a2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve in goroutine 1
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:301 +0xa5
Mar 11 11:37:04 hades teleport[2120659]: goroutine 158 [select, 9272 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).WaitForEvent(0xc001e20460, {0x55ebbf7afb00, 0x55ebc4503a00}, {0x55ebbbcd7328, 0xc})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:483 +0x3c5
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*TeleportProcess).initDiagnosticService.(*TeleportProcess).OnExit.func6()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/service.go:2249 +0x5a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalService).Serve(0xc00143e7e0?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:587 +0x12
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:312 +0x2a2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve in goroutine 1
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:301 +0xa5
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159890 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc0015f8000)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:329 +0x7c
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:344
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc001c12f70, {0x55ebbeafe740?, 0xc001f0dc50}, 0xc00191fda0?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1070 +0xc9
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x2?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc000d3fd40, 0xc0022dfe60, 0xc0022dfe50)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:776 +0x13a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc000d3fd40, {0x55ebbeafe740?, 0xc001f0dc50?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0xc000b76000?, {0x55ebbeafe740?, 0xc001f0dc50?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc001f94ec0, {0x55ebbeafe740, 0xc001f0dc50})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).recv(0xc0021b7c80)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:95 +0x45
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 580
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:47 +0x1ab
Mar 11 11:37:04 hades teleport[2120659]: goroutine 373 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/labels.(*Dynamic).periodicUpdateLabel(0xc0020e09c0, {0xc001b81100, 0x6}, {0x55ebbf7be398, 0xc0029a0320})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:143 +0x125
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/labels.(*Dynamic).Start in goroutine 436
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:126 +0x72
Mar 11 11:37:04 hades teleport[2120659]: goroutine 162 [select, 9272 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).WaitForEvent(0xc001e20460, {0x55ebbf7afcc0, 0xc001be5270}, {0x55ebbbcd7328, 0xc})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:483 +0x3c5
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*TeleportProcess).initSSH.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/service.go:2786 +0x25c9
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalService).Serve(0xc001446a80?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:587 +0x12
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:312 +0x2a2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve in goroutine 1
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:301 +0xa5
Mar 11 11:37:04 hades teleport[2120659]: goroutine 405 [select, 3 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/cache.(*Cache).fetchAndWatch(0xc002cec500, {0x55ebbf7afcc0, 0xc002894b90}, {0x55ebbf7b6b60, 0xc00270f1a0}, 0xc002895270)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/cache/cache.go:1306 +0xef6
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/cache.(*Cache).update(0xc002cec500, {0x55ebbf7afcc0, 0xc002894b90}, {0x55ebbf7b6b60?, 0xc00270f1a0?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/cache/cache.go:1104 +0x191
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/cache.(*Cache).Start in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/cache/cache.go:982 +0x225
Mar 11 11:37:04 hades teleport[2120659]: goroutine 164 [chan receive, 9272 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv/app.(*Server).Wait(0xc000fbf540)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/server.go:670 +0x2b
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*TeleportProcess).initApps.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/service.go:5474 +0x19c6
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalService).Serve(0xc001446bd0?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:587 +0x12
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:312 +0x2a2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve in goroutine 1
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:301 +0xa5
Mar 11 11:37:04 hades teleport[2120659]: goroutine 165 [select, 7 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncRotationStateCycle(0xc0015dc000)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/connect.go:703 +0x6b2
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*TeleportProcess).periodicSyncRotationState(0xc0015dc000)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/connect.go:644 +0x1f8
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalService).Serve(0xc00143eb60?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:587 +0x12
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:312 +0x2a2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve in goroutine 1
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:301 +0xa5
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158939 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0024950e0, {0xc002455d50, 0x5, 0x5})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:192 +0x98
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0024950e0, {0xc002455d50?, 0xc0024c4210?, 0xc002917960?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:172 +0x12d
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc002455d10, {0xc002455d50?, 0xc0029179d8?, 0x55ebb4b8e765?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:520 +0x2c
Mar 11 11:37:04 hades teleport[2120659]: io.ReadAtLeast({0x55ebbf7069c0, 0xc002455d10}, {0xc002455d50, 0x5, 0x5}, 0x5)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:335 +0x90
Mar 11 11:37:04 hades teleport[2120659]: io.ReadFull(...)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:354
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0022af7a0, {0xc002455d50, 0x5, 0x5})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:504 +0x96
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*parser).recvMsg(0xc002455d40, 0x400000)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:600 +0x46
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.recvAndDecompress(0xc0030e6bb8?, 0xc0022af7a0, {0x0, 0x0}, 0x400000, 0x0, {0x0, 0x0})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:730 +0x5a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.recv(0xc002455d40, {0x7f6e64451ef8, 0x55ebc4503a00}, 0x0?, {0x0, 0x0}, {0x55ebbf589b00, 0xc0022f7f40}, 0x108?, 0x0, ...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:796 +0xa6
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc00143de10, {0x55ebbf589b00?, 0xc0022f7f40}, 0xc0015fdba0?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1084 +0x289
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0000c8500?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc0022ae6c0, 0xc002917e08, 0xc002917df8)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:760 +0x3ae
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc0022ae6c0, {0x55ebbf589b00?, 0xc0022f7f40?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0x40?, {0x55ebbf589b00?, 0xc0022f7f40?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc0022f7c40, {0x55ebbf589b00, 0xc0022f7f40})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client/proto.(*authServiceWatchEventsClient).Recv(0xc0024ce4d0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/proto/authservice.pb.go:17344 +0x46
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamWatcher).receiveEvents(0xc001d8d260)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/streamwatcher.go:84 +0x3b
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewWatcher in goroutine 430
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/streamwatcher.go:47 +0x1d6
Mar 11 11:37:04 hades teleport[2120659]: goroutine 376 [select, 33 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/labels.(*Dynamic).periodicUpdateLabel(0xc0020e09c0, {0xc001b81060, 0x6}, {0x55ebbf7be398, 0xc0029a04b0})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:143 +0x125
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/labels.(*Dynamic).Start in goroutine 436
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:126 +0x72
Mar 11 11:37:04 hades teleport[2120659]: goroutine 236 [select, 9272 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00284df50, {0x55ebbf7afcc0, 0xc002870460})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 163
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159569 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc001d8bb00)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:329 +0x7c
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:344
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc0029112b0, {0x55ebbeafe740?, 0xc001f79500}, 0x2c565a802512060?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1070 +0xc9
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x55ebb53c1daf?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc001d8b8c0, 0xc0030e4e60, 0xc0030e4e50)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:776 +0x13a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc001d8b8c0, {0x55ebbeafe740?, 0xc001f79500?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0xc000ff87e0?, {0x55ebbeafe740?, 0xc001f79500?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc002ddf8c0, {0x55ebbeafe740, 0xc001f79500})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).recv(0xc0021d9b00)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:95 +0x45
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 561
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:47 +0x1ab
Mar 11 11:37:04 hades teleport[2120659]: goroutine 105 [select, 9272 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc001e5a940, {0x55ebbf7afcc0, 0xc001be4d20})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 159
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129
Mar 11 11:37:04 hades teleport[2120659]: goroutine 142 [select, 9272 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).WaitForEvent(0xc001e20460, {0x55ebbf7afb00, 0x55ebc4503a00}, {0x55ebbbcd7328, 0xc})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:483 +0x3c5
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*TeleportProcess).initUploaderService.(*TeleportProcess).OnExit.func5()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/service.go:2249 +0x5a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalService).Serve(0xc000faa230?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:587 +0x12
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:312 +0x2a2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve in goroutine 166
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:301 +0xa5
Mar 11 11:37:04 hades teleport[2120659]: goroutine 141 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/events/filesessions.(*Uploader).Serve(0xc001eac370, {0x55ebbf7afcc0, 0xc001be5270})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/events/filesessions/fileasync.go:204 +0x433
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*TeleportProcess).initUploaderService.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/service.go:2962 +0x52
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalService).Serve(0xc0014468c0?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:587 +0x12
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:312 +0x2a2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve in goroutine 166
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:301 +0xa5
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3148592 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0025888c0, {0xc0022041f0, 0x5, 0x5})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:192 +0x98
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0025888c0, {0xc0022041f0?, 0xc001f342d0?, 0xc0020f0948?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:172 +0x12d
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0022041b0, {0xc0022041f0?, 0xc0020f09c0?, 0x55ebb4b8e765?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:520 +0x2c
Mar 11 11:37:04 hades teleport[2120659]: io.ReadAtLeast({0x55ebbf7069c0, 0xc0022041b0}, {0xc0022041f0, 0x5, 0x5}, 0x5)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:335 +0x90
Mar 11 11:37:04 hades teleport[2120659]: io.ReadFull(...)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:354
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0029fcb40, {0xc0022041f0, 0x5, 0x5})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:504 +0x96
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*parser).recvMsg(0xc0022041e0, 0x400000)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:600 +0x46
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.recvAndDecompress(0xc0020f0ba0?, 0xc0029fcb40, {0x0, 0x0}, 0x400000, 0x0, {0x0, 0x0})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:730 +0x5a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.recv(0xc0022041e0, {0x7f6e64451ef8, 0x55ebc4503a00}, 0x55ebb46a1b0a?, {0x0, 0x0}, {0x55ebbf3c2220, 0xc002204660}, 0xc002900000?, 0x0, ...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:796 +0xa6
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc0016801a0, {0x55ebbf3c2220?, 0xc002204660}, 0xc00257ca20?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1084 +0x289
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc002e4e548?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc0022af200, 0xc0020f0df0, 0xc0020f0de0)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:760 +0x3ae
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc0022af200, {0x55ebbf3c2220?, 0xc002204660?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0x30?, {0x55ebbf3c2220?, 0xc002204660?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc002586a80, {0x55ebbf3c2220, 0xc002204660})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client/proto.(*authServiceInventoryControlStreamClient).Recv(0xc002584d20)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/proto/authservice.pb.go:17043 +0x46
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*downstreamICS).runRecvLoop(0xc002586ac0, {0x55ebbf7caa80, 0xc002584d20})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/inventory.go:284 +0x4d
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.newDownstreamInventoryControlStream in goroutine 146
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/inventory.go:257 +0x12c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159759 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).forwardKeepAlives(0xc002c31800)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:67 +0xb1
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 559
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:46 +0x16f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3148582 [IO wait]:
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.runtime_pollWait(0x7f6e6445a338, 0x72)
Mar 11 11:37:04 hades teleport[2120659]:         runtime/netpoll.go:343 +0x85
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.(*pollDesc).wait(0xc002041700?, 0xc002d14000?, 0x0)
Mar 11 11:37:04 hades teleport[2120659]:         internal/poll/fd_poll_runtime.go:84 +0x27
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.(*pollDesc).waitRead(...)
Mar 11 11:37:04 hades teleport[2120659]:         internal/poll/fd_poll_runtime.go:89
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.(*FD).Read(0xc002041700, {0xc002d14000, 0x4000, 0x4000})
Mar 11 11:37:04 hades teleport[2120659]:         internal/poll/fd_unix.go:164 +0x27a
Mar 11 11:37:04 hades teleport[2120659]: net.(*netFD).Read(0xc002041700, {0xc002d14000?, 0xc002d14005?, 0x75?})
Mar 11 11:37:04 hades teleport[2120659]:         net/fd_posix.go:55 +0x25
Mar 11 11:37:04 hades teleport[2120659]: net.(*conn).Read(0xc001a92048, {0xc002d14000?, 0x18?, 0xc001356c38?})
Mar 11 11:37:04 hades teleport[2120659]:         net/net.go:179 +0x45
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*atLeastReader).Read(0xc002b322b8, {0xc002d14000?, 0xc002b322b8?, 0x0?})
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:805 +0x3b
Mar 11 11:37:04 hades teleport[2120659]: bytes.(*Buffer).ReadFrom(0xc001356d28, {0x55ebbf6f9820, 0xc002b322b8})
Mar 11 11:37:04 hades teleport[2120659]:         bytes/buffer.go:211 +0x98
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).readFromUntil(0xc001356a80, {0x55ebbf6f6840?, 0xc001a92048}, 0x4000?)
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:827 +0xde
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).readRecordOrCCS(0xc001356a80, 0x0)
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:625 +0x250
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).readRecord(...)
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:587
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).Read(0xc001356a80, {0xc0025d9000, 0x1000, 0xc00291fda0?})
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:1369 +0x158
Mar 11 11:37:04 hades teleport[2120659]: bufio.(*Reader).Read(0xc0027d88a0, {0xc002588420, 0x4, 0x55ebb46948fc?})
Mar 11 11:37:04 hades teleport[2120659]:         bufio/bufio.go:244 +0x197
Mar 11 11:37:04 hades teleport[2120659]: io.ReadAtLeast({0x55ebbf6f3220, 0xc0027d88a0}, {0xc002588420, 0x4, 0x4}, 0x4)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:335 +0x90
Mar 11 11:37:04 hades teleport[2120659]: io.ReadFull(...)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:354
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*gcmCipher).readCipherPacket(0xc002588410, 0xbded9680?, {0x55ebbf6f3220, 0xc0027d88a0})
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/cipher.go:373 +0x56
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*connectionState).readPacket(0xc0029fc120, 0x55ebbe0fef00?, 0x1)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/transport.go:147 +0x42
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*transport).readPacket(0xc0029fc120)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/transport.go:130 +0x2b
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*handshakeTransport).readOnePacket(0xc001a70340, 0x0)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/handshake.go:400 +0x37
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*handshakeTransport).readLoop(0xc001a70340)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/handshake.go:219 +0x2a
Mar 11 11:37:04 hades teleport[2120659]: created by golang.org/x/crypto/ssh.newClientTransport in goroutine 3158876
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/handshake.go:155 +0x21f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 358 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/cache.(*Cache).fetchAndWatch(0xc0012d4f00, {0x55ebbf7afcc0, 0xc002870a50}, {0x55ebbf7b6b60, 0xc001d8c9c0}, 0xc002871130)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/cache/cache.go:1306 +0xef6
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/cache.(*Cache).update(0xc0012d4f00, {0x55ebbf7afcc0, 0xc002870a50}, {0x55ebbf7b6b60?, 0xc001d8c9c0?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/cache/cache.go:1104 +0x191
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/cache.(*Cache).Start in goroutine 162
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/cache/cache.go:982 +0x225
Mar 11 11:37:04 hades teleport[2120659]: goroutine 374 [select, 33 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/labels.(*Dynamic).periodicUpdateLabel(0xc0020e09c0, {0xc0011aed80, 0x11}, {0x55ebbf7be398, 0xc0029a0370})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:143 +0x125
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/labels.(*Dynamic).Start in goroutine 436
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:126 +0x72
Mar 11 11:37:04 hades teleport[2120659]: goroutine 237 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00284df80, {0x55ebbf7afcc0, 0xc0028704b0})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 163
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129
Mar 11 11:37:04 hades teleport[2120659]: goroutine 238 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00284dfb0, {0x55ebbf7afcc0, 0xc002870500})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 163
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129
Mar 11 11:37:04 hades teleport[2120659]: goroutine 172 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/utils/interval.(*Interval).run(0xc002884f00, {0x55ebbf78b300?, 0xc002894280})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/interval/interval.go:160 +0x1c5
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/utils/interval.New in goroutine 144
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/interval/interval.go:105 +0x225
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3148587 [sync.Cond.Wait, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: sync.runtime_notifyListWait(0xc002586550, 0x0)
Mar 11 11:37:04 hades teleport[2120659]:         runtime/sema.go:527 +0x15a
Mar 11 11:37:04 hades teleport[2120659]: sync.(*Cond).Wait(0xc00011e060?)
Mar 11 11:37:04 hades teleport[2120659]:         sync/cond.go:70 +0x85
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*mux).Wait(0xc000f5f500)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/mux.go:110 +0x8e
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.NewClient.func1()
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/client.go:62 +0x28
Mar 11 11:37:04 hades teleport[2120659]: created by golang.org/x/crypto/ssh.NewClient in goroutine 3158876
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/client.go:61 +0x165
Mar 11 11:37:04 hades teleport[2120659]: goroutine 429 [select, 2402 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/events.(*AsyncEmitter).forward(0xc0020bd980)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/events/emitter.go:94 +0x8b
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/events.NewAsyncEmitter in goroutine 162
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/events/emitter.go:73 +0x18f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 536 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv.(*Heartbeat).Run(0xc00294ed80)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/heartbeat.go:287 +0x1b0
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/srv/app.(*Server).startHeartbeat in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/server.go:475 +0x273
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159567 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 561
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 375 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/labels.(*Dynamic).periodicUpdateLabel(0xc0020e09c0, {0xc0011aed50, 0x12}, {0x55ebbf7be398, 0xc0029a0410})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:143 +0x125
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/labels.(*Dynamic).Start in goroutine 436
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:126 +0x72
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3148585 [chan receive, 3 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*Client).handleGlobalRequests(0x0?, 0x0?)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/client.go:142 +0x4c
Mar 11 11:37:04 hades teleport[2120659]: created by golang.org/x/crypto/ssh.NewClient in goroutine 3158876
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/client.go:59 +0xd3
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159518 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 563
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 430 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/services.(*resourceWatcher).watch(0xc0020d6300)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/services/watcher.go:336 +0x7b0
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/services.(*resourceWatcher).runWatchLoop(0xc0020d6300)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/services/watcher.go:239 +0x77
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/services.newResourceWatcher in goroutine 162
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/services/watcher.go:149 +0x39f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3148590 [select]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc002588640, 0x1)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:418 +0x113
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001e3e000)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:552 +0x86
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:452 +0x85
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 3158876
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:450 +0x246b
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158831 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 536
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3156080 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc000d3e240)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:329 +0x7c
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:344
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc001e049c0, {0x55ebbeafe740?, 0xc001f00cf0}, 0xc00165b680?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1070 +0xc9
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc00165b5c0?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc000d3e000, 0xc002922e60, 0xc002922e50)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:776 +0x13a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc000d3e000, {0x55ebbeafe740?, 0xc001f00cf0?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0xc002f289a8?, {0x55ebbeafe740?, 0xc001f00cf0?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc001f0e240, {0x55ebbeafe740, 0xc001f00cf0})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).recv(0xc0026562a0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:95 +0x45
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 579
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:47 +0x1ab
Mar 11 11:37:04 hades teleport[2120659]: goroutine 437 [select, 3 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv.(*HostUserManagement).UserCleanup(0xc0020bda40)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/usermgmt.go:469 +0x14d
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/srv/regular.(*Server).startPeriodicOperations in goroutine 162
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/regular/sshserver.go:398 +0xdf
Mar 11 11:37:04 hades teleport[2120659]: goroutine 438 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv.(*HeartbeatV2).runWithSender(0xc00123f3b0, {0x55ebbf7943d0, 0xc002588a00})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/heartbeatv2.go:356 +0x3bc
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv.(*HeartbeatV2).run(0xc00123f3b0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/heartbeatv2.go:301 +0xad3
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv.(*HeartbeatV2).Run(0xc001d902d0?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/heartbeatv2.go:394 +0x13
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/srv/regular.(*Server).startPeriodicOperations in goroutine 162
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/regular/sshserver.go:404 +0x178
Mar 11 11:37:04 hades teleport[2120659]: goroutine 439 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).waitForBackoff(0xc00120b1e0, {0x55ebbf7afcc0, 0xc002008c30}, 0xc0015b21e0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:398 +0xfe
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).run(0xc00120b1e0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:287 +0x1b2
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:257 +0x25
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start in goroutine 162
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:256 +0x138
Mar 11 11:37:04 hades teleport[2120659]: goroutine 377 [select, 33 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/labels.(*Dynamic).periodicUpdateLabel(0xc0020e09c0, {0xc001b81170, 0xa}, {0x55ebbf7be398, 0xc0029a0550})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:143 +0x125
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/labels.(*Dynamic).Start in goroutine 436
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:126 +0x72
Mar 11 11:37:04 hades teleport[2120659]: goroutine 378 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/labels.(*Dynamic).periodicUpdateLabel(0xc0020e09c0, {0xc0011aed68, 0x12}, {0x55ebbf7be398, 0xc0029a02d0})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:143 +0x125
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/labels.(*Dynamic).Start in goroutine 436
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:126 +0x72
Mar 11 11:37:04 hades teleport[2120659]: goroutine 379 [select, 33 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/labels.(*Dynamic).periodicUpdateLabel(0xc0020e09c0, {0xc001b81018, 0x2}, {0x55ebbf7be398, 0xc0029a03c0})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:143 +0x125
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/labels.(*Dynamic).Start in goroutine 436
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:126 +0x72
Mar 11 11:37:04 hades teleport[2120659]: goroutine 380 [select, 33 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/labels.(*Dynamic).periodicUpdateLabel(0xc0020e09c0, {0xc001b80f40, 0x8}, {0x55ebbf7be398, 0xc0029a0460})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:143 +0x125
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/labels.(*Dynamic).Start in goroutine 436
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:126 +0x72
Mar 11 11:37:04 hades teleport[2120659]: goroutine 381 [select, 33 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/labels.(*Dynamic).periodicUpdateLabel(0xc0020e09c0, {0xc001b80f90, 0x10}, {0x55ebbf7be398, 0xc0029a0500})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:143 +0x125
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/labels.(*Dynamic).Start in goroutine 436
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:126 +0x72
Mar 11 11:37:04 hades teleport[2120659]: goroutine 382 [select, 33 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/labels.(*Dynamic).periodicUpdateLabel(0xc0020e09c0, {0xc001b80fd0, 0x4}, {0x55ebbf7be398, 0xc0029a05a0})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:143 +0x125
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/labels.(*Dynamic).Start in goroutine 436
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/labels/labels.go:126 +0x72
Mar 11 11:37:04 hades teleport[2120659]: goroutine 441 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/utils/interval.(*Interval).run(0xc0020fab40, {0x55ebbf78b300?, 0xc002008e60})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/interval/interval.go:160 +0x1c5
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/utils/interval.New in goroutine 438
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/interval/interval.go:105 +0x225
Mar 11 11:37:04 hades teleport[2120659]: goroutine 442 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/utils/interval.(*Interval).run(0xc0020fac60, {0x55ebbf78b300?, 0xc002008eb0})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/interval/interval.go:160 +0x1c5
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/utils/interval.New in goroutine 438
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/interval/interval.go:105 +0x225
Mar 11 11:37:04 hades teleport[2120659]: goroutine 443 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/utils/interval.(*Interval).run(0xc0020fad80, {0x55ebbf78b300?, 0xc002008f00})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/interval/interval.go:160 +0x1c5
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/utils/interval.New in goroutine 438
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/interval/interval.go:105 +0x225
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159760 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc00243d8c0)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:329 +0x7c
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:344
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc001a3fee0, {0x55ebbeafe740?, 0xc001fb2a20}, 0xc002bf85a0?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1070 +0xc9
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x2?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc00243d680, 0xc001de5e60, 0xc001de5e50)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:776 +0x13a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc00243d680, {0x55ebbeafe740?, 0xc001fb2a20?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0xc0016a1820?, {0x55ebbeafe740?, 0xc001fb2a20?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc002096e80, {0x55ebbeafe740, 0xc001fb2a20})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).recv(0xc002c31800)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:95 +0x45
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 559
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:47 +0x1ab
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159316 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 581
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159620 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).forwardKeepAlives(0xc0029c4480)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:67 +0xb1
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 578
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:46 +0x16f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3148589 [select]:
Mar 11 11:37:04 hades teleport[2120659]: net.(*pipe).read(0xc002041780, {0xc002d22000, 0x3500, 0xc0028f8920?})
Mar 11 11:37:04 hades teleport[2120659]:         net/pipe.go:159 +0x155
Mar 11 11:37:04 hades teleport[2120659]: net.(*pipe).Read(0x2f?, {0xc002d22000?, 0xc002d22000?, 0x5?})
Mar 11 11:37:04 hades teleport[2120659]:         net/pipe.go:142 +0x1c
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/sshutils.(*ChConn).Read(0xc00251d830?, {0xc002d22000?, 0xc002b32300?, 0x18?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/sshutils/chconn.go:135 +0x25
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*atLeastReader).Read(0xc002b32300, {0xc002d22000?, 0xc002b32300?, 0x0?})
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:805 +0x3b
Mar 11 11:37:04 hades teleport[2120659]: bytes.(*Buffer).ReadFrom(0xc00120e2a8, {0x55ebbf6f9820, 0xc002b32300})
Mar 11 11:37:04 hades teleport[2120659]:         bytes/buffer.go:211 +0x98
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).readFromUntil(0xc00120e000, {0x7f6e643a12e8?, 0xc0000360c0}, 0x3500?)
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:827 +0xde
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).readRecordOrCCS(0xc00120e000, 0x0)
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:625 +0x250
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).readRecord(...)
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:587
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).Read(0xc00120e000, {0xc00273a000, 0x8000, 0x8b4000601?})
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:1369 +0x158
Mar 11 11:37:04 hades teleport[2120659]: bufio.(*Reader).Read(0xc0027284e0, {0xc002582820, 0x9, 0x7f6e66105878?})
Mar 11 11:37:04 hades teleport[2120659]:         bufio/bufio.go:244 +0x197
Mar 11 11:37:04 hades teleport[2120659]: io.ReadAtLeast({0x55ebbf6f3220, 0xc0027284e0}, {0xc002582820, 0x9, 0x9}, 0x9)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:335 +0x90
Mar 11 11:37:04 hades teleport[2120659]: io.ReadFull(...)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:354
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/net/http2.readFrameHeader({0xc002582820, 0x9, 0x55ebbea7f380?}, {0x55ebbf6f3220?, 0xc0027284e0?})
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/net@v0.21.0/http2/frame.go:237 +0x65
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/net/http2.(*Framer).ReadFrame(0xc0025827e0)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/net@v0.21.0/http2/frame.go:498 +0x85
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc002598240, 0xc0025c9880?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:1587 +0x245
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 3158876
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:398 +0x1ebe
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158892 [sync.Cond.Wait]:
Mar 11 11:37:04 hades teleport[2120659]: sync.runtime_notifyListWait(0xc002792310, 0x46fe)
Mar 11 11:37:04 hades teleport[2120659]:         runtime/sema.go:527 +0x15a
Mar 11 11:37:04 hades teleport[2120659]: sync.(*Cond).Wait(0x8000?)
Mar 11 11:37:04 hades teleport[2120659]:         sync/cond.go:70 +0x85
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*buffer).Read(0xc0023c9420, {0xc0024d8000, 0x8000, 0x8000})
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/buffer.go:94 +0x1fb
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*channel).ReadExtended(0xc0027fed80, {0xc0024d8000?, 0x8000?, 0x55ebbe013d60?}, 0x122b501?)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/channel.go:363 +0x93
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*channel).Read(0x0?, {0xc0024d8000?, 0xc00122b500?, 0xc002865020?})
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/channel.go:540 +0x25
Mar 11 11:37:04 hades teleport[2120659]: io.copyBuffer({0x7f6e5ef62048, 0xc00122b500}, {0x7f6e5ef62068, 0xc0027fed80}, {0x0, 0x0, 0x0})
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:430 +0x1a6
Mar 11 11:37:04 hades teleport[2120659]: io.Copy(...)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:389
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/sshutils.newChConn.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/sshutils/chconn.go:66 +0x7a
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/utils/sshutils.newChConn in goroutine 3158903
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/sshutils/chconn.go:65 +0x189
Mar 11 11:37:04 hades teleport[2120659]: goroutine 417 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv.(*Heartbeat).Run(0xc000c6d680)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/heartbeat.go:287 +0x1b0
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/srv/app.(*Server).startHeartbeat in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/server.go:475 +0x273
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158932 [select, 7 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/utils/interval.(*Interval).run(0xc0023d3140, {0x55ebbf78b300?, 0xc002494b40})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/interval/interval.go:160 +0x1c5
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/utils/interval.New in goroutine 165
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/interval/interval.go:105 +0x225
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158891 [sync.Cond.Wait, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: sync.runtime_notifyListWait(0xc002792290, 0x0)
Mar 11 11:37:04 hades teleport[2120659]:         runtime/sema.go:527 +0x15a
Mar 11 11:37:04 hades teleport[2120659]: sync.(*Cond).Wait(0x0?)
Mar 11 11:37:04 hades teleport[2120659]:         sync/cond.go:70 +0x85
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*mux).Wait(0xc000faa150)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/mux.go:110 +0x8e
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.NewClient.func1()
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/client.go:62 +0x28
Mar 11 11:37:04 hades teleport[2120659]: created by golang.org/x/crypto/ssh.NewClient in goroutine 3158903
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/client.go:61 +0x165
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158925 [select]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc001c9f680)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:1661 +0x151
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 3158903
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:388 +0x1e4b
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3148588 [select]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc002598240)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:1661 +0x151
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 3158876
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:388 +0x1e4b
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3148593 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*downstreamICS).runSendLoop(0xc002586ac0, {0x55ebbf7caa80, 0xc002584d20})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/inventory.go:322 +0xab
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.newDownstreamInventoryControlStream in goroutine 146
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/inventory.go:258 +0x196
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3148584 [chan receive]:
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*handshakeTransport).readPacket(0xc001a70340)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/handshake.go:209 +0x36
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*mux).onePacket(0xc000f5f500)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/mux.go:215 +0x2a
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*mux).loop(0xc000f5f500)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/mux.go:190 +0x25
Mar 11 11:37:04 hades teleport[2120659]: created by golang.org/x/crypto/ssh.newMux in goroutine 3158876
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/mux.go:128 +0x178
Mar 11 11:37:04 hades teleport[2120659]: goroutine 556 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/services.(*resourceWatcher).watch(0xc001c4f500)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/services/watcher.go:336 +0x7b0
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/services.(*resourceWatcher).runWatchLoop(0xc001c4f500)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/services/watcher.go:239 +0x77
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/services.newResourceWatcher in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/services/watcher.go:149 +0x39f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159318 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc00243c000)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:329 +0x7c
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:344
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc00223c8f0, {0x55ebbeafe740?, 0xc002f3f3b0}, 0x165c5a80165c5a0?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1070 +0xc9
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x55ebb53c1daf?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc002651d40, 0xc0030e9e60, 0xc0030e9e50)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:776 +0x13a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc002651d40, {0x55ebbeafe740?, 0xc002f3f3b0?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0xc001ffd280?, {0x55ebbeafe740?, 0xc002f3f3b0?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc001d638c0, {0x55ebbeafe740, 0xc002f3f3b0})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).recv(0xc0029c7680)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:95 +0x45
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 581
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:47 +0x1ab
Mar 11 11:37:04 hades teleport[2120659]: goroutine 530 [IO wait, 9272 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.runtime_pollWait(0x7f6e643e47a8, 0x72)
Mar 11 11:37:04 hades teleport[2120659]:         runtime/netpoll.go:343 +0x85
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.(*pollDesc).wait(0xc0028c5500?, 0xc0025a2d58?, 0x0)
Mar 11 11:37:04 hades teleport[2120659]:         internal/poll/fd_poll_runtime.go:84 +0x27
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.(*pollDesc).waitRead(...)
Mar 11 11:37:04 hades teleport[2120659]:         internal/poll/fd_poll_runtime.go:89
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.(*FD).Accept(0xc0028c5500)
Mar 11 11:37:04 hades teleport[2120659]:         internal/poll/fd_unix.go:611 +0x2ac
Mar 11 11:37:04 hades teleport[2120659]: net.(*netFD).accept(0xc0028c5500)
Mar 11 11:37:04 hades teleport[2120659]:         net/fd_unix.go:172 +0x29
Mar 11 11:37:04 hades teleport[2120659]: net.(*TCPListener).accept(0xc0028f8040)
Mar 11 11:37:04 hades teleport[2120659]:         net/tcpsock_posix.go:152 +0x1e
Mar 11 11:37:04 hades teleport[2120659]: net.(*TCPListener).Accept(0xc0028f8040)
Mar 11 11:37:04 hades teleport[2120659]:         net/tcpsock.go:315 +0x30
Mar 11 11:37:04 hades teleport[2120659]: net/http.(*Server).Serve(0xc002690690, {0x55ebbf789bb0, 0xc0028f8040})
Mar 11 11:37:04 hades teleport[2120659]:         net/http/server.go:3056 +0x364
Mar 11 11:37:04 hades teleport[2120659]: net/http/httptest.(*Server).goServe.func1()
Mar 11 11:37:04 hades teleport[2120659]:         net/http/httptest/server.go:310 +0x5b
Mar 11 11:37:04 hades teleport[2120659]: created by net/http/httptest.(*Server).goServe in goroutine 529
Mar 11 11:37:04 hades teleport[2120659]:         net/http/httptest/server.go:308 +0x65
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158832 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).forwardKeepAlives(0xc0029c7440)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:67 +0xb1
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 536
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:46 +0x16f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 532 [select, 9272 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).WaitForEvent(0xc001e20460, {0x55ebbf7afb00, 0x55ebc4503a00}, {0x55ebbbcd7328, 0xc})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:483 +0x3c5
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*TeleportProcess).OnExit.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/service.go:2249 +0x47
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalService).Serve(0xc0001dde30?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:587 +0x12
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:312 +0x2a2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve in goroutine 529
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:301 +0xa5
Mar 11 11:37:04 hades teleport[2120659]: goroutine 557 [select, 2070 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/events.(*AsyncEmitter).forward(0xc002539cc0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/events/emitter.go:94 +0x8b
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/events.NewAsyncEmitter in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/events/emitter.go:73 +0x18f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 558 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv/app.(*Server).expireSessions(0xc000fbf540)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/server.go:371 +0xde
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/srv/app.New in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/server.go:357 +0x8b9
Mar 11 11:37:04 hades teleport[2120659]: goroutine 559 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv.(*Heartbeat).Run(0xc0026467e0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/heartbeat.go:287 +0x1b0
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/srv/app.(*Server).startHeartbeat in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/server.go:475 +0x273
Mar 11 11:37:04 hades teleport[2120659]: goroutine 560 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv.(*Heartbeat).Run(0xc002646ea0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/heartbeat.go:287 +0x1b0
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/srv/app.(*Server).startHeartbeat in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/server.go:475 +0x273
Mar 11 11:37:04 hades teleport[2120659]: goroutine 561 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv.(*Heartbeat).Run(0xc0026478c0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/heartbeat.go:287 +0x1b0
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/srv/app.(*Server).startHeartbeat in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/server.go:475 +0x273
Mar 11 11:37:04 hades teleport[2120659]: goroutine 578 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv.(*Heartbeat).Run(0xc002647e60)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/heartbeat.go:287 +0x1b0
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/srv/app.(*Server).startHeartbeat in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/server.go:475 +0x273
Mar 11 11:37:04 hades teleport[2120659]: goroutine 579 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv.(*Heartbeat).Run(0xc0026505a0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/heartbeat.go:287 +0x1b0
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/srv/app.(*Server).startHeartbeat in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/server.go:475 +0x273
Mar 11 11:37:04 hades teleport[2120659]: goroutine 580 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv.(*Heartbeat).Run(0xc002650b40)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/heartbeat.go:287 +0x1b0
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/srv/app.(*Server).startHeartbeat in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/server.go:475 +0x273
Mar 11 11:37:04 hades teleport[2120659]: goroutine 581 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv.(*Heartbeat).Run(0xc0026510e0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/heartbeat.go:287 +0x1b0
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/srv/app.(*Server).startHeartbeat in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/server.go:475 +0x273
Mar 11 11:37:04 hades teleport[2120659]: goroutine 582 [select, 9272 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv/app.(*Server).startReconciler.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/watcher.go:50 +0xb2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/srv/app.(*Server).startReconciler in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/watcher.go:48 +0x2d9
Mar 11 11:37:04 hades teleport[2120659]: goroutine 583 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/utils.(*FnCache).get(0xc00159c180, {0x55ebbf7afcc0, 0xc000d35590}, {0x55ebbe013a20, 0x55ebbf6e67b0}, 0xb2d05e00, 0xc001a58c30)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/fncache.go:292 +0x465
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/utils.FnCacheGetWithTTL[...]({0x55ebbf7afcc0, 0xc000d35590}, 0xc000411080, {0x55ebbe013a20?, 0x55ebbf6e67b0}, 0x0, 0xc002b31480?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/fncache.go:212 +0xe5
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/utils.FnCacheGet[...](...)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/fncache.go:206
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/reversetunnelclient.CachingResolver.func1({0x55ebbf7afcc0, 0xc000d35590})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnelclient/resolver.go:56 +0x9d
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).newAgent(0xc0026604e0, {0x55ebbf7afcc0, 0xc000d35590}, 0xc000d355e0, 0xc001e85980)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:440 +0x58
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).connectAgent(0xc0026604e0, {0x55ebbf7afcc0, 0xc000d35590}, 0xc002547920?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:316 +0x123
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).run(0xc0026604e0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:270 +0x38
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:257 +0x25
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/reversetunnel.(*AgentPool).Start in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnel/agentpool.go:256 +0x138
Mar 11 11:37:04 hades teleport[2120659]: goroutine 494 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/utils/interval.(*Interval).run(0xc002458f00, {0x55ebbf78b300?, 0xc000c64960})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/interval/interval.go:160 +0x1c5
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/utils/interval.New in goroutine 165
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/interval/interval.go:105 +0x225
Mar 11 11:37:04 hades teleport[2120659]: goroutine 586 [select, 9272 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).WaitForEvent(0xc001e20460, {0x55ebbf7afb00, 0x55ebc4503a00}, {0x55ebbbcd7328, 0xc})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:483 +0x3c5
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*TeleportProcess).initApps.func1.(*TeleportProcess).OnExit.func8()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/service.go:2249 +0x5a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalService).Serve(0xc00160bea0?)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:587 +0x12
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:312 +0x2a2
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve in goroutine 164
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/service/supervisor.go:301 +0xa5
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159758 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 559
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3148586 [chan receive, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*Client).handleChannelOpens(0xc002588500, 0xc0027db680?)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/client.go:151 +0x45
Mar 11 11:37:04 hades teleport[2120659]: created by golang.org/x/crypto/ssh.NewClient in goroutine 3158876
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/client.go:60 +0x125
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159915 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).forwardKeepAlives(0xc0025556e0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:67 +0xb1
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 417
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:46 +0x16f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158833 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc002651c20)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:329 +0x7c
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:344
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc00223c820, {0x55ebbeafe740?, 0xc002879290}, 0xc001c82da0?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1070 +0xc9
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x2?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc0026519e0, 0xc00246fe60, 0xc00246fe50)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:776 +0x13a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc0026519e0, {0x55ebbeafe740?, 0xc002879290?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0xc001c82f58?, {0x55ebbeafe740?, 0xc002879290?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc001d63740, {0x55ebbeafe740, 0xc002879290})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).recv(0xc0029c7440)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:95 +0x45
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 536
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:47 +0x1ab
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159317 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).forwardKeepAlives(0xc0029c7680)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:67 +0xb1
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 581
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:46 +0x16f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3156079 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).forwardKeepAlives(0xc0026562a0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:67 +0xb1
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 579
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:46 +0x16f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159039 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc00297b9a0, {0xc002125ed0, 0x5, 0x5})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:192 +0x98
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00297b9a0, {0xc002125ed0?, 0xc001f34e28?, 0xc0022a1960?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:172 +0x12d
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc002125e90, {0xc002125ed0?, 0xc0022a19d8?, 0x55ebb4b8e765?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:520 +0x2c
Mar 11 11:37:04 hades teleport[2120659]: io.ReadAtLeast({0x55ebbf7069c0, 0xc002125e90}, {0xc002125ed0, 0x5, 0x5}, 0x5)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:335 +0x90
Mar 11 11:37:04 hades teleport[2120659]: io.ReadFull(...)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:354
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).Read(0xc002adbc20, {0xc002125ed0, 0x5, 0x5})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:504 +0x96
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*parser).recvMsg(0xc002125ec0, 0x400000)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:600 +0x46
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.recvAndDecompress(0xc0030e7bb8?, 0xc002adbc20, {0x0, 0x0}, 0x400000, 0x0, {0x0, 0x0})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:730 +0x5a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.recv(0xc002125ec0, {0x7f6e64451ef8, 0x55ebc4503a00}, 0x55ebb46edb18?, {0x0, 0x0}, {0x55ebbf589b00, 0xc00297fa80}, 0xc0033871e0?, 0x0, ...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:796 +0xa6
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc001211450, {0x55ebbf589b00?, 0xc00297fa80}, 0xc0023c7040?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1084 +0x289
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0000caa00?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc002adb9e0, 0xc0022a1e08, 0xc0022a1df8)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:760 +0x3ae
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc002adb9e0, {0x55ebbf589b00?, 0xc00297fa80?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0x40?, {0x55ebbf589b00?, 0xc00297fa80?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc00297f940, {0x55ebbf589b00, 0xc00297fa80})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client/proto.(*authServiceWatchEventsClient).Recv(0xc002961f60)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/proto/authservice.pb.go:17344 +0x46
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamWatcher).receiveEvents(0xc002f35a40)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/streamwatcher.go:84 +0x3b
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewWatcher in goroutine 556
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/streamwatcher.go:47 +0x1d6
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158934 [chan receive, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.DiscardRequests(0xc00285bf60?)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/connection.go:80 +0x2d
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/utils/sshutils.ConnectProxyTransport in goroutine 3158903
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/sshutils/conn.go:54 +0x14f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159621 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc001c0fc20)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:329 +0x7c
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:344
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc00223c000, {0x55ebbeafe740?, 0xc00257ff80}, 0xc002bf55a0?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1070 +0xc9
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x2?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc001c0f9e0, 0xc001a21e60, 0xc001a21e50)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:776 +0x13a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc001c0f9e0, {0x55ebbeafe740?, 0xc00257ff80?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0xc002bf5758?, {0x55ebbeafe740?, 0xc00257ff80?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc001a94880, {0x55ebbeafe740, 0xc00257ff80})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).recv(0xc0029c4480)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:95 +0x45
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 578
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:47 +0x1ab
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3148591 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 146
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159871 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 560
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158889 [chan receive, 3 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*Client).handleGlobalRequests(0x55ebb53b5b65?, 0xc002399b80?)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/client.go:142 +0x4c
Mar 11 11:37:04 hades teleport[2120659]: created by golang.org/x/crypto/ssh.NewClient in goroutine 3158903
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/client.go:59 +0xd3
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158982 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 358
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159914 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 417
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158926 [select]:
Mar 11 11:37:04 hades teleport[2120659]: net.(*pipe).read(0xc00122b400, {0xc002b0a000, 0x3500, 0xc0027fbd80?})
Mar 11 11:37:04 hades teleport[2120659]:         net/pipe.go:159 +0x155
Mar 11 11:37:04 hades teleport[2120659]: net.(*pipe).Read(0x22?, {0xc002b0a000?, 0xc002b0a000?, 0x5?})
Mar 11 11:37:04 hades teleport[2120659]:         net/pipe.go:142 +0x1c
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/sshutils.(*ChConn).Read(0xc001a22830?, {0xc002b0a000?, 0xc001a58330?, 0x18?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/sshutils/chconn.go:135 +0x25
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*atLeastReader).Read(0xc001a58330, {0xc002b0a000?, 0xc001a58330?, 0x0?})
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:805 +0x3b
Mar 11 11:37:04 hades teleport[2120659]: bytes.(*Buffer).ReadFrom(0xc00204b0a8, {0x55ebbf6f9820, 0xc001a58330})
Mar 11 11:37:04 hades teleport[2120659]:         bytes/buffer.go:211 +0x98
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).readFromUntil(0xc00204ae00, {0x7f6e643a12e8?, 0xc0024337a0}, 0x3500?)
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:827 +0xde
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).readRecordOrCCS(0xc00204ae00, 0x0)
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:625 +0x250
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).readRecord(...)
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:587
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).Read(0xc00204ae00, {0xc002900000, 0x8000, 0x8b4010601?})
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:1369 +0x158
Mar 11 11:37:04 hades teleport[2120659]: bufio.(*Reader).Read(0xc00257ca20, {0xc001cb4d60, 0x9, 0x7f6e661045b8?})
Mar 11 11:37:04 hades teleport[2120659]:         bufio/bufio.go:244 +0x197
Mar 11 11:37:04 hades teleport[2120659]: io.ReadAtLeast({0x55ebbf6f3220, 0xc00257ca20}, {0xc001cb4d60, 0x9, 0x9}, 0x9)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:335 +0x90
Mar 11 11:37:04 hades teleport[2120659]: io.ReadFull(...)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:354
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/net/http2.readFrameHeader({0xc001cb4d60, 0x9, 0x55ebc44c2660?}, {0x55ebbf6f3220?, 0xc00257ca20?})
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/net@v0.21.0/http2/frame.go:237 +0x65
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/net/http2.(*Framer).ReadFrame(0xc001cb4d20)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/net@v0.21.0/http2/frame.go:498 +0x85
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001c9f680, 0x0?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:1587 +0x245
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 3158903
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:398 +0x1ebe
Mar 11 11:37:04 hades teleport[2120659]: goroutine 5140461 [chan receive, 2514 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/srv/app.(*Server).createTracker.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/session.go:355 +0x45
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/srv/app.(*Server).createTracker in goroutine 5140460
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/srv/app/session.go:354 +0x58b
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158983 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc002495900, {0xc00265b360, 0x5, 0x5})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:192 +0x98
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002495900, {0xc00265b360?, 0xc0024c43d8?, 0xc0010ab960?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:172 +0x12d
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00265b320, {0xc00265b360?, 0xc0010ab9d8?, 0x55ebb4b8e765?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:520 +0x2c
Mar 11 11:37:04 hades teleport[2120659]: io.ReadAtLeast({0x55ebbf7069c0, 0xc00265b320}, {0xc00265b360, 0x5, 0x5}, 0x5)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:335 +0x90
Mar 11 11:37:04 hades teleport[2120659]: io.ReadFull(...)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:354
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000c6db00, {0xc00265b360, 0x5, 0x5})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:504 +0x96
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*parser).recvMsg(0xc00265b350, 0x400000)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:600 +0x46
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.recvAndDecompress(0xc0000f8bb8?, 0xc000c6db00, {0x0, 0x0}, 0x400000, 0x0, {0x0, 0x0})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:730 +0x5a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.recv(0xc00265b350, {0x7f6e64451ef8, 0x55ebc4503a00}, 0x0?, {0x0, 0x0}, {0x55ebbf589b00, 0xc0025877c0}, 0x108?, 0x0, ...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:796 +0xa6
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc0016815f0, {0x55ebbf589b00?, 0xc0025877c0}, 0xc0022068c0?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1084 +0x289
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001f74688?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc000c6d8c0, 0xc0010abe08, 0xc0010abdf8)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:760 +0x3ae
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc000c6d8c0, {0x55ebbf589b00?, 0xc0025877c0?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0x40?, {0x55ebbf589b00?, 0xc0025877c0?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc0028c0600, {0x55ebbf589b00, 0xc0025877c0})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client/proto.(*authServiceWatchEventsClient).Recv(0xc0024cee70)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/proto/authservice.pb.go:17344 +0x46
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamWatcher).receiveEvents(0xc002300f60)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/streamwatcher.go:84 +0x3b
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewWatcher in goroutine 358
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/streamwatcher.go:47 +0x1d6
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158888 [chan receive]:
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*handshakeTransport).readPacket(0xc00196cd00)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/handshake.go:209 +0x36
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*mux).onePacket(0xc000faa150)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/mux.go:215 +0x2a
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*mux).loop(0xc000faa150)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/mux.go:190 +0x25
Mar 11 11:37:04 hades teleport[2120659]: created by golang.org/x/crypto/ssh.newMux in goroutine 3158903
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/mux.go:128 +0x178
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159872 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).forwardKeepAlives(0xc002554a20)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:67 +0xb1
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 560
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:46 +0x16f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 7291154 [select]:
Mar 11 11:37:04 hades teleport[2120659]: net/http.setRequestCancel.func4()
Mar 11 11:37:04 hades teleport[2120659]:         net/http/client.go:406 +0x7f
Mar 11 11:37:04 hades teleport[2120659]: created by net/http.setRequestCancel in goroutine 7291121
Mar 11 11:37:04 hades teleport[2120659]:         net/http/client.go:405 +0x3e5
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158938 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 430
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3148583 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*handshakeTransport).kexLoop(0xc001a70340)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/handshake.go:293 +0x53a
Mar 11 11:37:04 hades teleport[2120659]: created by golang.org/x/crypto/ssh.newClientTransport in goroutine 3158876
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/handshake.go:156 +0x25b
Mar 11 11:37:04 hades teleport[2120659]: goroutine 7291121 [select]:
Mar 11 11:37:04 hades teleport[2120659]: net/http.(*persistConn).roundTrip(0xc0034fb200, 0xc002721940)
Mar 11 11:37:04 hades teleport[2120659]:         net/http/transport.go:2652 +0x979
Mar 11 11:37:04 hades teleport[2120659]: net/http.(*Transport).roundTrip(0xc0032d0780, 0xc0032f9200)
Mar 11 11:37:04 hades teleport[2120659]:         net/http/transport.go:604 +0x79a
Mar 11 11:37:04 hades teleport[2120659]: net/http.(*Transport).RoundTrip(...)
Mar 11 11:37:04 hades teleport[2120659]:         net/http/roundtrip.go:17
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils.(*HTTPRoundTripper).RoundTrip(0xc001a58c60, 0xc0032f9200)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/proxy.go:118 +0x1e6
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Transport).RoundTrip(0xc001ea21c0, 0xc0032f9100)
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.46.1/transport.go:116 +0x52b
Mar 11 11:37:04 hades teleport[2120659]: net/http.send(0xc0032f9000, {0x55ebbf702080, 0xc0017db5a0}, {0x1?, 0x55ebb4706874?, 0x55ebc44c2660?})
Mar 11 11:37:04 hades teleport[2120659]:         net/http/client.go:260 +0x606
Mar 11 11:37:04 hades teleport[2120659]: net/http.(*Client).send(0xc001e85bc0, 0xc0032f9000, {0x55ebbf479260?, 0xc002382cf0?, 0x55ebc44c2660?})
Mar 11 11:37:04 hades teleport[2120659]:         net/http/client.go:181 +0x98
Mar 11 11:37:04 hades teleport[2120659]: net/http.(*Client).do(0xc001e85bc0, 0xc0032f9000)
Mar 11 11:37:04 hades teleport[2120659]:         net/http/client.go:724 +0x912
Mar 11 11:37:04 hades teleport[2120659]: net/http.(*Client).Do(...)
Mar 11 11:37:04 hades teleport[2120659]:         net/http/client.go:590
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client/webclient.doWithFallback(0x55ebbf7afc88?, 0x0, 0x55ebbbc260b5?, 0xc0032f9000)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/webclient/webclient.go:131 +0x314
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client/webclient.Find(0xc00158d080)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/webclient/webclient.go:177 +0x1f7
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/service.NewTeleport.WebClientResolver.func5({0x30?, 0x55ebbef7d8c0?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnelclient/resolver.go:82 +0x1b
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/reversetunnelclient.CachingResolver.func1.1({0x55ebbf7afc88?, 0xc001e85b90?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/reversetunnelclient/resolver.go:57 +0x22
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/utils.FnCacheGetWithTTL[...].func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/fncache.go:213 +0x2d
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/utils.(*FnCache).get.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/fncache.go:283 +0x7a
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/utils.(*FnCache).get in goroutine 583
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/utils/fncache.go:279 +0x390
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158930 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 165
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158902 [chan receive, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.DiscardRequests(0x0?)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/connection.go:80 +0x2d
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/utils/sshutils.ConnectProxyTransport in goroutine 3158876
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/sshutils/conn.go:54 +0x14f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159038 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 556
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159825 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).forwardKeepAlives(0xc0021b7c80)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:67 +0xb1
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 580
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:46 +0x16f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159916 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc001594a20)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:329 +0x7c
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:344
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc001597ba0, {0x55ebbeafe740?, 0xc001e84930}, 0xc002f65da0?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1070 +0xc9
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x2?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc0015947e0, 0xc00305de60, 0xc00305de50)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:776 +0x13a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc0015947e0, {0x55ebbeafe740?, 0xc001e84930?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0xc000b76000?, {0x55ebbeafe740?, 0xc001e84930?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc0030271c0, {0x55ebbeafe740, 0xc001e84930})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).recv(0xc0025556e0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:95 +0x45
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 417
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:47 +0x1ab
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159824 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 580
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159519 [select]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).forwardKeepAlives(0xc0021d8cc0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:67 +0xb1
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 563
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:46 +0x16f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158886 [IO wait]:
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.runtime_pollWait(0x7f6e643e4d78, 0x72)
Mar 11 11:37:04 hades teleport[2120659]:         runtime/netpoll.go:343 +0x85
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.(*pollDesc).wait(0xc002041980?, 0xc001eb6a00?, 0x0)
Mar 11 11:37:04 hades teleport[2120659]:         internal/poll/fd_poll_runtime.go:84 +0x27
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.(*pollDesc).waitRead(...)
Mar 11 11:37:04 hades teleport[2120659]:         internal/poll/fd_poll_runtime.go:89
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.(*FD).Read(0xc002041980, {0xc001eb6a00, 0x3500, 0x3500})
Mar 11 11:37:04 hades teleport[2120659]:         internal/poll/fd_unix.go:164 +0x27a
Mar 11 11:37:04 hades teleport[2120659]: net.(*netFD).Read(0xc002041980, {0xc001eb6a00?, 0xc001eb6a05?, 0x65?})
Mar 11 11:37:04 hades teleport[2120659]:         net/fd_posix.go:55 +0x25
Mar 11 11:37:04 hades teleport[2120659]: net.(*conn).Read(0xc001b63b88, {0xc001eb6a00?, 0x155?, 0xc000fcefb8?})
Mar 11 11:37:04 hades teleport[2120659]:         net/net.go:179 +0x45
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*atLeastReader).Read(0xc001a58318, {0xc001eb6a00?, 0xc001a58318?, 0x0?})
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:805 +0x3b
Mar 11 11:37:04 hades teleport[2120659]: bytes.(*Buffer).ReadFrom(0xc000fcf0a8, {0x55ebbf6f9820, 0xc001a58318})
Mar 11 11:37:04 hades teleport[2120659]:         bytes/buffer.go:211 +0x98
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).readFromUntil(0xc000fcee00, {0x55ebbf6f6840?, 0xc001b63b88}, 0x3500?)
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:827 +0xde
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).readRecordOrCCS(0xc000fcee00, 0x0)
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:625 +0x250
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).readRecord(...)
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:587
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).Read(0xc000fcee00, {0xc002798000, 0x1000, 0x2e50?})
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:1369 +0x158
Mar 11 11:37:04 hades teleport[2120659]: bufio.(*Reader).Read(0xc0024332c0, {0xc002494dd0, 0x4, 0xc001e35000?})
Mar 11 11:37:04 hades teleport[2120659]:         bufio/bufio.go:244 +0x197
Mar 11 11:37:04 hades teleport[2120659]: io.ReadAtLeast({0x55ebbf6f3220, 0xc0024332c0}, {0xc002494dd0, 0x4, 0x4}, 0x4)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:335 +0x90
Mar 11 11:37:04 hades teleport[2120659]: io.ReadFull(...)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:354
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*gcmCipher).readCipherPacket(0xc002494dc0, 0xbded9680?, {0x55ebbf6f3220, 0xc0024332c0})
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/cipher.go:373 +0x56
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*connectionState).readPacket(0xc00222aea0, 0x55ebbe0fef00?, 0x1)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/transport.go:147 +0x42
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*transport).readPacket(0xc00222aea0)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/transport.go:130 +0x2b
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*handshakeTransport).readOnePacket(0xc00196cd00, 0x0)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/handshake.go:400 +0x37
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*handshakeTransport).readLoop(0xc00196cd00)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/handshake.go:219 +0x2a
Mar 11 11:37:04 hades teleport[2120659]: created by golang.org/x/crypto/ssh.newClientTransport in goroutine 3158903
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/handshake.go:155 +0x21f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3156078 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 579
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 7291147 [IO wait]:
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.runtime_pollWait(0x7f6e643a4b48, 0x72)
Mar 11 11:37:04 hades teleport[2120659]:         runtime/netpoll.go:343 +0x85
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.(*pollDesc).wait(0xc001fa8200?, 0xc003489800?, 0x0)
Mar 11 11:37:04 hades teleport[2120659]:         internal/poll/fd_poll_runtime.go:84 +0x27
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.(*pollDesc).waitRead(...)
Mar 11 11:37:04 hades teleport[2120659]:         internal/poll/fd_poll_runtime.go:89
Mar 11 11:37:04 hades teleport[2120659]: internal/poll.(*FD).Read(0xc001fa8200, {0xc003489800, 0x1800, 0x1800})
Mar 11 11:37:04 hades teleport[2120659]:         internal/poll/fd_unix.go:164 +0x27a
Mar 11 11:37:04 hades teleport[2120659]: net.(*netFD).Read(0xc001fa8200, {0xc003489800?, 0xc001fa8200?, 0xc002d42f00?})
Mar 11 11:37:04 hades teleport[2120659]:         net/fd_posix.go:55 +0x25
Mar 11 11:37:04 hades teleport[2120659]: net.(*conn).Read(0xc00169b1e0, {0xc003489800?, 0x10f?, 0x10?})
Mar 11 11:37:04 hades teleport[2120659]:         net/net.go:179 +0x45
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*atLeastReader).Read(0xc002b32b10, {0xc003489800?, 0xc002b32b10?, 0x0?})
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:805 +0x3b
Mar 11 11:37:04 hades teleport[2120659]: bytes.(*Buffer).ReadFrom(0xc00120f7a8, {0x55ebbf6f9820, 0xc002b32b10})
Mar 11 11:37:04 hades teleport[2120659]:         bytes/buffer.go:211 +0x98
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).readFromUntil(0xc00120f500, {0x55ebbf6f6840?, 0xc00169b1e0}, 0xc0030618c0?)
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:827 +0xde
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).readRecordOrCCS(0xc00120f500, 0x0)
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:625 +0x250
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).readRecord(...)
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:587
Mar 11 11:37:04 hades teleport[2120659]: crypto/tls.(*Conn).Read(0xc00120f500, {0xc0034c5000, 0x1000, 0x55ebbf154a20?})
Mar 11 11:37:04 hades teleport[2120659]:         crypto/tls/conn.go:1369 +0x158
Mar 11 11:37:04 hades teleport[2120659]: net/http.(*persistConn).Read(0xc0034fb200, {0xc0034c5000?, 0x55ebb4677525?, 0x60?})
Mar 11 11:37:04 hades teleport[2120659]:         net/http/transport.go:1954 +0x4a
Mar 11 11:37:04 hades teleport[2120659]: bufio.(*Reader).fill(0xc0039a12c0)
Mar 11 11:37:04 hades teleport[2120659]:         bufio/bufio.go:113 +0x103
Mar 11 11:37:04 hades teleport[2120659]: bufio.(*Reader).Peek(0xc0039a12c0, 0x1)
Mar 11 11:37:04 hades teleport[2120659]:         bufio/bufio.go:151 +0x53
Mar 11 11:37:04 hades teleport[2120659]: net/http.(*persistConn).readLoop(0xc0034fb200)
Mar 11 11:37:04 hades teleport[2120659]:         net/http/transport.go:2118 +0x1b9
Mar 11 11:37:04 hades teleport[2120659]: created by net/http.(*Transport).dialConn in goroutine 7291155
Mar 11 11:37:04 hades teleport[2120659]:         net/http/transport.go:1776 +0x169f
Mar 11 11:37:04 hades teleport[2120659]: goroutine 7291148 [select]:
Mar 11 11:37:04 hades teleport[2120659]: net/http.(*persistConn).writeLoop(0xc0034fb200)
Mar 11 11:37:04 hades teleport[2120659]:         net/http/transport.go:2421 +0xe5
Mar 11 11:37:04 hades teleport[2120659]: created by net/http.(*Transport).dialConn in goroutine 7291155
Mar 11 11:37:04 hades teleport[2120659]:         net/http/transport.go:1777 +0x16f1
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159069 [select, 3 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc002b1db80, {0xc002bd17b0, 0x5, 0x5})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:192 +0x98
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002b1db80, {0xc002bd17b0?, 0xc002a66e40?, 0xc0010a9960?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:172 +0x12d
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc002bd1770, {0xc002bd17b0?, 0xc0010a99d8?, 0x55ebb4b8e765?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:520 +0x2c
Mar 11 11:37:04 hades teleport[2120659]: io.ReadAtLeast({0x55ebbf7069c0, 0xc002bd1770}, {0xc002bd17b0, 0x5, 0x5}, 0x5)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:335 +0x90
Mar 11 11:37:04 hades teleport[2120659]: io.ReadFull(...)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:354
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).Read(0xc002bd2900, {0xc002bd17b0, 0x5, 0x5})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:504 +0x96
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*parser).recvMsg(0xc002bd17a0, 0x400000)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:600 +0x46
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.recvAndDecompress(0xc0030e3bb8?, 0xc002bd2900, {0x0, 0x0}, 0x400000, 0x0, {0x0, 0x0})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:730 +0x5a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.recv(0xc002bd17a0, {0x7f6e64451ef8, 0x55ebc4503a00}, 0xc002c55438?, {0x0, 0x0}, {0x55ebbf589b00, 0xc00263ea40}, 0x55ebb4687c48?, 0x0, ...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:796 +0xa6
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc00047c5b0, {0x55ebbf589b00?, 0xc00263ea40}, 0xc00209b520?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1084 +0x289
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x0?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc002bd26c0, 0xc0010a9e08, 0xc0010a9df8)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:760 +0x3ae
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc002bd26c0, {0x55ebbf589b00?, 0xc00263ea40?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0x40?, {0x55ebbf589b00?, 0xc00263ea40?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc002b5cec0, {0x55ebbf589b00, 0xc00263ea40})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client/proto.(*authServiceWatchEventsClient).Recv(0xc002be4380)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/proto/authservice.pb.go:17344 +0x46
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamWatcher).receiveEvents(0xc0015ff2c0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/streamwatcher.go:84 +0x3b
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewWatcher in goroutine 405
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/streamwatcher.go:47 +0x1d6
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158890 [chan receive, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*Client).handleChannelOpens(0xc0023a7e00, 0xc002494640?)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/client.go:151 +0x45
Mar 11 11:37:04 hades teleport[2120659]: created by golang.org/x/crypto/ssh.NewClient in goroutine 3158903
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/client.go:60 +0x125
Mar 11 11:37:04 hades teleport[2120659]: goroutine 5140463 [select, 2506 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/events.(*SessionWriter).processEvents(0xc000d3ea20)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/events/session_writer.go:429 +0x231
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/events.NewSessionWriter.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/events/session_writer.go:62 +0x1c
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/events.NewSessionWriter in goroutine 5140460
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/events/session_writer.go:61 +0x336
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159520 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc001d8a5a0)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:329 +0x7c
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:344
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc002911110, {0x55ebbeafe740?, 0xc001f00cc0}, 0xc001939da0?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1070 +0xc9
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x2?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc002c33e60, 0xc00291de60, 0xc00291de50)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:776 +0x13a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc002c33e60, {0x55ebbeafe740?, 0xc001f00cc0?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0xc00302cba0?, {0x55ebbeafe740?, 0xc001f00cc0?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc000fbb780, {0x55ebbeafe740, 0xc001f00cc0})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).recv(0xc0021d8cc0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:95 +0x45
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 563
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:47 +0x1ab
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158887 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*handshakeTransport).kexLoop(0xc00196cd00)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/handshake.go:293 +0x53a
Mar 11 11:37:04 hades teleport[2120659]: created by golang.org/x/crypto/ssh.newClientTransport in goroutine 3158903
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/handshake.go:156 +0x25b
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158931 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc002494aa0, {0xc002454880, 0x5, 0x5})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:192 +0x98
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002494aa0, {0xc002454880?, 0xc0024c4030?, 0xc00229d960?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:172 +0x12d
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc002454840, {0xc002454880?, 0xc00229d9d8?, 0x55ebb4b8e765?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:520 +0x2c
Mar 11 11:37:04 hades teleport[2120659]: io.ReadAtLeast({0x55ebbf7069c0, 0xc002454840}, {0xc002454880, 0x5, 0x5}, 0x5)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:335 +0x90
Mar 11 11:37:04 hades teleport[2120659]: io.ReadFull(...)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:354
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0022aeb40, {0xc002454880, 0x5, 0x5})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:504 +0x96
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*parser).recvMsg(0xc002454870, 0x400000)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:600 +0x46
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.recvAndDecompress(0xc0022e1bb8?, 0xc0022aeb40, {0x0, 0x0}, 0x400000, 0x0, {0x0, 0x0})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:730 +0x5a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.recv(0xc002454870, {0x7f6e64451ef8, 0x55ebc4503a00}, 0x0?, {0x0, 0x0}, {0x55ebbf589b00, 0xc0022f7800}, 0x55ebb4687c48?, 0x0, ...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/rpc_util.go:796 +0xa6
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc0016800d0, {0x55ebbf589b00?, 0xc0022f7800}, 0xc001f6b860?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1084 +0x289
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0000d1900?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc0022ae900, 0xc00229de08, 0xc00229ddf8)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:760 +0x3ae
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc0022ae900, {0x55ebbf589b00?, 0xc0022f7800?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0x40?, {0x55ebbf589b00?, 0xc0022f7800?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc0022f7700, {0x55ebbf589b00, 0xc0022f7800})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client/proto.(*authServiceWatchEventsClient).Recv(0xc002ddde60)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/proto/authservice.pb.go:17344 +0x46
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamWatcher).receiveEvents(0xc0023d3080)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/streamwatcher.go:84 +0x3b
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewWatcher in goroutine 165
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/streamwatcher.go:47 +0x1d6
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159873 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc001594360)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:329 +0x7c
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/internal/transport/transport.go:344
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*csAttempt).recvMsg(0xc001597a00, {0x55ebbeafe740?, 0xc00287ef30}, 0xc00165ada0?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:1070 +0xc9
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x2?)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).withRetry(0xc002d13d40, 0xc0022e2e60, 0xc0022e2e50)
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:776 +0x13a
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.(*clientStream).RecvMsg(0xc002d13d40, {0x55ebbeafe740?, 0xc00287ef30?})
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/grpc/interceptors.(*grpcClientStreamWrapper).RecvMsg(0xc002bed460?, {0x55ebbeafe740?, 0xc00287ef30?})
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/grpc/interceptors/errors.go:58 +0x25
Mar 11 11:37:04 hades teleport[2120659]: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.(*clientStream).RecvMsg(0xc003026680, {0x55ebbeafe740, 0xc00287ef30})
Mar 11 11:37:04 hades teleport[2120659]:         go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.46.1/interceptor.go:146 +0x3a
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/client.(*streamKeepAliver).recv(0xc002554a20)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:95 +0x45
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/client.(*Client).NewKeepAliver in goroutine 560
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/client/keepaliver.go:47 +0x1ab
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159068 [select, 5119 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 405
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3158919 [sync.Cond.Wait]:
Mar 11 11:37:04 hades teleport[2120659]: sync.runtime_notifyListWait(0xc002586610, 0xa572)
Mar 11 11:37:04 hades teleport[2120659]:         runtime/sema.go:527 +0x15a
Mar 11 11:37:04 hades teleport[2120659]: sync.(*Cond).Wait(0x8000?)
Mar 11 11:37:04 hades teleport[2120659]:         sync/cond.go:70 +0x85
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*buffer).Read(0xc0028f8820, {0xc00249a000, 0x8000, 0x8000})
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/buffer.go:94 +0x1fb
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*channel).ReadExtended(0xc001c4e3c0, {0xc00249a000?, 0x8000?, 0x55ebbe013d60?}, 0x2041801?)
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/channel.go:363 +0x93
Mar 11 11:37:04 hades teleport[2120659]: golang.org/x/crypto/ssh.(*channel).Read(0x55ebb4677ba0?, {0xc00249a000?, 0xc002041800?, 0xc002bda740?})
Mar 11 11:37:04 hades teleport[2120659]:         golang.org/x/crypto@v0.19.0/ssh/channel.go:540 +0x25
Mar 11 11:37:04 hades teleport[2120659]: io.copyBuffer({0x7f6e5ef62048, 0xc002041800}, {0x7f6e5ef62068, 0xc001c4e3c0}, {0x0, 0x0, 0x0})
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:430 +0x1a6
Mar 11 11:37:04 hades teleport[2120659]: io.Copy(...)
Mar 11 11:37:04 hades teleport[2120659]:         io/io.go:389
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/api/utils/sshutils.newChConn.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/sshutils/chconn.go:66 +0x7a
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/api/utils/sshutils.newChConn in goroutine 3158876
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/api@v0.0.0/utils/sshutils/chconn.go:65 +0x189
Mar 11 11:37:04 hades teleport[2120659]: goroutine 3159619 [select, 5117 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: google.golang.org/grpc.newClientStreamWithParams.func4()
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
Mar 11 11:37:04 hades teleport[2120659]: created by google.golang.org/grpc.newClientStreamWithParams in goroutine 578
Mar 11 11:37:04 hades teleport[2120659]:         google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c
Mar 11 11:37:04 hades teleport[2120659]: goroutine 5140462 [select, 2506 minutes]:
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/events.(*sliceWriter).receiveAndUpload(0xc0038ae7e0)
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/events/stream.go:523 +0x1f2
Mar 11 11:37:04 hades teleport[2120659]: github.com/gravitational/teleport/lib/events.NewProtoStream.func1()
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/events/stream.go:317 +0x25
Mar 11 11:37:04 hades teleport[2120659]: created by github.com/gravitational/teleport/lib/events.NewProtoStream in goroutine 5140460
Mar 11 11:37:04 hades teleport[2120659]:         github.com/gravitational/teleport/lib/events/stream.go:316 +0x447