search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.37k stars 1.74k forks source link

Changing a cluster's `proxy_service.public_addr` with no `https_keypairs` results in rotation state sync errors #43856

Closed webvictim closed 2 weeks ago

webvictim commented 2 months ago

Expected behavior

Changing the public_addr of a cluster's proxy_service should result in new self-signed certificates being generated for the Proxy without error.

Current behavior:

Changing proxy_service.public_addr causes a rotation state error to repeat ad infinitum:

2024-07-04T14:37:38Z DEBU [PROC:1]    Service has started. pid:7.1 service:proxy.shutdown service/supervisor.go:312
2024-07-04T14:37:38Z DEBU [PROXY:PRO] Not initializing Kube Cluster resource watcher. pid:7.1 proxy/watcher.go:38
2024-07-04T14:37:38Z INFO [PROC:1]    Rotation in progress, updating SSH principals. pid:7.1 identity:Proxy additional_principals:[teleport-local bastion.teleport.com 0.0.0.0 localhost 127.0.0.1 ::1 remote.kube.proxy.teleport.cluster.local] current_principals:[164cbed7-2832-472f-9711-6d868a02c555.127.0.0.1.nip.io 164cbed7-2832-472f-9711-6d868a02c555 teleport-local.127.0.0.1.nip.io teleport-local localhost 127.0.0.1 ::1 127.0.0.1.nip.io remote.kube.proxy.teleport.cluster.local] service/connect.go:903
2024-07-04T14:37:38Z DEBU [PROXY:PRO] Not initializing Kube Cluster resource watcher. pid:7.1 proxy/watcher.go:92
2024-07-04T14:37:38Z INFO [PROC:1]    Rotation in progress, updating DNS names. pid:7.1 identity:Proxy additional_dns_names:[*.teleport.cluster.local teleport.cluster.local *.bastion.teleport.com] current_dns_names:[teleport-local 127.0.0.1.nip.io localhost remote.kube.proxy.teleport.cluster.local *.teleport.cluster.local teleport.cluster.local *.127.0.0.1.nip.io] service/connect.go:907
2024-07-04T14:37:38Z DEBU [PROC:1]    Service has started. pid:7.1 service:proxy.tls.alpn.sni.proxy service/supervisor.go:312
2024-07-04T14:37:38Z INFO [PROC:1]    Service has updated principals and DNS Names, going to request new principals and update. pid:7.1 identity:Proxy principals:[teleport-local bastion.teleport.com 0.0.0.0 localhost 127.0.0.1 ::1 remote.kube.proxy.teleport.cluster.local] dns_names:[*.teleport.cluster.local teleport.cluster.local *.bastion.teleport.com] service/connect.go:998
2024-07-04T14:37:38Z DEBU [PROC:1]    Generating new key pair. pid:7.1 identity:Proxy reason:re-register service/connect.go:344
2024-07-04T14:37:38Z INFO [PROXY:SER] Starting TLS ALPN SNI proxy server on. pid:7.1 listen_address:[::]:443 service/service.go:4943
2024-07-04T14:37:38Z DEBU [MX:PROXY:] Starting serving MUX, ID "proxy:proxy:kube" on address [::]:443 multiplexer/multiplexer.go:260
2024-07-04T14:37:38Z DEBU [PROC:1]    Service is completed and removed. pid:7.1 service:update.aws-oidc.deploy.service service/supervisor.go:262
2024-07-04T14:37:38Z DEBU [PROC:1]    Service has started. pid:7.1 service:closer service/supervisor.go:312
2024-07-04T14:37:38Z WARN [PROC:1]    Failed to sync rotation state. pid:7.1 error:[
ERROR REPORT:
Original Error: *trace.BadParameterError missing parameter hostname
Stack Trace:
    github.com/gravitational/teleport/lib/utils/utils.go:286 github.com/gravitational/teleport/lib/utils.Host
    github.com/gravitational/teleport/lib/auth/auth.go:4242 github.com/gravitational/teleport/lib/auth.(*Server).GenerateHostCerts
    github.com/gravitational/teleport/lib/auth/register.go:122 github.com/gravitational/teleport/lib/auth.ReRegister
    github.com/gravitational/teleport/lib/service/connect.go:412 github.com/gravitational/teleport/lib/service.(*TeleportProcess).reRegister
    github.com/gravitational/teleport/lib/service/connect.go:999 github.com/gravitational/teleport/lib/service.(*TeleportProcess).rotate
    github.com/gravitational/teleport/lib/service/connect.go:869 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncServiceRotationState
    github.com/gravitational/teleport/lib/service/connect.go:848 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncRotationState
    github.com/gravitational/teleport/lib/service/connect.go:811 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncRotationStateAndBroadcast
    github.com/gravitational/teleport/lib/service/connect.go:743 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncRotationStateCycle
    github.com/gravitational/teleport/lib/service/connect.go:707 github.com/gravitational/teleport/lib/service.(*TeleportProcess).periodicSyncRotationState
    github.com/gravitational/teleport/lib/service/supervisor.go:588 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
    github.com/gravitational/teleport/lib/service/supervisor.go:313 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
    runtime/asm_arm64.s:1222 runtime.goexit
User Message: missing parameter hostname] service/connect.go:816
2024-07-04T14:37:38Z WARN [PROC:1]    Sync rotation state cycle failed. pid:7.1 retry_interval:10s service/connect.go:712
2024-07-04T14:37:38Z DEBU [PROC:1]    Teleport component has started. pid:7.1 component:proxy service/state.go:118
2024-07-04T14:37:39Z DEBU [PROXY:1]   List of known proxies updated: ["164cbed7-2832-472f-9711-6d868a02c555"]. pid:7.1 services/watcher.go:534
2024-07-04T14:37:41Z INFO [PROC:1]    Initiating new sync rotation state cycle after backoff. pid:7.1 backoff_time:3.483435535s service/connect.go:717
2024-07-04T14:37:41Z INFO [PROC:1]    Rotation in progress, updating SSH principals. pid:7.1 identity:Proxy additional_principals:[teleport-local bastion.teleport.com 0.0.0.0 localhost 127.0.0.1 ::1 remote.kube.proxy.teleport.cluster.local] current_principals:[164cbed7-2832-472f-9711-6d868a02c555.127.0.0.1.nip.io 164cbed7-2832-472f-9711-6d868a02c555 teleport-local.127.0.0.1.nip.io teleport-local localhost 127.0.0.1 ::1 127.0.0.1.nip.io remote.kube.proxy.teleport.cluster.local] service/connect.go:903
2024-07-04T14:37:41Z INFO [PROC:1]    Rotation in progress, updating DNS names. pid:7.1 identity:Proxy additional_dns_names:[*.teleport.cluster.local teleport.cluster.local *.bastion.teleport.com] current_dns_names:[teleport-local 127.0.0.1.nip.io localhost remote.kube.proxy.teleport.cluster.local *.teleport.cluster.local teleport.cluster.local *.127.0.0.1.nip.io] service/connect.go:907
2024-07-04T14:37:41Z INFO [PROC:1]    Service has updated principals and DNS Names, going to request new principals and update. pid:7.1 identity:Proxy principals:[teleport-local bastion.teleport.com 0.0.0.0 localhost 127.0.0.1 ::1 remote.kube.proxy.teleport.cluster.local] dns_names:[*.teleport.cluster.local teleport.cluster.local *.bastion.teleport.com] service/connect.go:998
2024-07-04T14:37:41Z DEBU [PROC:1]    Returning existing key pair for. pid:7.1 identity:Proxy reason:re-register service/connect.go:341
2024-07-04T14:37:41Z WARN [PROC:1]    Failed to sync rotation state. pid:7.1 error:[
ERROR REPORT:
Original Error: *trace.BadParameterError missing parameter hostname
Stack Trace:
    github.com/gravitational/teleport/lib/utils/utils.go:286 github.com/gravitational/teleport/lib/utils.Host
    github.com/gravitational/teleport/lib/auth/auth.go:4242 github.com/gravitational/teleport/lib/auth.(*Server).GenerateHostCerts
    github.com/gravitational/teleport/lib/auth/register.go:122 github.com/gravitational/teleport/lib/auth.ReRegister
    github.com/gravitational/teleport/lib/service/connect.go:412 github.com/gravitational/teleport/lib/service.(*TeleportProcess).reRegister
    github.com/gravitational/teleport/lib/service/connect.go:999 github.com/gravitational/teleport/lib/service.(*TeleportProcess).rotate
    github.com/gravitational/teleport/lib/service/connect.go:869 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncServiceRotationState
    github.com/gravitational/teleport/lib/service/connect.go:848 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncRotationState
    github.com/gravitational/teleport/lib/service/connect.go:811 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncRotationStateAndBroadcast
    github.com/gravitational/teleport/lib/service/connect.go:743 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncRotationStateCycle
    github.com/gravitational/teleport/lib/service/connect.go:707 github.com/gravitational/teleport/lib/service.(*TeleportProcess).periodicSyncRotationState
    github.com/gravitational/teleport/lib/service/supervisor.go:588 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
    github.com/gravitational/teleport/lib/service/supervisor.go:313 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
    runtime/asm_arm64.s:1222 runtime.goexit
User Message: missing parameter hostname] service/connect.go:816
2024-07-04T14:37:41Z WARN [PROC:1]    Sync rotation state cycle failed. pid:7.1 retry_interval:10s service/connect.go:712
2024-07-04T14:37:45Z DEBU [AUTH]      Checking for new teleport releases via github api. auth/auth.go:1458
2024-07-04T14:37:49Z INFO [PROC:1]    Initiating new sync rotation state cycle after backoff. pid:7.1 backoff_time:7.123259512s service/connect.go:717
2024-07-04T14:37:49Z INFO [PROC:1]    Rotation in progress, updating SSH principals. pid:7.1 identity:Proxy additional_principals:[teleport-local bastion.teleport.com 0.0.0.0 localhost 127.0.0.1 ::1 remote.kube.proxy.teleport.cluster.local] current_principals:[164cbed7-2832-472f-9711-6d868a02c555.127.0.0.1.nip.io 164cbed7-2832-472f-9711-6d868a02c555 teleport-local.127.0.0.1.nip.io teleport-local localhost 127.0.0.1 ::1 127.0.0.1.nip.io remote.kube.proxy.teleport.cluster.local] service/connect.go:903
2024-07-04T14:37:49Z INFO [PROC:1]    Rotation in progress, updating DNS names. pid:7.1 identity:Proxy additional_dns_names:[*.teleport.cluster.local teleport.cluster.local *.bastion.teleport.com] current_dns_names:[teleport-local 127.0.0.1.nip.io localhost remote.kube.proxy.teleport.cluster.local *.teleport.cluster.local teleport.cluster.local *.127.0.0.1.nip.io] service/connect.go:907
2024-07-04T14:37:49Z INFO [PROC:1]    Service has updated principals and DNS Names, going to request new principals and update. pid:7.1 identity:Proxy principals:[teleport-local bastion.teleport.com 0.0.0.0 localhost 127.0.0.1 ::1 remote.kube.proxy.teleport.cluster.local] dns_names:[*.teleport.cluster.local teleport.cluster.local *.bastion.teleport.com] service/connect.go:998
2024-07-04T14:37:49Z DEBU [PROC:1]    Returning existing key pair for. pid:7.1 identity:Proxy reason:re-register service/connect.go:341
2024-07-04T14:37:49Z WARN [PROC:1]    Failed to sync rotation state. pid:7.1 error:[
ERROR REPORT:
Original Error: *trace.BadParameterError missing parameter hostname
Stack Trace:
    github.com/gravitational/teleport/lib/utils/utils.go:286 github.com/gravitational/teleport/lib/utils.Host
    github.com/gravitational/teleport/lib/auth/auth.go:4242 github.com/gravitational/teleport/lib/auth.(*Server).GenerateHostCerts
    github.com/gravitational/teleport/lib/auth/register.go:122 github.com/gravitational/teleport/lib/auth.ReRegister
    github.com/gravitational/teleport/lib/service/connect.go:412 github.com/gravitational/teleport/lib/service.(*TeleportProcess).reRegister
    github.com/gravitational/teleport/lib/service/connect.go:999 github.com/gravitational/teleport/lib/service.(*TeleportProcess).rotate
    github.com/gravitational/teleport/lib/service/connect.go:869 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncServiceRotationState
    github.com/gravitational/teleport/lib/service/connect.go:848 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncRotationState
    github.com/gravitational/teleport/lib/service/connect.go:811 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncRotationStateAndBroadcast
    github.com/gravitational/teleport/lib/service/connect.go:743 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncRotationStateCycle
    github.com/gravitational/teleport/lib/service/connect.go:707 github.com/gravitational/teleport/lib/service.(*TeleportProcess).periodicSyncRotationState
    github.com/gravitational/teleport/lib/service/supervisor.go:588 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
    github.com/gravitational/teleport/lib/service/supervisor.go:313 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
    runtime/asm_arm64.s:1222 runtime.goexit
User Message: missing parameter hostname] service/connect.go:816
2024-07-04T14:37:49Z WARN [PROC:1]    Sync rotation state cycle failed. pid:7.1 retry_interval:10s service/connect.go:712

Config change was as shown here:

version: v3
teleport:
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: DEBUG
    format:
      output: text
  ca_pin: ""
  diag_addr: ""
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  proxy_listener_mode: multiplex
ssh_service:
  enabled: "yes"
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
proxy_service:
  enabled: "yes"
  web_listen_addr: 0.0.0.0:443
-  public_addr: 127.0.0.1.nip.io:443
+  public_addr: bastion.teleport.com:443
  https_keypairs: []
  https_keypairs_reload_interval: 0s
  acme:
    enabled: "yes"
    email: gus+acme@goteleport.com

Changing teleport.nodename also has a similar effect.

Bug details:

Hu3cx commented 1 month ago

i have same problem.

zhangzhuo0705 commented 1 month ago

i have same problem.

awcodify commented 1 month ago

same problem here. Any update?

for context, i want to deploy this on docker swarm and using traefik

webvictim commented 3 weeks ago

This also happens on 16.2.0 when changing teleport.nodename (with proxy_service.https_keypairs set):

Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z INFO [PROC:1]    Initiating new sync rotation state cycle after backoff. pid:2156.1 backoff_time:17.094732199s service/connect.go:719
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z INFO [PROC:1]    Rotation in progress, updating SSH principals. pid:2156.1 identity:Admin additional_principals:[ip-172-31-4-69] current_principals:[3bf51591-a1c8-4871-8dcc-be6fdad8c5da ip-172-31-4-118] service/connect.go:905
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z INFO [PROC:1]    Service has updated principals and DNS Names, going to request new principals and update. pid:2156.1 identity:Admin principals:[ip-172-31-4-69] dns_names:[*.teleport.cluster.local teleport.cluster.local] service/connect.go:1000
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z DEBU             Generated SSH host certificate for role Admin with principals: [3bf51591-a1c8-4871-8dcc-be6fdad8c5da ip-172-31-4-69]. keygen/keygen.go:128
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z DEBU [CA]        Generating TLS certificate common_name:3bf51591-a1c8-4871-8dcc-be6fdad8c5da.leaf1.gus.devteleport.com dns_names:[ip-172-31-4-69 *.teleport.cluster.local teleport.cluster.local] key_usage:5 not_after:2034-08-28 19:05:25.655616833 +0000 UTC tlsca/ca.go:1229
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z INFO [PROC:1]    Rotation in progress, updating SSH principals. pid:2156.1 identity:Instance additional_principals:[ip-172-31-4-69] current_principals:[3bf51591-a1c8-4871-8dcc-be6fdad8c5da.leaf1.gus.devteleport.com 3bf51591-a1c8-4871-8dcc-be6fdad8c5da ip-172-31-4-118.leaf1.gus.devteleport.com ip-172-31-4-118 localhost 127.0.0.1 ::1] service/connect.go:905
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z INFO [PROC:1]    Service has updated principals and DNS Names, going to request new principals and update. pid:2156.1 identity:Instance principals:[ip-172-31-4-69] dns_names:[] service/connect.go:1000
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z DEBU [PROC:1]    Generating new key pair. pid:2156.1 identity:Instance reason:re-register service/connect.go:346
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z DEBU             Generated SSH host certificate for role Instance with principals: [3bf51591-a1c8-4871-8dcc-be6fdad8c5da.leaf1.gus.devteleport.com 3bf51591-a1c8-4871-8dcc-be6fdad8c5da ip-172-31-4-69.leaf1.gus.devteleport.com ip-172-31-4-69 localhost 127.0.0.1 ::1]. keygen/keygen.go:128
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z DEBU [CA]        Generating TLS certificate common_name:3bf51591-a1c8-4871-8dcc-be6fdad8c5da.leaf1.gus.devteleport.com dns_names:[ip-172-31-4-69] key_usage:5 not_after:2034-08-28 19:05:25.686153637 +0000 UTC tlsca/ca.go:1229
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z DEBU [PROC:1]    Deleted generated key pair. pid:2156.1 identity:Instance reason:re-register service/connect.go:332
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z INFO [PROC:1]    Rotation in progress, updating SSH principals. pid:2156.1 identity:Proxy additional_principals:[ip-172-31-4-69 leaf1.gus.devteleport.com 0.0.0.0 localhost 127.0.0.1 ::1 remote.kube.proxy.teleport.cluster.local] current_principals:[3bf51591-a1c8-4871-8dcc-be6fdad8c5da.leaf1.gus.devteleport.com 3bf51591-a1c8-4871-8dcc-be6fdad8c5da ip-172-31-4-118.leaf1.gus.devteleport.com ip-172-31-4-118 localhost 127.0.0.1 ::1 leaf1.gus.devteleport.com remote.kube.proxy.teleport.cluster.local] service/connect.go:905
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z INFO [PROC:1]    Service has updated principals and DNS Names, going to request new principals and update. pid:2156.1 identity:Proxy principals:[ip-172-31-4-69 leaf1.gus.devteleport.com 0.0.0.0 localhost 127.0.0.1 ::1 remote.kube.proxy.teleport.cluster.local] dns_names:[*.teleport.cluster.local teleport.cluster.local *.leaf1.gus.devteleport.com] service/connect.go:1000
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z DEBU [PROC:1]    Returning existing key pair for. pid:2156.1 identity:Proxy reason:re-register service/connect.go:343
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z WARN [PROC:1]    Failed to sync rotation state. pid:2156.1 error:[
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: ERROR REPORT:
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: Original Error: *trace.BadParameterError missing parameter hostname
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: Stack Trace:
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]:         github.com/gravitational/teleport/lib/utils/utils.go:286 github.com/gravitational/teleport/lib/utils.Host
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]:         github.com/gravitational/teleport/lib/auth/auth.go:4291 github.com/gravitational/teleport/lib/auth.(*Server).GenerateHostCerts
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]:         github.com/gravitational/teleport/lib/auth/register.go:122 github.com/gravitational/teleport/lib/auth.ReRegister
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]:         github.com/gravitational/teleport/lib/service/connect.go:414 github.com/gravitational/teleport/lib/service.(*TeleportProcess).reRegister
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]:         github.com/gravitational/teleport/lib/service/connect.go:1001 github.com/gravitational/teleport/lib/service.(*TeleportProcess).rotate
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]:         github.com/gravitational/teleport/lib/service/connect.go:871 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncServiceRotationState
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]:         github.com/gravitational/teleport/lib/service/connect.go:850 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncRotationState
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]:         github.com/gravitational/teleport/lib/service/connect.go:813 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncRotationStateAndBroadcast
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]:         github.com/gravitational/teleport/lib/service/connect.go:745 github.com/gravitational/teleport/lib/service.(*TeleportProcess).syncRotationStateCycle
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]:         github.com/gravitational/teleport/lib/service/connect.go:709 github.com/gravitational/teleport/lib/service.(*TeleportProcess).periodicSyncRotationState
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]:         github.com/gravitational/teleport/lib/service/supervisor.go:588 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]:         github.com/gravitational/teleport/lib/service/supervisor.go:313 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]:         runtime/asm_amd64.s:1695 runtime.goexit
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: User Message: missing parameter hostname] service/connect.go:818
Aug 30 19:05:25 ip-172-31-4-69 teleport[2156]: 2024-08-30T19:05:25Z WARN [PROC:1]    Sync rotation state cycle failed. pid:2156.1 retry_interval:10s service/connect.go:714

This just repeats on a loop forever and the old nodename continues to be used.

Config change:

version: v3
teleport:
- nodename: ip-172-31-4-118
+ nodename: ip-172-31-4-69
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: INFO
    format:
      output: text
  ca_pin: ""
  diag_addr: ""
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  cluster_name: root.teleport.example.com
  proxy_listener_mode: multiplex
  authentication:
    second_factor: on
    webauthn:
      rp_id: root.teleport.example.com
ssh_service:
  enabled: "yes"
proxy_service:
  enabled: "yes"
  web_listen_addr: 0.0.0.0:3080
  public_addr: root.teleport.example.com:443
  https_keypairs:
  - key_file: /etc/letsencrypt/live/root.teleport.example.com/privkey.pem
    cert_file: /etc/letsencrypt/live/root.teleport.example.com/fullchain.pem
  https_keypairs_reload_interval: 12h
  acme: {}
  proxy_protocol: on
fspmarshall commented 3 weeks ago

This issue should be fixed once https://github.com/gravitational/teleport/pull/46087 gets in. In the meantime, specifying advertise_ip or explicitly binding a specific network interface rather than 0.0.0.0 will circumvent the bug.