search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
16.98k stars 1.71k forks source link

upgrading 2.2 to 2.3 thinks I am renaming my cluster #1501

Closed scottbessler closed 6 years ago

scottbessler commented 6 years ago

I currently have a 2.2.7 cluster running just fine with this config:

(redacted secrets and replaced the current auth server hostname with teleport.hostname.fake for purposes of this issue)

teleport:
    # Logging configuration. Possible output values are 'stdout', 'stderr' and
    # 'syslog'. Possible severity values are INFO, WARN and ERROR (default).
    #log:
    #    severity: DEBUG

# This section configures the 'node service':
ssh_service:
    # Turns 'ssh' role on. Default is 'yes'
    enabled: yes

    # IP and the port for SSH service to bind to.
    #listen_addr: 0.0.0.0:3022
    # See explanation of labels in "Labeling Nodes" section below
    labels:
        role: auth

# List (YAML array) of commands to periodically execute and use
    # their output as labels.
    # See explanation of how this works in "Labeling Nodes" section below
    commands:
    - name: hostname
      command: [/bin/hostname]
      period: 1m0s
    - name: arch
      command: [/bin/uname, -p]
      period: 1h0m0s

auth_service:
    cluster_name: cfe5152a-5494-45f4-8ade-fb531de0c4f7
    # Turns 'auth' role on. Default is 'yes'
    enabled: true
    tokens:
       - "node:REDACTED"
    authentication:
      type: local
      second_factor: otp

# This section configures the 'proxy servie'
proxy_service:
    # Turns 'proxy' role on. Default is 'yes'
    enabled: yes

    # SSH forwarding/proxy address. Command line (CLI) clients always begin their
    # SSH sessions by connecting to this port
    #listen_addr: 0.0.0.0:3023

    # Reverse tunnel listening address. An auth server (CA) can establish an
    # outbound (from behind the firewall) connection to this address.
    # This will allow users of the outside CA to connect to behind-the-firewall
    # nodes.
    #tunnel_listen_addr: 0.0.0.0:3024

    # The HTTPS listen address to serve the Web UI and also to authenticate the
    # command line (CLI) users via password+HOTP
    #web_listen_addr: 0.0.0.0:3080

    # TLS certificate for the HTTPS connection. Configuring these properly is
    # critical for Teleport security.
    https_key_file: /etc/letsencrypt/live/teleport.hostname.fake/privkey.pem
    https_cert_file: /etc/letsencrypt/live/teleport.hostname.fake/fullchain.pem

when i try and upgrade by stopping the 2.2.7 server and swapping in the 2.3.5 binaries (using same config an data dir) I get this error on startup:

cannot rename cluster "teleport.hostname.fake" to "cfe5152a-5494-45f4-8ade-fb531de0c4f7": clusters cannot be renamed, initializing teleport
klizhentas commented 6 years ago

Scott, sorry for the slow response. We need to dig into the code path to understand why it happens as it's not clear right away.

kontsevoy commented 6 years ago

@scottbessler have you always had cluster_name in there? It's a strange name to pick (a GUID) which I suspect you pulled from one of the node IDs. If you comment out cluster_name it should stop complaining. (the docs mention that this setting cannot be changed after a cluster is created)

scottbessler commented 6 years ago

yeah, @kontsevoy sorry i should have included that. without the cluster_name i get:

root@teleport:/home/ubuntu/teleport-235# ./teleport start --config ./teleport.yaml
[AUTH]  Auth service is starting on 0.0.0.0:4025
WARN advertise_ip is not set for this auth server. Trying to guess the IP this server can be reached at: 172.31.101.203:4025  file="service/service.go:443" func="service.(*TeleportProcess).initAuthService.func3"
WARN conn(127.0.0.1:55898->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55896->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55900->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55902->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55906->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55904->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55908->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55910->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55912->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55914->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55916->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55918->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55920->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55922->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55924->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55926->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55928->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55930->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
WARN conn(127.0.0.1:55932->127.0.0.1:4025, user=cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7) ERROR: failed auth user cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7, err: ssh: certificate signed by unrecognized authority  file="auth/tun.go:429" func="auth.(*AuthTunnel).keyAuth"
access denied to 'cfe5152a-5494-45f4-8ade-fb531de0c4f7.cfe5152a-5494-45f4-8ade-fb531de0c4f7': bad username or credentials
scottbessler commented 6 years ago

(i changed the ports so i could run it alongside my working teleport 2.2 instance)

kontsevoy commented 6 years ago

@scottbessler would you be comfortable sharing data from /var/lib/teleport with our engineers? I feel like this would be the fastest way to troubleshoot.

scottbessler commented 6 years ago

@kontsevoy i'd rather not share all of it, but are there specific things you want to see? perhaps a screen share debug session or something?

kontsevoy commented 6 years ago

we can do that also. can you send me a few time slots that work for you? email would be best: ev@gravitational.com

russjones commented 6 years ago

@scottbessler Can you update the configuration of the Auth Server to the following (replacing teleport.hostname.fake with the real cluster name) and trying to upgrade to Teleport 2.3.

auth_service:
    cluster_name: teleport.hostname.fake

We changed how we handle configuration substantially in Teleport 2.3, and I think you are seeing some side effects of that: properties that were (sometimes) ignored in Teleport 2.2 became mandatory in Teleport 2.3 so Teleport would always start up in a consistent manner.

russjones commented 6 years ago

I'm going to close this issue for now and remove it from the 2.4.2 release. If it's still occurring we can investigate in the 2.5.1 release.