Upgrade from 2.5.6 to 2.6.7 : missing parameter TLSCert, initialization failed

[jnancel]

What happened: I tried to upgrade from 2.5.6 to 2.6.7, starting with the auth server, as specified in the upgrading guide ( https://gravitational.com/teleport/docs/admin-guide/#upgrading-teleport ). But when I restart the teleport service after having upgraded the teleport and tctl binaries, I have the error missing parameter TLSCert, initialization failed

When I downgrade to 2.5.6, everything is back to normal.

I have the same issue when trying to upgrade from 2.5.6 to 2.7.0.

What you expected to happen: I expected to upgrade the binaries, restart the service and the upgrade would be done.

How to reproduce it (as minimally and precisely as possible):

• Install 2.5.6
• Upgrade to 2.6.7

Environment:

• Teleport version (use teleport version): Before : Teleport v2.5.6 git:v2.5.6-0-g81d77d2 After : Teleport v2.6.7 git:v2.6.7-0-g280f0fe3

• OS (e.g. from /etc/os-release):

  NAME="Debian GNU/Linux"
  VERSION_ID="7"
  VERSION="7 (wheezy)"
  ID=debian
  ANSI_COLOR="1;31"
  HOME_URL="http://www.debian.org/"
  SUPPORT_URL="http://www.debian.org/support/"
  BUG_REPORT_URL="http://bugs.debian.org/"

Relevant Debug Logs If Applicable

• teleport start --debug

  
  DEBU [PROC:1]    Using dir backend. service/service.go:1780
  DEBU [BACKEND:D] AcquireLock(bastion) dir:/var/lib/teleport/backend dir/impl.go:395
  INFO [AUTH]      Auth server is running periodic operations. auth/auth.go:109
  DEBU [AUTH]      Cluster configuration: ClusterName(bastion). auth/init.go:129
  INFO [AUTH]      Updating cluster configuration: StaticTokens([]). auth/init.go:129
  INFO [AUTH]      Updating cluster configuration: AuthPreference(Type="local",SecondFactor="otp"). auth/init.go:230
  INFO [AUTH]      Created namespace: "default". auth/init.go:237
  DEBU [AUTH]      Ticking with period: 10s. auth/auth.go:159
  DEBU [AUTH]      Migrations: skipping local cluster cert authority "bastion". auth/init.go:784
  DEBU [BACKEND:D] ReleaseLock(bastion) dir:/var/lib/teleport/backend dir/impl.go:425

ERROR REPORT: Original Error: trace.BadParameterError missing parameter TLSCert Stack Trace: /gopath/src/github.com/gravitational/teleport/lib/auth/state.go:191 github.com/gravitational/teleport/lib/auth.(IdentityV2).CheckAndSetDefaults /gopath/src/github.com/gravitational/teleport/lib/auth/state.go:131 github.com/gravitational/teleport/lib/auth.(ProcessStorage).WriteIdentity /gopath/src/github.com/gravitational/teleport/lib/auth/init.go:415 github.com/gravitational/teleport/lib/auth.migrateIdentity /gopath/src/github.com/gravitational/teleport/lib/auth/init.go:400 github.com/gravitational/teleport/lib/auth.migrateIdentities /gopath/src/github.com/gravitational/teleport/lib/auth/init.go:387 github.com/gravitational/teleport/lib/auth.migrateLegacyResources /gopath/src/github.com/gravitational/teleport/lib/auth/init.go:370 github.com/gravitational/teleport/lib/auth.Init /gopath/src/github.com/gravitational/teleport/lib/service/service.go:823 github.com/gravitational/teleport/lib/service.(TeleportProcess).initAuthService /gopath/src/github.com/gravitational/teleport/lib/service/service.go:597 github.com/gravitational/teleport/lib/service.NewTeleport /gopath/src/github.com/gravitational/teleport/lib/service/service.go:362 github.com/gravitational/teleport/lib/service.newTeleportProcess /gopath/src/github.com/gravitational/teleport/lib/service/service.go:372 github.com/gravitational/teleport/lib/service.Run /gopath/src/github.com/gravitational/teleport/tool/teleport/common/teleport.go:167 github.com/gravitational/teleport/tool/teleport/common.OnStart /gopath/src/github.com/gravitational/teleport/tool/teleport/common/teleport.go:148 github.com/gravitational/teleport/tool/teleport/common.Run /gopath/src/github.com/gravitational/teleport/tool/teleport/main.go:29 main.main /opt/go/src/runtime/proc.go:207 runtime.main /opt/go/src/runtime/asm_amd64.s:2362 runtime.goexit User Message: missing parameter TLSCert, initialization failed

[russjones]

@jnancel I was not able to reproduce this. Can you share your teleport.yaml file?

[russjones]

@jnancel Can you share the output of ls -laF on your Teleport data directory? You should see something like the following:

$ ls -laF
total 68
-rw------- 1 rjones rjones 1446 Oct 23 19:49 admin.cert
-rw------- 1 rjones rjones 1675 Oct 23 19:49 admin.key
-rw------- 1 rjones rjones 1269 Oct 23 19:49 admin.tlscacert
-rw------- 1 rjones rjones 1326 Oct 23 19:49 admin.tlscert
drwxr-x--- 6 rjones rjones 4096 Oct 23 19:49 cache/
-r-------- 1 rjones rjones   36 Oct 23 19:49 host_uuid
drwxr-x--- 6 rjones rjones 4096 Oct 23 19:49 log/
-rw------- 1 rjones rjones 1522 Oct 23 19:49 node.cert
-rw------- 1 rjones rjones 1679 Oct 23 19:49 node.key
-rw------- 1 rjones rjones 1269 Oct 23 19:49 node.tlscacert
-rw------- 1 rjones rjones 1273 Oct 23 19:49 node.tlscert
-rw------- 1 rjones rjones 1526 Oct 23 19:49 proxy.cert
-rw------- 1 rjones rjones 1679 Oct 23 19:49 proxy.key
-rw------- 1 rjones rjones 1269 Oct 23 19:49 proxy.tlscacert
-rw------- 1 rjones rjones 1277 Oct 23 19:49 proxy.tlscert
-rw------- 1 rjones rjones 1208 Oct 23 19:49 webproxy_cert.pem
-rw------- 1 rjones rjones 1675 Oct 23 19:49 webproxy_key.pem

My guess is the *.tlscacert and *.tlscert files are missing. Which is odd because making sure those files exist is part of the 2.5.6 startup.

[russjones]

@jnancel I'm going to close this ticket for now. If you continue to have problems, please create another ticket.