TestRunServer/SecureTenant is broken with CockroachDB master

[rafiss]

This test is currently broken:

=== RUN   TestRunServer
=== RUN   TestRunServer/SecureTenant
2020/08/06 16:17:51 Using custom cockroach binary: /Users/rafiss/go/src/github.com/cockroachdb/cockroach/cockroach
2020/08/06 16:17:51 process 95859 started: /Users/rafiss/go/src/github.com/cockroachdb/cockroach/cockroach start-single-node --logtostderr --certs-dir=/tmp/cockroach-testserver274475142/certs --host=localhost --port=0 --http-port=0 --store=/tmp/cockroach-testserver274475142 --listening-url-file=/tmp/cockroach-testserver274475142/listen-url
2020/08/06 16:17:53 process 95886 started: /Users/rafiss/go/src/github.com/cockroachdb/cockroach/cockroach mt start-sql --certs-dir=/tmp/cockroach-testserver274475142/certs --logtostderr --tenant-id=2 --kv-addrs=localhost:50535 --sql-addr=localhost:50545
2020/08/06 16:17:53 WaitForInit: Trying again after error: dial tcp [::1]:50545: connect: connection refused
2020/08/06 16:17:53 Process 95886 exited with status 2: exit status 2
2020/08/06 16:17:53 exit status 2
2020/08/06 16:17:53 WaitForInit: Trying again after error: dial tcp [::1]:50545: connect: connection refused
2020/08/06 16:17:53 WaitForInit: Trying again after error: dial tcp [::1]:50545: connect: connection refused
2020/08/06 16:17:53 WaitForInit: Trying again after error: dial tcp [::1]:50545: connect: connection refused
2020/08/06 16:17:53 WaitForInit: Trying again after error: dial tcp [::1]:50545: connect: connection refused
...

--- FAIL: TestRunServer (7.58s)
    testserver_test.go:44: dial tcp [::1]:50545: connect: connection refused
    --- FAIL: TestRunServer/SecureTenant (7.58s)

I was able to reproduce locally, and see this error when starting the tenant:

❯ /Users/rafiss/go/src/github.com/cockroachdb/cockroach/cockroach mt start-sql --certs-dir=/tmp/cockroach-testserver274475142/certs --logtostderr --tenant-id=2 --kv-addrs=localhost:50535 --sql-addr=localhost:50545
I200806 20:18:51.231010 15 util/stop/stopper.go:563  quiescing
E200806 20:18:51.230998 1 util/log/crash_reporting.go:138  a panic has occurred!
for 2: tenant client cert not found
(1) attached stack trace
  | runtime.gopanic
  |     /usr/local/Cellar/go/1.14.1/libexec/src/runtime/panic.go:967
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:503
  | runtime.gopanic
  |     /usr/local/Cellar/go/1.14.1/libexec/src/runtime/panic.go:967
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServer
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:216
  | github.com/cockroachdb/cockroach/pkg/server.makeSQLServerArgs
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:561
  | github.com/cockroachdb/cockroach/pkg/server.StartTenant
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:641
  | github.com/cockroachdb/cockroach/pkg/cli.runStartSQL
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/mt_start_sql.go:68
  | github.com/cockroachdb/cockroach/pkg/cli.MaybeDecorateGRPCError.func1
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/error.go:189
  | github.com/spf13/cobra.(*Command).execute
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:826
  | github.com/spf13/cobra.(*Command).ExecuteC
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
  | github.com/spf13/cobra.(*Command).Execute
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
  | github.com/cockroachdb/cockroach/pkg/cli.Run
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:225
  | github.com/cockroachdb/cockroach/pkg/cli.Main
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:67
  | main.main
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cmd/cockroach-short/main.go:26
  | runtime.main
  |     /usr/local/Cellar/go/1.14.1/libexec/src/runtime/proc.go:203
  | runtime.goexit
  |     /usr/local/Cellar/go/1.14.1/libexec/src/runtime/asm_amd64.s:1373
Wraps: (2) for 2: tenant client cert not found
Error types: (1) *withstack.withStack (2) *security.Error
E200806 20:18:51.231616 1 util/log/crash_reporting.go:138  a panic has occurred!
for 2: tenant client cert not found
(1) attached stack trace
  | runtime.gopanic
  |     /usr/local/Cellar/go/1.14.1/libexec/src/runtime/panic.go:967
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:503
  | runtime.gopanic
  |     /usr/local/Cellar/go/1.14.1/libexec/src/runtime/panic.go:967
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServer
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:216
  | github.com/cockroachdb/cockroach/pkg/server.makeSQLServerArgs
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:561
  | github.com/cockroachdb/cockroach/pkg/server.StartTenant
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:641
  | github.com/cockroachdb/cockroach/pkg/cli.runStartSQL
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/mt_start_sql.go:68
  | github.com/cockroachdb/cockroach/pkg/cli.MaybeDecorateGRPCError.func1
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/error.go:189
  | github.com/spf13/cobra.(*Command).execute
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:826
  | github.com/spf13/cobra.(*Command).ExecuteC
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914
  | github.com/spf13/cobra.(*Command).Execute
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
  | github.com/cockroachdb/cockroach/pkg/cli.Run
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:225
  | github.com/cockroachdb/cockroach/pkg/cli.Main
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:67
  | main.main
  |     /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cmd/cockroach-short/main.go:26
  | runtime.main
  |     /usr/local/Cellar/go/1.14.1/libexec/src/runtime/proc.go:203
  | runtime.goexit
  |     /usr/local/Cellar/go/1.14.1/libexec/src/runtime/asm_amd64.s:1373
Wraps: (2) for 2: tenant client cert not found
Error types: (1) *withstack.withStack (2) *security.Error
panic: for 2: tenant client cert not found [recovered]
    panic: for 2: tenant client cert not found [recovered]
    panic: for 2: tenant client cert not found [recovered]
    panic: for 2: tenant client cert not found

goroutine 1 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.RecoverAndReportPanic(0x9091a60, 0xc00007a160, 0xc000016a80)
    /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/util/log/crash_reporting.go:88 +0x9f
panic(0x7e15f20, 0xc000da45c0)
    /usr/local/Cellar/go/1.14.1/libexec/src/runtime/panic.go:967 +0x166
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Recover(0xc00011fdd0, 0x9091a60, 0xc00007a160)
    /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:207 +0x11f
panic(0x7e15f20, 0xc000da45c0)
    /usr/local/Cellar/go/1.14.1/libexec/src/runtime/panic.go:967 +0x166
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc00011fdd0, 0x9091a60, 0xc00007a160)
    /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:503 +0x68c
panic(0x7e15f20, 0xc000da45c0)
    /usr/local/Cellar/go/1.14.1/libexec/src/runtime/panic.go:967 +0x166
github.com/cockroachdb/cockroach/pkg/rpc.NewServer(0xc0000f6240, 0x0, 0x0, 0x0, 0x90db8a0)
    /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:216 +0xe7c
github.com/cockroachdb/cockroach/pkg/server.makeSQLServerArgs(0xc00011fdd0, 0x0, 0x0, 0xc000016a80, 0xc000197400, 0x906e860, 0xc000adf8a0, 0x0, 0x0, 0x0, ...)
    /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:561 +0xbda
github.com/cockroachdb/cockroach/pkg/server.StartTenant(0x9091a60, 0xc00007a160, 0xc00011fdd0, 0x0, 0x0, 0xc000016a80, 0xc000197400, 0x906e860, 0xc000adf8a0, 0x0, ...)
    /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/server/testserver.go:641 +0xf8
github.com/cockroachdb/cockroach/pkg/cli.runStartSQL(0xb7685c0, 0xc000b0b450, 0x0, 0x5, 0x0, 0x0)
    /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/mt_start_sql.go:68 +0x2c4
github.com/cockroachdb/cockroach/pkg/cli.MaybeDecorateGRPCError.func1(0xb7685c0, 0xc000b0b450, 0x0, 0x5, 0x0, 0x0)
    /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/error.go:189 +0x79
github.com/spf13/cobra.(*Command).execute(0xb7685c0, 0xc000b0b400, 0x5, 0x5, 0xb7685c0, 0xc000b0b400)
    /Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:826 +0x453
github.com/spf13/cobra.(*Command).ExecuteC(0xb75f740, 0xc00007a160, 0xc00062b62a, 0xc)
    /Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
    /Users/rafiss/go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:864
github.com/cockroachdb/cockroach/pkg/cli.Run(...)
    /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:225
github.com/cockroachdb/cockroach/pkg/cli.Main()
    /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:67 +0x270
main.main()
    /Users/rafiss/go/src/github.com/cockroachdb/cockroach/pkg/cmd/cockroach-short/main.go:26 +0x20

[rafiss]

@asubiotto or @tbg would one of you be able to look into this?

I see that this landed recently: https://github.com/cockroachdb/cockroach/pull/51503 Could it be related?

[tbg]

I'll take a look!

[tbg]

@darinpp and I are just about to simplify the certs story here quite a bit, so I'm going to hold off on fixing this test for a moment or two. We will need to touch this anway to add in the SQL proxy in https://github.com/cockroachdb/cockroach/pull/52478.