Segmentation fault when trying to debug a job

Hello,

For the past day all our systemd jobs keep failing and the logs are getting truncated, so I wanted to give it a try via the SSH debug feature. Unfortunately, all attempts to debug any of the failed jobs end up with a segfault:

# sem debug job 0a1308be-f530-4f79-b405-c6e35d9b6c34
* Creating debug session for job '0a1308be-f530-4f79-b405-c6e35d9b6c34'
* Setting duration to 60 minutes
* Waiting for debug session to boot up ..
* Waiting for ssh daemon to become ready .
* Stopping debug session ..
* Session stopped
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x7c2286]

goroutine 1 [running]:
os/exec.(*Cmd).CombinedOutput(0x0, 0x1b, 0xc0001038b8, 0x1, 0x1, 0x93e620)
    /usr/local/golang/1.11/go/src/os/exec/exec.go:516 +0x26
github.com/semaphoreci/cli/cmd/ssh.(*Connection).IsReady(0xc00011f620, 0x4c6382, 0x93e5a0, 0xc00000c018)
    /home/semaphore/cli/cmd/ssh/connection.go:90 +0xcc
github.com/semaphoreci/cli/cmd/ssh.(*Connection).WaitUntilReady(0xc00011f620, 0x14, 0x8de458, 0x0, 0x0)
    /home/semaphore/cli/cmd/ssh/connection.go:74 +0x76
github.com/semaphoreci/cli/cmd/ssh.StartDebugSession(0xc00000e940, 0x8d7b08, 0xf0, 0x0, 0x0)
    /home/semaphore/cli/cmd/ssh/session.go:71 +0x412
github.com/semaphoreci/cli/cmd/ssh.StartDebugJobSession(0xc00000aaa0, 0x8d7b08, 0xf0, 0x1, 0x1)
    /home/semaphore/cli/cmd/ssh/session.go:18 +0xb4
github.com/semaphoreci/cli/cmd.RunDebugJobCmd(0xc00010c500, 0xc000051470, 0x1, 0x1)
    /home/semaphore/cli/cmd/debug_job.go:53 +0x2ac
github.com/spf13/cobra.(*Command).execute(0xc00010c500, 0xc000051440, 0x1, 0x1, 0xc00010c500, 0xc000051440)
    /home/semaphore/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x2cc
github.com/spf13/cobra.(*Command).ExecuteC(0xc313a0, 0x0, 0x0, 0xc000103ee0)
    /home/semaphore/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2fd
github.com/spf13/cobra.(*Command).Execute(0xc313a0, 0x6a, 0xa0)
    /home/semaphore/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800 +0x2b
github.com/semaphoreci/cli/cmd.Execute()
    /home/semaphore/cli/cmd/root.go:37 +0x2d

I suspect this is caused by:

# sem -v debug job 0a1308be-f530-4f79-b405-c6e35d9b6c34
...
2021/03/02 10:54:29 response Status: 200 OK
2021/03/02 10:54:29 response Headers: map[Via:[1.1 google] Alt-Svc:[clear] Grpc-Metadata-Content-Type:[application/grpc+proto] Grpc-Metadata-Server:[envoy] Date:[Tue, 02 Mar 2021 10:54:29 GMT] X-Envoy-Upstream-Service-Time:[37] Server:[envoy] X-Envoy-Decorator-Operation:[public-api-gateway.default.svc.cluster.local:80/*] Content-Type:[application/json] Grpc-Metadata-Date:[Tue, 02 Mar 2021 10:54:29 GMT] Grpc-Metadata-X-Envoy-Upstream-Service-Time:[33] Content-Length:[1712]]
2021/03/02 10:54:29 {"key":"-----BEGIN RSA PRIVATE KEY-----<snip>-----END RSA PRIVATE KEY-----\n"}
* Waiting for ssh daemon to become ready .2021/03/02 10:54:30 SSH connection: Running <nil>

* Stopping debug session ..

Environment:

OS: Arch Linux, Fedora Rawhide
sem version: 0.21.0, commit e2866d25ce831675ac30b89dbc80446801357e32, built at 2020-12-01T13:31:24Z

Can confirm this is also happening to me.

* Creating debug session for job '340c4875-6dda-4528-96cb-fce86bdd695d'
* Setting duration to 60 minutes
* Waiting for debug session to boot up ..............................
* Stopping debug session ..
* Session stopped
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x20 pc=0x1028d2538]

goroutine 1 [running]:
github.com/semaphoreci/cli/cmd/ssh.waitUntilJobIsRunning(0x0, 0x102a28728, 0x1028f2fc8, 0x28)
    /home/semaphore/cli/cmd/ssh/session.go:99 +0x58
github.com/semaphoreci/cli/cmd/ssh.StartDebugSession(0x14000010100, 0x1028fc04b, 0xf0, 0x0, 0x0)
    /home/semaphore/cli/cmd/ssh/session.go:46 +0xe8
github.com/semaphoreci/cli/cmd/ssh.StartDebugJobSession(0x140001aa2d0, 0x1028fc04b, 0xf0, 0x21, 0x14000221c98)
    /home/semaphore/cli/cmd/ssh/session.go:18 +0x88
github.com/semaphoreci/cli/cmd.RunDebugJobCmd(0x14000228500, 0x140001973e0, 0x1, 0x1)
    /home/semaphore/cli/cmd/debug_job.go:53 +0x254
github.com/spf13/cobra.(*Command).execute(0x14000228500, 0x140001973b0, 0x1, 0x1, 0x14000228500, 0x140001973b0)
    /home/semaphore/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x1d0
github.com/spf13/cobra.(*Command).ExecuteC(0x102c673c0, 0x1400022e000, 0x6b, 0x14000230000)
    /home/semaphore/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x21c
github.com/spf13/cobra.(*Command).Execute(...)
    /home/semaphore/go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800
github.com/semaphoreci/cli/cmd.Execute()
    /home/semaphore/cli/cmd/root.go:37 +0x30
main.main()
    /home/semaphore/cli/main.go:27 +0x230

sem version: 0.24.0, commit 4082ec9107fd5cdaccfc62329821926935bbd438, built at 2021-09-16T10:06:21Z

semaphoreci / cli

Segmentation fault when trying to debug a job #176