roachtest: jepsen-batch2/register/strobe-skews failed

SHA: https://github.com/cockroachdb/cockroach/commits/c25518b4e9a723d8de0dba30a95ce0ade7963aed

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=jepsen-batch2/register/strobe-skews PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1286188&tab=buildLog

The test failed on branch=master, cloud=gce:
    cluster.go:1332,jepsen.go:64,jepsen.go:136,jepsen.go:322,test.go:1251: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1286188-jepsen-batch2:6 -- bash -e -c '
        if ! test -d /mnt/data1/jepsen; then
          git clone -b tc-nightly --depth 1 https://github.com/cockroachdb/jepsen /mnt/data1/jepsen
        else
          cd /mnt/data1/jepsen
          git fetch origin
          git checkout origin/tc-nightly
        fi
        ' returned:
        stderr:

        stdout:
        Connection to 34.73.229.13 closed by remote host.
        Error:  exit status 255
        : exit status 1

May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[26767]: debug1: Forked child 1833.
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: Set /proc/self/oom_score_adj to 0
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: inetd sockets after dupping: 3, 3
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: Connection from 104.196.61.98 port 49562 on 10.142.0.155 port 22
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: Client protocol version 2.0; client software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.6
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.6 pat OpenSSH* compat 0x04000000
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: Enabling compatibility mode for protocol 2.0
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: permanently_set_uid: 110/65534 [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: SSH2_MSG_KEXINIT sent [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: SSH2_MSG_KEXINIT received [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: kex: algorithm: curve25519-sha256@libssh.org [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: rekey after 134217728 blocks [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: SSH2_MSG_NEWKEYS received [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: rekey after 134217728 blocks [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: KEX done [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: userauth-request for user root service ssh-connection method none [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: attempt 0 failures 0 [preauth]
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: PAM: initializing for "root"
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: PAM: setting PAM_RHOST to "104.196.61.98"
May 12 10:27:48 teamcity-1286188-jepsen-batch2-0006 sshd[1833]: debug1: PAM: setting PAM_TTY to "ssh"

Closing, future debugging should be assisted by https://github.com/cockroachdb/cockroach/pull/37483.

cockroachdb / cockroach

roachtest: jepsen-batch2/register/strobe-skews failed #37481