Closed cockroach-teamcity closed 7 years ago
This was happening on a lot of my PRs and then it just went away. Concurrently, I was unable to run TestPut
locally because of cert errors and then that problem went away without any change on my part. Very strange. I'm going to leave this open for a bit in case it reproduces.
Happened again in #12796. I'm also going to consolidate the other two issues that are still open for the same timeout (#12738 and #10367).
The cert issue with TestPut
reappeared when I'm running locally. Once per second the acceptance test reports:
I170109 21:09:39.024322 33 vendor/google.golang.org/grpc/clientconn.go:559 Failed to dial 127.0.0.1:32856: connection error: desc = "transport: remote error: tls: bad certificate"; please retry.
Not sure if this is related.
It's interesting that every TestPut
failure that I looked at is blocked on a call to client.(*DB).CheckConsistency
. And those calls are all blocked waiting for a gRPC response. This is the only test which calls CheckConsistency
so frequently (once per second).
Kept run into the cert issue locally. Turns out the clock in my docker VM was a few days behind reality. It reported the date as Jan 7, 2017 while the certs were not created until Jan 9, 2017. Restarting docker fixed the issue and I've now run TestPut
a few dozen times locally without failure.
go test -count 100 -tags acceptance -run TestPut -timeout 24h -v --nodes 3 -l . --logtostderr
passed on all 100 runs locally. I think I need to add some more debugging info as this failure is happening very frequently on teamcity.
Did you ever get around to adding debugging info? It's still timing out pretty regularly.
Not yet. Still on my TODO list.
The following tests appear to have failed:
#110121:
Please assign, take a look and update the issue accordingly.