kv/kvnemesis: TestKVNemesisMultiNode failed

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

Other

30k stars 3.79k forks source link

(kv/kvnemesis).TestKVNemesisMultiNode failed on master@7b0ccdda99b81613e70f421c9374483c3feddff3:

          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).ChangeReplicas
          |     /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_command.go:964
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeAdminBatch
          |     /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:584
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID
          |     /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:100
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send
          |     /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send
          |     /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:195
          | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send
          |     /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:177
          | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1
          |     /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:874
          | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
          |     /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:326
          | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
          |     /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:862
          | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
          |     /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:900
          | github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1
          |     /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9083
          | github.com/cockroachdb/cockroach/pkg/util/tracing.ServerInterceptor.func1
          |     /go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/grpc_interceptor.go:135
          | google.golang.org/grpc.getChainUnaryHandler.func1
          |     /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:921
          | github.com/cockroachdb/cockroach/pkg/rpc.NewServer.func1
          |     /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:176
          | google.golang.org/grpc.getChainUnaryHandler.func1
          |     /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:921
          | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
          |     /go/src/github.com/cockroachdb/cockroach/pkg/rpc/auth.go:60
          | google.golang.org/grpc.chainUnaryServerInterceptors.func1
          |     /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:907
          | github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler
          |     /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9085
          | google.golang.org/grpc.(*Server).processUnaryRPC
          |     /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1082
          | google.golang.org/grpc.(*Server).handleStream
          |     /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1405
          | google.golang.org/grpc.(*Server).serveStreams.func1.1
          |     /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:746
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) change replicas of r39 failed
        Wraps: (5) unexpected value: raw_bytes:"\201\002\275\373\003\010(\022\014\272\0227afe0cef\000\001\032\014\272\022e63b8d39\000\001\"\006\010\001\020\001\030\001(\0020\004:\020\010\377\377\377\377\377\377\377\377\177\020\377\377\377\377\007" timestamp:<wall_time:1610866837105858095 logical:1 > 
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errbase.opaqueWrapper (4) *errutil.withPrefix (5) *roachpb.ConditionFailedError
    kvnemesis_test.go:84: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode215830822
--- FAIL: TestKVNemesisMultiNode (0.58s)

Parameters: - TAGS= - GOFLAGS=-parallel=4 ``` make stressrace TESTS=TestKVNemesisMultiNode PKG=./pkg/kv/kvnemesis TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestKVNemesisMultiNode.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

| /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_command.go:1045 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).ChangeReplicas | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_command.go:964 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeAdminBatch | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:584 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).sendWithRangeID | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:100 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).Send | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_send.go:36 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Send | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_send.go:195 | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).Send | /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/stores.go:177 | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1 | /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:874 | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr | /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:326 | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal | /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:862 | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch | /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:900 | github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1 | /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9083 | github.com/cockroachdb/cockroach/pkg/util/tracing.ServerInterceptor.func1 | /go/src/github.com/cockroachdb/cockroach/pkg/util/tracing/grpc_interceptor.go:135 | google.golang.org/grpc.getChainUnaryHandler.func1 | /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:921 | github.com/cockroachdb/cockroach/pkg/rpc.NewServer.func1 | /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:176 | google.golang.org/grpc.getChainUnaryHandler.func1 | /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:921 | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor | /go/src/github.com/cockroachdb/cockroach/pkg/rpc/auth.go:60 | google.golang.org/grpc.chainUnaryServerInterceptors.func1 | /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:907 | github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler | /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9085 | google.golang.org/grpc.(*Server).processUnaryRPC | /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1082 | google.golang.org/grpc.(*Server).handleStream | /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1405 | google.golang.org/grpc.(*Server).serveStreams.func1.1 | /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:746 | runtime.goexit | /usr/local/go/src/runtime/asm_amd64.s:1374 Wraps: (4) attempting to send snapshot that does not contain the recipient as a replica; snapshot type: LEARNER_INITIAL, recipient: s(n4,s4):3LEARNER, desc: r35:/{Table/39-Max} [(n1,s1):1, (n3,s3):2, (n2,s2):4, next=5, gen=6] Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errbase.opaqueWrapper (4) *errutil.leafError kvnemesis_test.go:84: -- test log scope end -- test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestKVNemesisMultiNode274091238 --- FAIL: TestKVNemesisMultiNode (1.66s)

cockroachdb / cockroach

kv/kvnemesis: TestKVNemesisMultiNode failed #59092