roachtest: kv50/rangelookups/relocate/nodes=8 failed

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

https://www.cockroachlabs.com

Other

30.05k stars 3.8k forks source link

roachtest: kv50/rangelookups/relocate/nodes=8 failed #57857

Closed cockroach-teamcity closed 3 years ago

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@960b4cfc54c8b78df56e62f07d7a07b986ceacff:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n6,s6} {REMOVE_VOTER n2,s2}]: must disable replicate queue to use ChangeReplicas manually
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n6,s6} {REMOVE_VOTER n2,s2}]: must disable replicate queue to use ChangeReplicas manually
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2515122&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@3018f66208f68fb90c77e3fad01f395a8f10ca8b:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n8,s8} {REMOVE_VOTER n3,s3}]: must disable replicate queue to use ChangeReplicas manually
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n8,s8} {REMOVE_VOTER n3,s3}]: must disable replicate queue to use ChangeReplicas manually
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2519773&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@de91557c0634ab3797356d445ba39e37d45d8205:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n2,s2} {REMOVE_VOTER n4,s4}]: must disable replicate queue to use ChangeReplicas manually
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n2,s2} {REMOVE_VOTER n4,s4}]: must disable replicate queue to use ChangeReplicas manually
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2522960&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@eda9189cecbbc279f1857f6e6b992bdfd363397e:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n7,s7} {REMOVE_VOTER n5,s5}]: (n7,s7):6LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 04340a8e at applied index 3508 with error: [n7,s7],r614: cannot apply snapshot: raft group deleted
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n7,s7} {REMOVE_VOTER n5,s5}]: (n7,s7):6LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 04340a8e at applied index 3508 with error: [n7,s7],r614: cannot apply snapshot: raft group deleted
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2525484&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@46919380225dba7122130c338744b561d7eb6c56:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n8,s8} {REMOVE_VOTER n7,s7}]: (n8,s8):10LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 9c41f15b at applied index 6756 with error: [n8,s8],r170: cannot apply snapshot: raft group deleted
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n8,s8} {REMOVE_VOTER n7,s7}]: (n8,s8):10LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 9c41f15b at applied index 6756 with error: [n8,s8],r170: cannot apply snapshot: raft group deleted
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2528188&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@a782ee8d93a23fc53eedada3c51893c19f7bb41e:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n1,s1} {REMOVE_VOTER n4,s4}]: (n1,s1):14LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 690ef968 at applied index 5634 with error: [n1,s1],r298: cannot apply snapshot: [n1,s1]: canApplySnapshotLocked: cannot add placeholder, have an existing placeholder range=298 [/Table/53/1/1465051102757152768-/Table/53/1/1483479418515103744) (placeholder) (n8,s8):12
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n1,s1} {REMOVE_VOTER n4,s4}]: (n1,s1):14LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 690ef968 at applied index 5634 with error: [n1,s1],r298: cannot apply snapshot: [n1,s1]: canApplySnapshotLocked: cannot add placeholder, have an existing placeholder range=298 [/Table/53/1/1465051102757152768-/Table/53/1/1483479418515103744) (placeholder) (n8,s8):12
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2530481&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@75efa1fe6f2096adc9db474026a2b7235e53c388:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n4,s4} {REMOVE_VOTER n5,s5}]: (n4,s4):6LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 78538ead at applied index 1288 with error: [n4,s4],r249: cannot apply snapshot: raft group deleted
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n4,s4} {REMOVE_VOTER n5,s5}]: (n4,s4):6LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 78538ead at applied index 1288 with error: [n4,s4],r249: cannot apply snapshot: raft group deleted
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2531670&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@cd62a02833eeb02d14944668d09ee54806927a2b:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n5,s5} {REMOVE_VOTER n7,s7}]: (n5,s5):11LEARNER: remote couldn't accept LEARNER_INITIAL snapshot e607649b at applied index 1743 with error: [n5,s5],r163: cannot apply snapshot: raft group deleted
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n5,s5} {REMOVE_VOTER n7,s7}]: (n5,s5):11LEARNER: remote couldn't accept LEARNER_INITIAL snapshot e607649b at applied index 1743 with error: [n5,s5],r163: cannot apply snapshot: raft group deleted
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2538659&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@152c3b30ae1dcb7a7cefc6aa7fa86c2dd264bd2e:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n4,s4} {REMOVE_VOTER n2,s2}]: (n4,s4):9LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 6e53a443 at applied index 1847 with error: [n4,s4],r502: cannot apply snapshot: [n4,s4]: canApplySnapshotLocked: cannot add placeholder, have an existing placeholder range=502 [/Table/53/1/2183755417317265408-/Table/53/1/2202183733075216384) (placeholder) (n6,s6):8
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n4,s4} {REMOVE_VOTER n2,s2}]: (n4,s4):9LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 6e53a443 at applied index 1847 with error: [n4,s4],r502: cannot apply snapshot: [n4,s4]: canApplySnapshotLocked: cannot add placeholder, have an existing placeholder range=502 [/Table/53/1/2183755417317265408-/Table/53/1/2202183733075216384) (placeholder) (n6,s6):8
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2543440&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@4e3a9ccdb9cac89ae32d6248bb9b119b5bc250f7:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n1,s1} {REMOVE_VOTER n3,s3}]: (n1,s1):4LEARNER: remote couldn't accept LEARNER_INITIAL snapshot b57768b4 at applied index 1039 with error: [n1,s1],r227: cannot apply snapshot: raft group deleted
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n1,s1} {REMOVE_VOTER n3,s3}]: (n1,s1):4LEARNER: remote couldn't accept LEARNER_INITIAL snapshot b57768b4 at applied index 1039 with error: [n1,s1],r227: cannot apply snapshot: raft group deleted
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2544831&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@08c89a597a06520c30faf01965f9c74fe9b9854f:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n3,s3} {REMOVE_VOTER n7,s7}]: (n3,s3):7LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 2219c597 at applied index 3553 with error: [n3,s3],r922: cannot apply snapshot: raft group deleted
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n3,s3} {REMOVE_VOTER n7,s7}]: (n3,s3):7LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 2219c597 at applied index 3553 with error: [n3,s3],r922: cannot apply snapshot: raft group deleted
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2546130&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@08c89a597a06520c30faf01965f9c74fe9b9854f:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n5,s5} {REMOVE_VOTER n8,s8}]: (n5,s5):6LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 286443ec at applied index 153 with error: [n5,s5],r169: cannot apply snapshot: raft group deleted
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n5,s5} {REMOVE_VOTER n8,s8}]: (n5,s5):6LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 286443ec at applied index 153 with error: [n5,s5],r169: cannot apply snapshot: raft group deleted
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2547217&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@08c89a597a06520c30faf01965f9c74fe9b9854f:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n8,s8} {REMOVE_VOTER n4,s4}]: (n8,s8):4LEARNER: remote couldn't accept LEARNER_INITIAL snapshot ad97149d at applied index 2347 with error: [n8,s8],r634: cannot apply snapshot: [n8,s8]: canApplySnapshotLocked: cannot add placeholder, have an existing placeholder range=634 [/Table/53/1/8191386354409486336-/Table/53/1/8209814670167437312) (placeholder) (n5,s5):3
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n8,s8} {REMOVE_VOTER n4,s4}]: (n8,s8):4LEARNER: remote couldn't accept LEARNER_INITIAL snapshot ad97149d at applied index 2347 with error: [n8,s8],r634: cannot apply snapshot: [n8,s8]: canApplySnapshotLocked: cannot add placeholder, have an existing placeholder range=634 [/Table/53/1/8191386354409486336-/Table/53/1/8209814670167437312) (placeholder) (n5,s5):3
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2547724&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@365c5504b75c9a9260365a628a5110c48312178b:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n4,s4} {REMOVE_VOTER n5,s5}]: (n4,s4):9LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 2e4008a9 at applied index 7781 with error: [n4,s4],r655: cannot apply snapshot: [n4,s4]: canApplySnapshotLocked: cannot add placeholder, have an existing placeholder range=655 [/Table/53/1/2146898785801361408-/Table/53/1/2165327101559312384) (placeholder) (n3,s3):1
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n4,s4} {REMOVE_VOTER n5,s5}]: (n4,s4):9LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 2e4008a9 at applied index 7781 with error: [n4,s4],r655: cannot apply snapshot: [n4,s4]: canApplySnapshotLocked: cannot add placeholder, have an existing placeholder range=655 [/Table/53/1/2146898785801361408-/Table/53/1/2165327101559312384) (placeholder) (n3,s3):1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2548186&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@6d49a323b52966becfe8a2c38a1a8ccdf8ee58a1:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n8,s8} {REMOVE_VOTER n7,s7}]: (n8,s8):9LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 9ff11058 at applied index 2688 with error: [n8,s8],r896: cannot apply snapshot: raft group deleted
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n8,s8} {REMOVE_VOTER n7,s7}]: (n8,s8):9LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 9ff11058 at applied index 2688 with error: [n8,s8],r896: cannot apply snapshot: raft group deleted
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2550095&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@b93fd531b93cb010729cb73fe679cdff9388cf27:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n1,s1} {REMOVE_VOTER n2,s2}]: (n1,s1):5LEARNER: remote couldn't accept LEARNER_INITIAL snapshot d61943ef at applied index 1944 with error: [n1,s1],r608: cannot apply snapshot: raft group deleted
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n1,s1} {REMOVE_VOTER n2,s2}]: (n1,s1):5LEARNER: remote couldn't accept LEARNER_INITIAL snapshot d61943ef at applied index 1944 with error: [n1,s1],r608: cannot apply snapshot: raft group deleted
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2552340&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@d7bbe0060531063b9bee29f69bc4d23d41b84e3d:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n8,s8} {REMOVE_VOTER n7,s7}]: (n8,s8):15LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 75c8a16e at applied index 8375 with error: [n8,s8],r779: cannot apply snapshot: raft group deleted
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n8,s8} {REMOVE_VOTER n7,s7}]: (n8,s8):15LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 75c8a16e at applied index 8375 with error: [n8,s8],r779: cannot apply snapshot: raft group deleted
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2553797&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}

cockroach-teamcity commented 3 years ago

(roachtest).kv50/rangelookups/relocate/nodes=8 failed on master@6de6fca6a320c33ce0f9033b527138526d02127b:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/kv50/rangelookups/relocate/nodes=8/run_1
    cluster.go:2648,kv.go:651,kv.go:680,test_runner.go:760: monitor failure: monitor task failed: pq: while carrying out changes [{ADD_VOTER n7,s7} {REMOVE_VOTER n6,s6}]: (n7,s7):6LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 7a5a96a9 at applied index 1529 with error: [n7,s7],r70: cannot apply snapshot: [n7,s7]: canApplySnapshotLocked: cannot add placeholder, have an existing placeholder range=70 [/Table/53/1/1041199840324265984-/Table/53/1/1059628156082216960) (placeholder) (n4,s4):5
        (1) attached stack trace
          -- stack trace:
          | main.(*monitor).WaitE
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2636
          | main.(*monitor).Wait
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2644
          | main.registerKVRangeLookups.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:651
          | main.registerKVRangeLookups.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/kv.go:680
          | [...repeated from below...]
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitor).wait.func2
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2692
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1374
        Wraps: (4) monitor task failed
        Wraps: (5) pq: while carrying out changes [{ADD_VOTER n7,s7} {REMOVE_VOTER n6,s6}]: (n7,s7):6LEARNER: remote couldn't accept LEARNER_INITIAL snapshot 7a5a96a9 at applied index 1529 with error: [n7,s7],r70: cannot apply snapshot: [n7,s7]: canApplySnapshotLocked: cannot add placeholder, have an existing placeholder range=70 [/Table/53/1/1041199840324265984-/Table/53/1/1059628156082216960) (placeholder) (n4,s4):5
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *pq.Error

Artifacts: [/kv50/rangelookups/relocate/nodes=8](https://teamcity.cockroachdb.com/viewLog.html?buildId=2558315&tab=artifacts#/kv50/rangelookups/relocate/nodes=8) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Akv50%2Frangelookups%2Frelocate%2Fnodes%3D8.%2A&sort=title&restgroup=false&display=lastcommented+project) _{powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)}