search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.21k stars 3.82k forks source link

ccl/partitionccl: TestRepartitioning failed under stress [skipped] #28786

Closed cockroach-teamcity closed 5 years ago

cockroach-teamcity commented 6 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/8bd6caec6eed07ecff29801f3f11993352544387

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=845680&tab=buildLog

  [n2] 1 Scan
  [n2,s2] executing 1 requests
  [n2,s2,r183/1:/Table/88/1/{3-4}] read-only path
  [n2,s2,r183/1:/Table/88/1/{3-4}] command queue
  [n2,s2,r183/1:/Table/88/1/{3-4}] waiting for read lock
  [n2,s2,r183/1:/Table/88/1/{3-4}] read completed
   === SPAN START: ordered aggregator ===
cockroach.processorid: 1
cockroach.stat.aggregator.input.rows: 0
cockroach.stat.aggregator.mem.max: 10 KiB
cockroach.stat.aggregator.stalltime: 282µs
  [n2] accumulation complete
  [n2] Consumer sent handshake. Consuming flow scheduled: false
   === SPAN START: flow ===
  [n1,client=127.0.0.1:38578,user=root] starting (1 processors, 0 startables)
   === SPAN START: noop ===
cockroach.processorid: 2
  [n1,client=127.0.0.1:38578,user=root] execution ends
  [n1,client=127.0.0.1:38578,user=root] rows affected: 1
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] AutoCommit. err: <nil>
  [n1,client=127.0.0.1:38578,user=root] releasing 1 tables
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4874] executing ExecStmt: SET TRACING = off
  [n1,client=127.0.0.1:38578,user=root] executing: SET TRACING = off in state: NoTxn
I180818 05:39:12.473888 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3
I180818 05:39:12.659281 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
I180818 05:39:12.832072 82190 ccl/partitionccl/partition_test.go:1261  SucceedsSoon: expected to scan on n2: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
[n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
full trace:
   === SPAN START: session recording ===
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4885] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: NoTxn
   === SPAN START: sql txn ===
  [n1,client=127.0.0.1:38578,user=root] [Open pos:4885] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: Open
  [n1,client=127.0.0.1:38578,user=root] planning starts: SELECT
  [n1,client=127.0.0.1:38578,user=root] generating optimizer plan
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [client=127.0.0.1:38578,user=root,txn=25b7c1cc,n1] querying next range at /Table/2/1/87/"single col list partitioning"/3/1
  [client=127.0.0.1:38578,user=root,txn=25b7c1cc,n1] r7: sending batch 1 Get to (n1,s1):1
  [client=127.0.0.1:38578,user=root,txn=25b7c1cc,n1] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n1] 1 Get
  [n1,s1] executing 1 requests
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read-only path
  [n1,s1,r7/1:/Table/{SystemCon…-11}] command queue
  [n1,s1,r7/1:/Table/{SystemCon…-11}] waiting for read lock
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] client.Txn did AutoCommit. err: <nil>

  [n1,client=127.0.0.1:38578,user=root] added table 'data.public."single col list partitioning"' to table collection
  [n1,client=127.0.0.1:38578,user=root] optimizer plan succeeded
  [n1,client=127.0.0.1:38578,user=root] planning ends
  [n1,client=127.0.0.1:38578,user=root] checking distributability
  [n1,client=127.0.0.1:38578,user=root] distributable plan: true
  [n1,client=127.0.0.1:38578,user=root] execution starts: distributed
   === SPAN START: consuming rows ===
  [n1,client=127.0.0.1:38578,user=root] creating DistSQL plan with distributedMode=true
  [n1,client=127.0.0.1:38578,user=root] querying next range at /Table/88/1/3
  [n1,client=127.0.0.1:38578,user=root] running DistSQL plan
   === SPAN START: /cockroach.sql.distsqlrun.DistSQL/SetupFlow ===
   === SPAN START: table reader ===
cockroach.processorid: 0
cockroach.stat.tablereader.input.rows: 0
cockroach.stat.tablereader.stalltime: 2µs
  [n2] Scan /Table/88/1/3{-/#}
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [txn=8049687a,n2] querying next range at /Table/88/1/3
  [txn=8049687a,n2] r183: sending batch 1 Scan to (n2,s2):1
  [txn=8049687a,n2] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n2] 1 Scan
  [n2,s2] executing 1 requests
   === SPAN START: outbox ===
cockroach.stat.outbox.bytes_sent: ߂
cockroach.streamid: 1
   === SPAN START: [async] drain ===
  [n2] Consumer sent handshake. Consuming flow scheduled: false
   === SPAN START: flow ===
  [n1,client=127.0.0.1:38578,user=root] starting (1 processors, 0 startables)
   === SPAN START: noop ===
cockroach.processorid: 2
  [n2] Consumer sent handshake. Consuming flow scheduled: true
  [n2,s2,r183/1:/Table/88/1/{3-4}] read-only path
  [n2,s2,r183/1:/Table/88/1/{3-4}] command queue
  [n2,s2,r183/1:/Table/88/1/{3-4}] waiting for read lock
  [n2,s2,r183/1:/Table/88/1/{3-4}] read completed
   === SPAN START: ordered aggregator ===
cockroach.processorid: 1
cockroach.stat.aggregator.input.rows: 0
cockroach.stat.aggregator.mem.max: 10 KiB
cockroach.stat.aggregator.stalltime: 346µs
  [n2] accumulation complete
  [n1,client=127.0.0.1:38578,user=root] execution ends
  [n1,client=127.0.0.1:38578,user=root] rows affected: 1
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] AutoCommit. err: <nil>
  [n1,client=127.0.0.1:38578,user=root] releasing 1 tables
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4886] executing ExecStmt: SET TRACING = off
  [n1,client=127.0.0.1:38578,user=root] executing: SET TRACING = off in state: NoTxn
I180818 05:39:13.832521 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3
I180818 05:39:13.950845 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
I180818 05:39:14.296365 82190 ccl/partitionccl/partition_test.go:1261  SucceedsSoon: expected to scan on n2: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
[n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
full trace:
   === SPAN START: session recording ===
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4897] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: NoTxn
   === SPAN START: sql txn ===
  [n1,client=127.0.0.1:38578,user=root] [Open pos:4897] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: Open
  [n1,client=127.0.0.1:38578,user=root] planning starts: SELECT
  [n1,client=127.0.0.1:38578,user=root] generating optimizer plan
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [client=127.0.0.1:38578,user=root,txn=9536f489,n1] querying next range at /Table/2/1/87/"single col list partitioning"/3/1
  [client=127.0.0.1:38578,user=root,txn=9536f489,n1] r7: sending batch 1 Get to (n1,s1):1
  [client=127.0.0.1:38578,user=root,txn=9536f489,n1] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n1] 1 Get
  [n1,s1] executing 1 requests
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read-only path
  [n1,s1,r7/1:/Table/{SystemCon…-11}] command queue
  [n1,s1,r7/1:/Table/{SystemCon…-11}] waiting for read lock
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] client.Txn did AutoCommit. err: <nil>

  [n1,client=127.0.0.1:38578,user=root] added table 'data.public."single col list partitioning"' to table collection
  [n1,client=127.0.0.1:38578,user=root] optimizer plan succeeded
  [n1,client=127.0.0.1:38578,user=root] planning ends
  [n1,client=127.0.0.1:38578,user=root] checking distributability
  [n1,client=127.0.0.1:38578,user=root] distributable plan: true
  [n1,client=127.0.0.1:38578,user=root] execution starts: distributed
   === SPAN START: consuming rows ===
  [n1,client=127.0.0.1:38578,user=root] creating DistSQL plan with distributedMode=true
  [n1,client=127.0.0.1:38578,user=root] querying next range at /Table/88/1/3
  [n1,client=127.0.0.1:38578,user=root] running DistSQL plan
   === SPAN START: /cockroach.sql.distsqlrun.DistSQL/SetupFlow ===
   === SPAN START: outbox ===
cockroach.stat.outbox.bytes_sent: ޲
cockroach.streamid: 1
   === SPAN START: table reader ===
cockroach.stat.tablereader.stalltime: 2µs
cockroach.processorid: 0
cockroach.stat.tablereader.input.rows: 0
  [n2] Scan /Table/88/1/3{-/#}
   === SPAN START: [async] drain ===
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [txn=e3cc58d9,n2] querying next range at /Table/88/1/3
  [txn=e3cc58d9,n2] r183: sending batch 1 Scan to (n2,s2):1
  [txn=e3cc58d9,n2] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n2] 1 Scan
  [n2,s2] executing 1 requests
  [n2,s2,r183/1:/Table/88/1/{3-4}] read-only path
  [n2,s2,r183/1:/Table/88/1/{3-4}] command queue
  [n2,s2,r183/1:/Table/88/1/{3-4}] waiting for read lock
  [n2,s2,r183/1:/Table/88/1/{3-4}] read completed
   === SPAN START: ordered aggregator ===
cockroach.processorid: 1
cockroach.stat.aggregator.input.rows: 0
cockroach.stat.aggregator.mem.max: 10 KiB
cockroach.stat.aggregator.stalltime: 307µs
  [n2] accumulation complete
   === SPAN START: flow ===
  [n1,client=127.0.0.1:38578,user=root] starting (1 processors, 0 startables)
   === SPAN START: noop ===
cockroach.processorid: 2
  [n1,client=127.0.0.1:38578,user=root] execution ends
  [n1,client=127.0.0.1:38578,user=root] rows affected: 1
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] AutoCommit. err: <nil>
  [n1,client=127.0.0.1:38578,user=root] releasing 1 tables
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4898] executing ExecStmt: SET TRACING = off
  [n1,client=127.0.0.1:38578,user=root] executing: SET TRACING = off in state: NoTxn
I180818 05:39:14.413306 45724 server/status/runtime.go:433  [n2] runtime stats: 1.1 GiB RSS, 634 goroutines, 34 MiB/42 MiB/105 MiB GO alloc/idle/total, 120 MiB/165 MiB CGO alloc/total, 1685.55cgo/sec, 0.73/0.02 %(u/s)time, 0.02 %gc (5x)
I180818 05:39:14.849349 45726 server/status/recorder.go:652  [n2,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180818 05:39:15.296748 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4
I180818 05:39:15.541586 46101 server/status/runtime.go:433  [n3] runtime stats: 1.1 GiB RSS, 634 goroutines, 45 MiB/35 MiB/105 MiB GO alloc/idle/total, 120 MiB/165 MiB CGO alloc/total, 1693.28cgo/sec, 0.71/0.02 %(u/s)time, 0.01 %gc (4x)
I180818 05:39:15.574707 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3
I180818 05:39:15.766261 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
I180818 05:39:15.915351 82190 ccl/partitionccl/partition_test.go:1261  SucceedsSoon: expected to scan on n2: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
[n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
full trace:
   === SPAN START: session recording ===
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4915] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: NoTxn
   === SPAN START: sql txn ===
  [n1,client=127.0.0.1:38578,user=root] [Open pos:4915] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: Open
  [n1,client=127.0.0.1:38578,user=root] planning starts: SELECT
  [n1,client=127.0.0.1:38578,user=root] generating optimizer plan
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [client=127.0.0.1:38578,user=root,txn=cf99a17c,n1] querying next range at /Table/2/1/87/"single col list partitioning"/3/1
  [client=127.0.0.1:38578,user=root,txn=cf99a17c,n1] r7: sending batch 1 Get to (n1,s1):1
  [client=127.0.0.1:38578,user=root,txn=cf99a17c,n1] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n1] 1 Get
  [n1,s1] executing 1 requests
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read-only path
  [n1,s1,r7/1:/Table/{SystemCon…-11}] command queue
  [n1,s1,r7/1:/Table/{SystemCon…-11}] waiting for read lock
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] client.Txn did AutoCommit. err: <nil>

  [n1,client=127.0.0.1:38578,user=root] added table 'data.public."single col list partitioning"' to table collection
  [n1,client=127.0.0.1:38578,user=root] optimizer plan succeeded
  [n1,client=127.0.0.1:38578,user=root] planning ends
  [n1,client=127.0.0.1:38578,user=root] checking distributability
  [n1,client=127.0.0.1:38578,user=root] distributable plan: true
  [n1,client=127.0.0.1:38578,user=root] execution starts: distributed
   === SPAN START: consuming rows ===
  [n1,client=127.0.0.1:38578,user=root] creating DistSQL plan with distributedMode=true
  [n1,client=127.0.0.1:38578,user=root] querying next range at /Table/88/1/3
  [n1,client=127.0.0.1:38578,user=root] running DistSQL plan
   === SPAN START: /cockroach.sql.distsqlrun.DistSQL/SetupFlow ===
   === SPAN START: outbox ===
cockroach.stat.outbox.bytes_sent: ߅
cockroach.streamid: 1
   === SPAN START: [async] drain ===
   === SPAN START: table reader ===
cockroach.processorid: 0
cockroach.stat.tablereader.input.rows: 0
cockroach.stat.tablereader.stalltime: 1µs
  [n2] Scan /Table/88/1/3{-/#}
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [txn=cdabfeab,n2] querying next range at /Table/88/1/3
  [txn=cdabfeab,n2] r183: sending batch 1 Scan to (n2,s2):1
  [txn=cdabfeab,n2] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n2] 1 Scan
  [n2,s2] executing 1 requests
  [n2,s2,r183/1:/Table/88/1/{3-4}] read-only path
  [n2,s2,r183/1:/Table/88/1/{3-4}] command queue
  [n2,s2,r183/1:/Table/88/1/{3-4}] waiting for read lock
  [n2,s2,r183/1:/Table/88/1/{3-4}] read completed
   === SPAN START: ordered aggregator ===
cockroach.stat.aggregator.mem.max: 10 KiB
cockroach.stat.aggregator.stalltime: 299µs
cockroach.processorid: 1
cockroach.stat.aggregator.input.rows: 0
  [n2] accumulation complete
   === SPAN START: flow ===
  [n1,client=127.0.0.1:38578,user=root] starting (1 processors, 0 startables)
   === SPAN START: noop ===
cockroach.processorid: 2
  [n1,client=127.0.0.1:38578,user=root] execution ends
  [n1,client=127.0.0.1:38578,user=root] rows affected: 1
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] AutoCommit. err: <nil>
  [n1,client=127.0.0.1:38578,user=root] releasing 1 tables
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4916] executing ExecStmt: SET TRACING = off
  [n1,client=127.0.0.1:38578,user=root] executing: SET TRACING = off in state: NoTxn
I180818 05:39:16.177601 46103 server/status/recorder.go:652  [n3,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180818 05:39:16.947962 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3
I180818 05:39:16.999285 45091 gossip/gossip.go:538  [n1] gossip status (ok, 3 nodes)
gossip client (0/3 cur/max conns)
gossip server (2/3 cur/max conns, infos 6117/1246 sent/received, bytes 17455427B/401387B sent/received)
  2: 127.0.0.1:36675 (7m53s)
  3: 127.0.0.1:38777 (7m52s)
I180818 05:39:17.034043 45094 server/status/runtime.go:433  [n1] runtime stats: 1.1 GiB RSS, 635 goroutines, 40 MiB/40 MiB/105 MiB GO alloc/idle/total, 120 MiB/165 MiB CGO alloc/total, 1677.72cgo/sec, 0.70/0.02 %(u/s)time, 0.02 %gc (5x)
I180818 05:39:17.126774 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
I180818 05:39:17.264873 45096 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180818 05:39:17.501107 82190 ccl/partitionccl/partition_test.go:1261  SucceedsSoon: expected to scan on n2: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
[n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
full trace:
   === SPAN START: session recording ===
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4927] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: NoTxn
   === SPAN START: sql txn ===
  [n1,client=127.0.0.1:38578,user=root] [Open pos:4927] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: Open
  [n1,client=127.0.0.1:38578,user=root] planning starts: SELECT
  [n1,client=127.0.0.1:38578,user=root] generating optimizer plan
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [client=127.0.0.1:38578,user=root,txn=90d41f29,n1] querying next range at /Table/2/1/87/"single col list partitioning"/3/1
  [client=127.0.0.1:38578,user=root,txn=90d41f29,n1] r7: sending batch 1 Get to (n1,s1):1
  [client=127.0.0.1:38578,user=root,txn=90d41f29,n1] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n1] 1 Get
  [n1,s1] executing 1 requests
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read-only path
  [n1,s1,r7/1:/Table/{SystemCon…-11}] command queue
  [n1,s1,r7/1:/Table/{SystemCon…-11}] waiting for read lock
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] client.Txn did AutoCommit. err: <nil>

  [n1,client=127.0.0.1:38578,user=root] added table 'data.public."single col list partitioning"' to table collection
  [n1,client=127.0.0.1:38578,user=root] optimizer plan succeeded
  [n1,client=127.0.0.1:38578,user=root] planning ends
  [n1,client=127.0.0.1:38578,user=root] checking distributability
  [n1,client=127.0.0.1:38578,user=root] distributable plan: true
  [n1,client=127.0.0.1:38578,user=root] execution starts: distributed
   === SPAN START: consuming rows ===
  [n1,client=127.0.0.1:38578,user=root] creating DistSQL plan with distributedMode=true
  [n1,client=127.0.0.1:38578,user=root] querying next range at /Table/88/1/3
  [n1,client=127.0.0.1:38578,user=root] running DistSQL plan
   === SPAN START: /cockroach.sql.distsqlrun.DistSQL/SetupFlow ===
   === SPAN START: outbox ===
cockroach.stat.outbox.bytes_sent: ߀
cockroach.streamid: 1
   === SPAN START: [async] drain ===
  [n2] Consumer sent handshake. Consuming flow scheduled: false
   === SPAN START: table reader ===
cockroach.processorid: 0
cockroach.stat.tablereader.input.rows: 0
cockroach.stat.tablereader.stalltime: 1µs
  [n2] Scan /Table/88/1/3{-/#}
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [txn=45541499,n2] querying next range at /Table/88/1/3
  [txn=45541499,n2] r183: sending batch 1 Scan to (n2,s2):1
  [txn=45541499,n2] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n2] 1 Scan
  [n2,s2] executing 1 requests
  [n2,s2,r183/1:/Table/88/1/{3-4}] read-only path
  [n2,s2,r183/1:/Table/88/1/{3-4}] command queue
   === SPAN START: flow ===
  [n1,client=127.0.0.1:38578,user=root] starting (1 processors, 0 startables)
   === SPAN START: noop ===
cockroach.processorid: 2
  [n2] Consumer sent handshake. Consuming flow scheduled: true
  [n2,s2,r183/1:/Table/88/1/{3-4}] waiting for read lock
  [n2,s2,r183/1:/Table/88/1/{3-4}] read completed
   === SPAN START: ordered aggregator ===
cockroach.stat.aggregator.stalltime: 342µs
cockroach.processorid: 1
cockroach.stat.aggregator.input.rows: 0
cockroach.stat.aggregator.mem.max: 10 KiB
  [n2] accumulation complete
  [n1,client=127.0.0.1:38578,user=root] execution ends
  [n1,client=127.0.0.1:38578,user=root] rows affected: 1
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] AutoCommit. err: <nil>
  [n1,client=127.0.0.1:38578,user=root] releasing 1 tables
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4928] executing ExecStmt: SET TRACING = off
  [n1,client=127.0.0.1:38578,user=root] executing: SET TRACING = off in state: NoTxn
I180818 05:39:18.505326 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4
I180818 05:39:18.585773 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3
I180818 05:39:18.927768 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
I180818 05:39:19.238699 82190 ccl/partitionccl/partition_test.go:1261  SucceedsSoon: expected to scan on n2: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
[n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
full trace:
   === SPAN START: session recording ===
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4945] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: NoTxn
   === SPAN START: sql txn ===
  [n1,client=127.0.0.1:38578,user=root] [Open pos:4945] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: Open
  [n1,client=127.0.0.1:38578,user=root] planning starts: SELECT
  [n1,client=127.0.0.1:38578,user=root] generating optimizer plan
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [client=127.0.0.1:38578,user=root,txn=def560ea,n1] querying next range at /Table/2/1/87/"single col list partitioning"/3/1
  [client=127.0.0.1:38578,user=root,txn=def560ea,n1] r7: sending batch 1 Get to (n1,s1):1
  [client=127.0.0.1:38578,user=root,txn=def560ea,n1] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n1] 1 Get
  [n1,s1] executing 1 requests
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read-only path
  [n1,s1,r7/1:/Table/{SystemCon…-11}] command queue
  [n1,s1,r7/1:/Table/{SystemCon…-11}] waiting for read lock
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] client.Txn did AutoCommit. err: <nil>

  [n1,client=127.0.0.1:38578,user=root] added table 'data.public."single col list partitioning"' to table collection
  [n1,client=127.0.0.1:38578,user=root] optimizer plan succeeded
  [n1,client=127.0.0.1:38578,user=root] planning ends
  [n1,client=127.0.0.1:38578,user=root] checking distributability
  [n1,client=127.0.0.1:38578,user=root] distributable plan: true
  [n1,client=127.0.0.1:38578,user=root] execution starts: distributed
   === SPAN START: consuming rows ===
  [n1,client=127.0.0.1:38578,user=root] creating DistSQL plan with distributedMode=true
  [n1,client=127.0.0.1:38578,user=root] querying next range at /Table/88/1/3
  [n1,client=127.0.0.1:38578,user=root] running DistSQL plan
   === SPAN START: /cockroach.sql.distsqlrun.DistSQL/SetupFlow ===
   === SPAN START: outbox ===
cockroach.stat.outbox.bytes_sent: ް
cockroach.streamid: 1
   === SPAN START: table reader ===
cockroach.stat.tablereader.input.rows: 0
cockroach.stat.tablereader.stalltime: 2µs
cockroach.processorid: 0
  [n2] Scan /Table/88/1/3{-/#}
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [txn=f1ee5bd1,n2] querying next range at /Table/88/1/3
  [txn=f1ee5bd1,n2] r183: sending batch 1 Scan to (n2,s2):1
  [txn=f1ee5bd1,n2] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n2] 1 Scan
  [n2,s2] executing 1 requests
  [n2,s2,r183/1:/Table/88/1/{3-4}] read-only path
  [n2,s2,r183/1:/Table/88/1/{3-4}] command queue
  [n2,s2,r183/1:/Table/88/1/{3-4}] waiting for read lock
  [n2,s2,r183/1:/Table/88/1/{3-4}] read completed
   === SPAN START: ordered aggregator ===
cockroach.processorid: 1
cockroach.stat.aggregator.input.rows: 0
cockroach.stat.aggregator.mem.max: 10 KiB
cockroach.stat.aggregator.stalltime: 369µs
  [n2] accumulation complete
   === SPAN START: flow ===
  [n1,client=127.0.0.1:38578,user=root] starting (1 processors, 0 startables)
   === SPAN START: noop ===
cockroach.processorid: 2
   === SPAN START: [async] drain ===
  [n1,client=127.0.0.1:38578,user=root] execution ends
  [n1,client=127.0.0.1:38578,user=root] rows affected: 1
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] AutoCommit. err: <nil>
  [n1,client=127.0.0.1:38578,user=root] releasing 1 tables
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4946] executing ExecStmt: SET TRACING = off
  [n1,client=127.0.0.1:38578,user=root] executing: SET TRACING = off in state: NoTxn
I180818 05:39:20.239041 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3
I180818 05:39:20.439791 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
I180818 05:39:20.632307 82190 ccl/partitionccl/partition_test.go:1261  SucceedsSoon: expected to scan on n2: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
[n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
full trace:
   === SPAN START: session recording ===
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4957] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: NoTxn
   === SPAN START: sql txn ===
  [n1,client=127.0.0.1:38578,user=root] [Open pos:4957] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: Open
  [n1,client=127.0.0.1:38578,user=root] planning starts: SELECT
  [n1,client=127.0.0.1:38578,user=root] generating optimizer plan
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [client=127.0.0.1:38578,user=root,txn=6d4417ab,n1] querying next range at /Table/2/1/87/"single col list partitioning"/3/1
  [client=127.0.0.1:38578,user=root,txn=6d4417ab,n1] r7: sending batch 1 Get to (n1,s1):1
  [client=127.0.0.1:38578,user=root,txn=6d4417ab,n1] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n1] 1 Get
  [n1,s1] executing 1 requests
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read-only path
  [n1,s1,r7/1:/Table/{SystemCon…-11}] command queue
  [n1,s1,r7/1:/Table/{SystemCon…-11}] waiting for read lock
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] client.Txn did AutoCommit. err: <nil>

  [n1,client=127.0.0.1:38578,user=root] added table 'data.public."single col list partitioning"' to table collection
  [n1,client=127.0.0.1:38578,user=root] optimizer plan succeeded
  [n1,client=127.0.0.1:38578,user=root] planning ends
  [n1,client=127.0.0.1:38578,user=root] checking distributability
  [n1,client=127.0.0.1:38578,user=root] distributable plan: true
  [n1,client=127.0.0.1:38578,user=root] execution starts: distributed
   === SPAN START: consuming rows ===
  [n1,client=127.0.0.1:38578,user=root] creating DistSQL plan with distributedMode=true
  [n1,client=127.0.0.1:38578,user=root] querying next range at /Table/88/1/3
  [n1,client=127.0.0.1:38578,user=root] running DistSQL plan
   === SPAN START: /cockroach.sql.distsqlrun.DistSQL/SetupFlow ===
   === SPAN START: outbox ===
cockroach.stat.outbox.bytes_sent: ߂
cockroach.streamid: 1
   === SPAN START: [async] drain ===
   === SPAN START: table reader ===
cockroach.processorid: 0
cockroach.stat.tablereader.input.rows: 0
cockroach.stat.tablereader.stalltime: 1µs
  [n2] Scan /Table/88/1/3{-/#}
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [txn=7c295d62,n2] querying next range at /Table/88/1/3
  [txn=7c295d62,n2] r183: sending batch 1 Scan to (n2,s2):1
  [txn=7c295d62,n2] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n2] 1 Scan
  [n2,s2] executing 1 requests
  [n2,s2,r183/1:/Table/88/1/{3-4}] read-only path
  [n2,s2,r183/1:/Table/88/1/{3-4}] command queue
  [n2,s2,r183/1:/Table/88/1/{3-4}] waiting for read lock
  [n2,s2,r183/1:/Table/88/1/{3-4}] read completed
   === SPAN START: ordered aggregator ===
cockroach.processorid: 1
cockroach.stat.aggregator.input.rows: 0
cockroach.stat.aggregator.mem.max: 10 KiB
cockroach.stat.aggregator.stalltime: 321µs
  [n2] accumulation complete
   === SPAN START: flow ===
  [n1,client=127.0.0.1:38578,user=root] starting (1 processors, 0 startables)
   === SPAN START: noop ===
cockroach.processorid: 2
  [n1,client=127.0.0.1:38578,user=root] execution ends
  [n1,client=127.0.0.1:38578,user=root] rows affected: 1
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] AutoCommit. err: <nil>
  [n1,client=127.0.0.1:38578,user=root] releasing 1 tables
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4958] executing ExecStmt: SET TRACING = off
  [n1,client=127.0.0.1:38578,user=root] executing: SET TRACING = off in state: NoTxn
I180818 05:39:21.632690 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3
I180818 05:39:21.847921 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
I180818 05:39:22.052963 82190 ccl/partitionccl/partition_test.go:1261  SucceedsSoon: expected to scan on n2: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
[n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
full trace:
   === SPAN START: session recording ===
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4969] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: NoTxn
   === SPAN START: sql txn ===
  [n1,client=127.0.0.1:38578,user=root] [Open pos:4969] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: Open
  [n1,client=127.0.0.1:38578,user=root] planning starts: SELECT
  [n1,client=127.0.0.1:38578,user=root] generating optimizer plan
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [client=127.0.0.1:38578,user=root,txn=5951cbce,n1] querying next range at /Table/2/1/87/"single col list partitioning"/3/1
  [client=127.0.0.1:38578,user=root,txn=5951cbce,n1] r7: sending batch 1 Get to (n1,s1):1
  [client=127.0.0.1:38578,user=root,txn=5951cbce,n1] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n1] 1 Get
  [n1,s1] executing 1 requests
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read-only path
  [n1,s1,r7/1:/Table/{SystemCon…-11}] command queue
  [n1,s1,r7/1:/Table/{SystemCon…-11}] waiting for read lock
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] client.Txn did AutoCommit. err: <nil>

  [n1,client=127.0.0.1:38578,user=root] added table 'data.public."single col list partitioning"' to table collection
  [n1,client=127.0.0.1:38578,user=root] optimizer plan succeeded
  [n1,client=127.0.0.1:38578,user=root] planning ends
  [n1,client=127.0.0.1:38578,user=root] checking distributability
  [n1,client=127.0.0.1:38578,user=root] distributable plan: true
  [n1,client=127.0.0.1:38578,user=root] execution starts: distributed
   === SPAN START: consuming rows ===
  [n1,client=127.0.0.1:38578,user=root] creating DistSQL plan with distributedMode=true
  [n1,client=127.0.0.1:38578,user=root] querying next range at /Table/88/1/3
  [n1,client=127.0.0.1:38578,user=root] running DistSQL plan
   === SPAN START: /cockroach.sql.distsqlrun.DistSQL/SetupFlow ===
   === SPAN START: table reader ===
cockroach.processorid: 0
cockroach.stat.tablereader.input.rows: 0
cockroach.stat.tablereader.stalltime: 2µs
  [n2] Scan /Table/88/1/3{-/#}
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [txn=ae53e674,n2] querying next range at /Table/88/1/3
  [txn=ae53e674,n2] r183: sending batch 1 Scan to (n2,s2):1
  [txn=ae53e674,n2] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n2] 1 Scan
  [n2,s2] executing 1 requests
  [n2,s2,r183/1:/Table/88/1/{3-4}] read-only path
  [n2,s2,r183/1:/Table/88/1/{3-4}] command queue
  [n2,s2,r183/1:/Table/88/1/{3-4}] waiting for read lock
  [n2,s2,r183/1:/Table/88/1/{3-4}] read completed
   === SPAN START: ordered aggregator ===
cockroach.processorid: 1
cockroach.stat.aggregator.input.rows: 0
cockroach.stat.aggregator.mem.max: 10 KiB
cockroach.stat.aggregator.stalltime: 370µs
  [n2] accumulation complete
   === SPAN START: outbox ===
cockroach.stat.outbox.bytes_sent: ߄
cockroach.streamid: 1
   === SPAN START: [async] drain ===
  [n2] Consumer sent handshake. Consuming flow scheduled: false
   === SPAN START: flow ===
  [n1,client=127.0.0.1:38578,user=root] starting (1 processors, 0 startables)
   === SPAN START: noop ===
cockroach.processorid: 2
  [n1,client=127.0.0.1:38578,user=root] execution ends
  [n1,client=127.0.0.1:38578,user=root] rows affected: 1
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] AutoCommit. err: <nil>
  [n1,client=127.0.0.1:38578,user=root] releasing 1 tables
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4970] executing ExecStmt: SET TRACING = off
  [n1,client=127.0.0.1:38578,user=root] executing: SET TRACING = off in state: NoTxn
I180818 05:39:23.053414 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
I180818 05:39:23.400496 82190 ccl/partitionccl/partition_test.go:1261  SucceedsSoon: expected to scan on n2: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
[n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
full trace:
   === SPAN START: session recording ===
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4975] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: NoTxn
   === SPAN START: sql txn ===
  [n1,client=127.0.0.1:38578,user=root] [Open pos:4975] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: Open
  [n1,client=127.0.0.1:38578,user=root] planning starts: SELECT
  [n1,client=127.0.0.1:38578,user=root] generating optimizer plan
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [client=127.0.0.1:38578,user=root,txn=5de3a81c,n1] querying next range at /Table/2/1/87/"single col list partitioning"/3/1
  [client=127.0.0.1:38578,user=root,txn=5de3a81c,n1] r7: sending batch 1 Get to (n1,s1):1
  [client=127.0.0.1:38578,user=root,txn=5de3a81c,n1] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n1] 1 Get
  [n1,s1] executing 1 requests
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read-only path
  [n1,s1,r7/1:/Table/{SystemCon…-11}] command queue
  [n1,s1,r7/1:/Table/{SystemCon…-11}] waiting for read lock
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] client.Txn did AutoCommit. err: <nil>

  [n1,client=127.0.0.1:38578,user=root] added table 'data.public."single col list partitioning"' to table collection
  [n1,client=127.0.0.1:38578,user=root] optimizer plan succeeded
  [n1,client=127.0.0.1:38578,user=root] planning ends
  [n1,client=127.0.0.1:38578,user=root] checking distributability
  [n1,client=127.0.0.1:38578,user=root] distributable plan: true
  [n1,client=127.0.0.1:38578,user=root] execution starts: distributed
   === SPAN START: consuming rows ===
  [n1,client=127.0.0.1:38578,user=root] creating DistSQL plan with distributedMode=true
  [n1,client=127.0.0.1:38578,user=root] querying next range at /Table/88/1/3
  [n1,client=127.0.0.1:38578,user=root] running DistSQL plan
   === SPAN START: /cockroach.sql.distsqlrun.DistSQL/SetupFlow ===
   === SPAN START: table reader ===
cockroach.processorid: 0
cockroach.stat.tablereader.input.rows: 0
cockroach.stat.tablereader.stalltime: 2µs
   === SPAN START: outbox ===
cockroach.stat.outbox.bytes_sent: ޵
cockroach.streamid: 1
  [n2] Scan /Table/88/1/3{-/#}
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [txn=c00a7035,n2] querying next range at /Table/88/1/3
  [txn=c00a7035,n2] r183: sending batch 1 Scan to (n2,s2):1
  [txn=c00a7035,n2] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n2] 1 Scan
  [n2,s2] executing 1 requests
  [n2,s2,r183/1:/Table/88/1/{3-4}] read-only path
  [n2,s2,r183/1:/Table/88/1/{3-4}] command queue
  [n2,s2,r183/1:/Table/88/1/{3-4}] waiting for read lock
   === SPAN START: [async] drain ===
   === SPAN START: flow ===
  [n1,client=127.0.0.1:38578,user=root] starting (1 processors, 0 startables)
   === SPAN START: noop ===
cockroach.processorid: 2
  [n2] Consumer sent handshake. Consuming flow scheduled: false
  [n2] Consumer sent handshake. Consuming flow scheduled: true
  [n2,s2,r183/1:/Table/88/1/{3-4}] read completed
   === SPAN START: ordered aggregator ===
cockroach.stat.aggregator.stalltime: 488µs
cockroach.processorid: 1
cockroach.stat.aggregator.input.rows: 0
cockroach.stat.aggregator.mem.max: 10 KiB
  [n2] accumulation complete
  [n1,client=127.0.0.1:38578,user=root] execution ends
  [n1,client=127.0.0.1:38578,user=root] rows affected: 1
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] AutoCommit. err: <nil>
  [n1,client=127.0.0.1:38578,user=root] releasing 1 tables
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4976] executing ExecStmt: SET TRACING = off
  [n1,client=127.0.0.1:38578,user=root] executing: SET TRACING = off in state: NoTxn
I180818 05:39:24.375587 45721 gossip/gossip.go:538  [n2] gossip status (ok, 3 nodes)
gossip client (1/3 cur/max conns)
  1: 127.0.0.1:34311 (8m1s: infos 591/3277 sent/received, bytes 231199B/8699687B sent/received)
gossip server (0/3 cur/max conns, infos 591/3277 sent/received, bytes 231199B/8699687B sent/received)
I180818 05:39:24.402298 45724 server/status/runtime.go:433  [n2] runtime stats: 1.1 GiB RSS, 635 goroutines, 44 MiB/37 MiB/105 MiB GO alloc/idle/total, 120 MiB/165 MiB CGO alloc/total, 1681.54cgo/sec, 0.72/0.02 %(u/s)time, 0.01 %gc (4x)
I180818 05:39:24.402833 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
I180818 05:39:24.604141 82190 ccl/partitionccl/partition_test.go:1261  SucceedsSoon: expected to scan on n2: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
[n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
full trace:
   === SPAN START: session recording ===
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4981] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: NoTxn
   === SPAN START: sql txn ===
  [n1,client=127.0.0.1:38578,user=root] [Open pos:4981] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: Open
  [n1,client=127.0.0.1:38578,user=root] planning starts: SELECT
  [n1,client=127.0.0.1:38578,user=root] generating optimizer plan
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [client=127.0.0.1:38578,user=root,txn=179b3049,n1] querying next range at /Table/2/1/87/"single col list partitioning"/3/1
  [client=127.0.0.1:38578,user=root,txn=179b3049,n1] r7: sending batch 1 Get to (n1,s1):1
  [client=127.0.0.1:38578,user=root,txn=179b3049,n1] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n1] 1 Get
  [n1,s1] executing 1 requests
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read-only path
  [n1,s1,r7/1:/Table/{SystemCon…-11}] command queue
  [n1,s1,r7/1:/Table/{SystemCon…-11}] waiting for read lock
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] client.Txn did AutoCommit. err: <nil>

  [n1,client=127.0.0.1:38578,user=root] added table 'data.public."single col list partitioning"' to table collection
  [n1,client=127.0.0.1:38578,user=root] optimizer plan succeeded
  [n1,client=127.0.0.1:38578,user=root] planning ends
  [n1,client=127.0.0.1:38578,user=root] checking distributability
  [n1,client=127.0.0.1:38578,user=root] distributable plan: true
  [n1,client=127.0.0.1:38578,user=root] execution starts: distributed
   === SPAN START: consuming rows ===
  [n1,client=127.0.0.1:38578,user=root] creating DistSQL plan with distributedMode=true
  [n1,client=127.0.0.1:38578,user=root] querying next range at /Table/88/1/3
  [n1,client=127.0.0.1:38578,user=root] running DistSQL plan
   === SPAN START: /cockroach.sql.distsqlrun.DistSQL/SetupFlow ===
   === SPAN START: outbox ===
cockroach.stat.outbox.bytes_sent: ߀
cockroach.streamid: 1
   === SPAN START: table reader ===
cockroach.stat.tablereader.input.rows: 0
cockroach.stat.tablereader.stalltime: 1µs
cockroach.processorid: 0
  [n2] Scan /Table/88/1/3{-/#}
   === SPAN START: txn coordinator send ===
   === SPAN START: [async] drain ===
   === SPAN START: dist sender send ===
  [txn=a79a0a0b,n2] querying next range at /Table/88/1/3
  [txn=a79a0a0b,n2] r183: sending batch 1 Scan to (n2,s2):1
  [txn=a79a0a0b,n2] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n2] 1 Scan
  [n2,s2] executing 1 requests
  [n2,s2,r183/1:/Table/88/1/{3-4}] read-only path
  [n2,s2,r183/1:/Table/88/1/{3-4}] command queue
  [n2,s2,r183/1:/Table/88/1/{3-4}] waiting for read lock
  [n2,s2,r183/1:/Table/88/1/{3-4}] read completed
   === SPAN START: ordered aggregator ===
cockroach.processorid: 1
cockroach.stat.aggregator.input.rows: 0
cockroach.stat.aggregator.mem.max: 10 KiB
cockroach.stat.aggregator.stalltime: 285µs
  [n2] accumulation complete
   === SPAN START: flow ===
  [n1,client=127.0.0.1:38578,user=root] starting (1 processors, 0 startables)
   === SPAN START: noop ===
cockroach.processorid: 2
  [n1,client=127.0.0.1:38578,user=root] execution ends
  [n1,client=127.0.0.1:38578,user=root] rows affected: 1
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] AutoCommit. err: <nil>
  [n1,client=127.0.0.1:38578,user=root] releasing 1 tables
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4982] executing ExecStmt: SET TRACING = off
  [n1,client=127.0.0.1:38578,user=root] executing: SET TRACING = off in state: NoTxn
I180818 05:39:24.852235 45726 server/status/recorder.go:652  [n2,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180818 05:39:25.460363 46094 gossip/gossip.go:538  [n3] gossip status (ok, 3 nodes)
gossip client (1/3 cur/max conns)
  1: 127.0.0.1:34311 (8m0s: infos 666/2866 sent/received, bytes 171629B/8760446B sent/received)
gossip server (0/3 cur/max conns, infos 666/2866 sent/received, bytes 171629B/8760446B sent/received)
I180818 05:39:25.473470 46101 server/status/runtime.go:433  [n3] runtime stats: 1.1 GiB RSS, 634 goroutines, 54 MiB/29 MiB/105 MiB GO alloc/idle/total, 120 MiB/165 MiB CGO alloc/total, 1683.46cgo/sec, 0.71/0.02 %(u/s)time, 0.01 %gc (4x)
I180818 05:39:25.604540 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4
I180818 05:39:25.694616 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3
I180818 05:39:25.854677 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
I180818 05:39:26.165886 82190 ccl/partitionccl/partition_test.go:1261  SucceedsSoon: expected to scan on n2: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
[n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
full trace:
   === SPAN START: session recording ===
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:4999] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: NoTxn
   === SPAN START: sql txn ===
  [n1,client=127.0.0.1:38578,user=root] [Open pos:4999] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: Open
  [n1,client=127.0.0.1:38578,user=root] planning starts: SELECT
  [n1,client=127.0.0.1:38578,user=root] generating optimizer plan
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [client=127.0.0.1:38578,user=root,txn=fbf84152,n1] querying next range at /Table/2/1/87/"single col list partitioning"/3/1
  [client=127.0.0.1:38578,user=root,txn=fbf84152,n1] r7: sending batch 1 Get to (n1,s1):1
  [client=127.0.0.1:38578,user=root,txn=fbf84152,n1] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n1] 1 Get
  [n1,s1] executing 1 requests
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read-only path
  [n1,s1,r7/1:/Table/{SystemCon…-11}] command queue
  [n1,s1,r7/1:/Table/{SystemCon…-11}] waiting for read lock
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] client.Txn did AutoCommit. err: <nil>

  [n1,client=127.0.0.1:38578,user=root] added table 'data.public."single col list partitioning"' to table collection
  [n1,client=127.0.0.1:38578,user=root] optimizer plan succeeded
  [n1,client=127.0.0.1:38578,user=root] planning ends
  [n1,client=127.0.0.1:38578,user=root] checking distributability
  [n1,client=127.0.0.1:38578,user=root] distributable plan: true
  [n1,client=127.0.0.1:38578,user=root] execution starts: distributed
   === SPAN START: consuming rows ===
  [n1,client=127.0.0.1:38578,user=root] creating DistSQL plan with distributedMode=true
  [n1,client=127.0.0.1:38578,user=root] querying next range at /Table/88/1/3
  [n1,client=127.0.0.1:38578,user=root] running DistSQL plan
   === SPAN START: /cockroach.sql.distsqlrun.DistSQL/SetupFlow ===
   === SPAN START: outbox ===
cockroach.stat.outbox.bytes_sent: ޿
cockroach.streamid: 1
   === SPAN START: [async] drain ===
   === SPAN START: table reader ===
cockroach.processorid: 0
cockroach.stat.tablereader.input.rows: 0
cockroach.stat.tablereader.stalltime: 2µs
  [n2] Scan /Table/88/1/3{-/#}
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [txn=2fbab321,n2] querying next range at /Table/88/1/3
  [txn=2fbab321,n2] r183: sending batch 1 Scan to (n2,s2):1
  [txn=2fbab321,n2] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n2] 1 Scan
  [n2,s2] executing 1 requests
  [n2,s2,r183/1:/Table/88/1/{3-4}] read-only path
  [n2,s2,r183/1:/Table/88/1/{3-4}] command queue
  [n2,s2,r183/1:/Table/88/1/{3-4}] waiting for read lock
  [n2,s2,r183/1:/Table/88/1/{3-4}] read completed
   === SPAN START: ordered aggregator ===
cockroach.processorid: 1
cockroach.stat.aggregator.input.rows: 0
cockroach.stat.aggregator.mem.max: 10 KiB
cockroach.stat.aggregator.stalltime: 278µs
  [n2] accumulation complete
   === SPAN START: flow ===
  [n1,client=127.0.0.1:38578,user=root] starting (1 processors, 0 startables)
   === SPAN START: noop ===
cockroach.processorid: 2
  [n1,client=127.0.0.1:38578,user=root] execution ends
  [n1,client=127.0.0.1:38578,user=root] rows affected: 1
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] AutoCommit. err: <nil>
  [n1,client=127.0.0.1:38578,user=root] releasing 1 tables
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:5000] executing ExecStmt: SET TRACING = off
  [n1,client=127.0.0.1:38578,user=root] executing: SET TRACING = off in state: NoTxn
I180818 05:39:26.178173 46103 server/status/recorder.go:652  [n3,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180818 05:39:27.075247 45094 server/status/runtime.go:433  [n1] runtime stats: 1.1 GiB RSS, 634 goroutines, 45 MiB/35 MiB/105 MiB GO alloc/idle/total, 120 MiB/165 MiB CGO alloc/total, 1693.71cgo/sec, 0.72/0.02 %(u/s)time, 0.02 %gc (4x)
I180818 05:39:27.168062 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3
I180818 05:39:27.272358 45096 server/status/recorder.go:652  [n1,summaries] available memory from cgroups (8.0 EiB) exceeds system memory 16 GiB, using system memory
I180818 05:39:27.314531 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
I180818 05:39:27.666862 82190 ccl/partitionccl/partition_test.go:1261  SucceedsSoon: expected to scan on n2: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
[n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
full trace:
   === SPAN START: session recording ===
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:5011] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: NoTxn
   === SPAN START: sql txn ===
  [n1,client=127.0.0.1:38578,user=root] [Open pos:5011] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 3
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 in state: Open
  [n1,client=127.0.0.1:38578,user=root] planning starts: SELECT
  [n1,client=127.0.0.1:38578,user=root] generating optimizer plan
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [client=127.0.0.1:38578,user=root,txn=13d397c0,n1] querying next range at /Table/2/1/87/"single col list partitioning"/3/1
  [client=127.0.0.1:38578,user=root,txn=13d397c0,n1] r7: sending batch 1 Get to (n1,s1):1
  [client=127.0.0.1:38578,user=root,txn=13d397c0,n1] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n1] 1 Get
  [n1,s1] executing 1 requests
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read-only path
  [n1,s1,r7/1:/Table/{SystemCon…-11}] command queue
  [n1,s1,r7/1:/Table/{SystemCon…-11}] waiting for read lock
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] client.Txn did AutoCommit. err: <nil>

  [n1,client=127.0.0.1:38578,user=root] added table 'data.public."single col list partitioning"' to table collection
  [n1,client=127.0.0.1:38578,user=root] optimizer plan succeeded
  [n1,client=127.0.0.1:38578,user=root] planning ends
  [n1,client=127.0.0.1:38578,user=root] checking distributability
  [n1,client=127.0.0.1:38578,user=root] distributable plan: true
  [n1,client=127.0.0.1:38578,user=root] execution starts: distributed
   === SPAN START: consuming rows ===
  [n1,client=127.0.0.1:38578,user=root] creating DistSQL plan with distributedMode=true
  [n1,client=127.0.0.1:38578,user=root] querying next range at /Table/88/1/3
  [n1,client=127.0.0.1:38578,user=root] running DistSQL plan
   === SPAN START: /cockroach.sql.distsqlrun.DistSQL/SetupFlow ===
   === SPAN START: table reader ===
cockroach.processorid: 0
cockroach.stat.tablereader.input.rows: 0
cockroach.stat.tablereader.stalltime: 1µs
  [n2] Scan /Table/88/1/3{-/#}
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [txn=3a84cf82,n2] querying next range at /Table/88/1/3
  [txn=3a84cf82,n2] r183: sending batch 1 Scan to (n2,s2):1
  [txn=3a84cf82,n2] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n2] 1 Scan
  [n2,s2] executing 1 requests
  [n2,s2,r183/1:/Table/88/1/{3-4}] read-only path
  [n2,s2,r183/1:/Table/88/1/{3-4}] command queue
  [n2,s2,r183/1:/Table/88/1/{3-4}] waiting for read lock
  [n2,s2,r183/1:/Table/88/1/{3-4}] read completed
   === SPAN START: ordered aggregator ===
cockroach.processorid: 1
cockroach.stat.aggregator.input.rows: 0
cockroach.stat.aggregator.mem.max: 10 KiB
cockroach.stat.aggregator.stalltime: 271µs
  [n2] accumulation complete
   === SPAN START: outbox ===
cockroach.stat.outbox.bytes_sent: ߀
cockroach.streamid: 1
   === SPAN START: [async] drain ===
   === SPAN START: flow ===
  [n1,client=127.0.0.1:38578,user=root] starting (1 processors, 0 startables)
   === SPAN START: noop ===
cockroach.processorid: 2
  [n1,client=127.0.0.1:38578,user=root] execution ends
  [n1,client=127.0.0.1:38578,user=root] rows affected: 1
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] AutoCommit. err: <nil>
  [n1,client=127.0.0.1:38578,user=root] releasing 1 tables
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:5012] executing ExecStmt: SET TRACING = off
  [n1,client=127.0.0.1:38578,user=root] executing: SET TRACING = off in state: NoTxn
I180818 05:39:28.672210 82190 ccl/partitionccl/partition_test.go:200  query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4
I180818 05:39:28.893306 82190 ccl/partitionccl/partition_test.go:1261  SucceedsSoon: expected to scan on n3: SELECT count(*) FROM "single col list partitioning" WHERE a = 4
[n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
full trace:
   === SPAN START: session recording ===
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:5017] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 4
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 in state: NoTxn
   === SPAN START: sql txn ===
  [n1,client=127.0.0.1:38578,user=root] [Open pos:5017] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 4
  [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 in state: Open
  [n1,client=127.0.0.1:38578,user=root] planning starts: SELECT
  [n1,client=127.0.0.1:38578,user=root] generating optimizer plan
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [client=127.0.0.1:38578,user=root,txn=acdd2ae4,n1] querying next range at /Table/2/1/87/"single col list partitioning"/3/1
  [client=127.0.0.1:38578,user=root,txn=acdd2ae4,n1] r7: sending batch 1 Get to (n1,s1):1
  [client=127.0.0.1:38578,user=root,txn=acdd2ae4,n1] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n1] 1 Get
  [n1,s1] executing 1 requests
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read-only path
  [n1,s1,r7/1:/Table/{SystemCon…-11}] command queue
  [n1,s1,r7/1:/Table/{SystemCon…-11}] waiting for read lock
  [n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] client.Txn did AutoCommit. err: <nil>

  [n1,client=127.0.0.1:38578,user=root] added table 'data.public."single col list partitioning"' to table collection
  [n1,client=127.0.0.1:38578,user=root] optimizer plan succeeded
  [n1,client=127.0.0.1:38578,user=root] planning ends
  [n1,client=127.0.0.1:38578,user=root] checking distributability
  [n1,client=127.0.0.1:38578,user=root] distributable plan: true
  [n1,client=127.0.0.1:38578,user=root] execution starts: distributed
   === SPAN START: consuming rows ===
  [n1,client=127.0.0.1:38578,user=root] creating DistSQL plan with distributedMode=true
  [n1,client=127.0.0.1:38578,user=root] querying next range at /Table/88/1/4
  [n1,client=127.0.0.1:38578,user=root] running DistSQL plan
   === SPAN START: /cockroach.sql.distsqlrun.DistSQL/SetupFlow ===
   === SPAN START: outbox ===
cockroach.stat.outbox.bytes_sent: ߀
cockroach.streamid: 1
   === SPAN START: [async] drain ===
   === SPAN START: table reader ===
cockroach.stat.tablereader.input.rows: 0
cockroach.stat.tablereader.stalltime: 2µs
cockroach.processorid: 0
  [n3] Scan /Table/88/1/4{-/#}
   === SPAN START: txn coordinator send ===
   === SPAN START: dist sender send ===
  [txn=cec3a39b,n3] querying next range at /Table/88/1/4
  [txn=cec3a39b,n3] r184: sending batch 1 Scan to (n3,s3):2
  [txn=cec3a39b,n3] sending request to local server
   === SPAN START: /cockroach.roachpb.Internal/Batch ===
  [n3] 1 Scan
  [n3,s3] executing 1 requests
  [n3,s3,r184/2:/Table/88/1/{4-5}] read-only path
  [n3,s3,r184/2:/Table/88/1/{4-5}] command queue
  [n3,s3,r184/2:/Table/88/1/{4-5}] waiting for read lock
  [n3,s3,r184/2:/Table/88/1/{4-5}] read completed
   === SPAN START: ordered aggregator ===
cockroach.stat.aggregator.input.rows: 0
cockroach.stat.aggregator.mem.max: 10 KiB
cockroach.stat.aggregator.stalltime: 343µs
cockroach.processorid: 1
  [n3] accumulation complete
   === SPAN START: flow ===
  [n1,client=127.0.0.1:38578,user=root] starting (1 processors, 0 startables)
   === SPAN START: noop ===
cockroach.processorid: 2
  [n1,client=127.0.0.1:38578,user=root] execution ends
  [n1,client=127.0.0.1:38578,user=root] rows affected: 1
   === SPAN START: txn coordinator send ===
  [n1,client=127.0.0.1:38578,user=root] AutoCommit. err: <nil>
  [n1,client=127.0.0.1:38578,user=root] releasing 1 tables
  [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:5018] executing ExecStmt: SET TRACING = off
  [n1,client=127.0.0.1:38578,user=root] executing: SET TRACING = off in state: NoTxn
    --- FAIL: TestRepartitioning/single_col_list_partitioning/single_col_range_partitioning (50.72s)
        soon.go:49: condition failed to evaluate within 45s: expected to scan on n3: SELECT count(*) FROM "single col list partitioning" WHERE a = 4
            [n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
            full trace:
               === SPAN START: session recording ===
              [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:5017] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 4
              [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 in state: NoTxn
               === SPAN START: sql txn ===
              [n1,client=127.0.0.1:38578,user=root] [Open pos:5017] executing ExecStmt: SELECT count(*) FROM "single col list partitioning" WHERE a = 4
              [n1,client=127.0.0.1:38578,user=root] executing: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 in state: Open
              [n1,client=127.0.0.1:38578,user=root] planning starts: SELECT
              [n1,client=127.0.0.1:38578,user=root] generating optimizer plan
               === SPAN START: txn coordinator send ===
               === SPAN START: dist sender send ===
              [client=127.0.0.1:38578,user=root,txn=acdd2ae4,n1] querying next range at /Table/2/1/87/"single col list partitioning"/3/1
              [client=127.0.0.1:38578,user=root,txn=acdd2ae4,n1] r7: sending batch 1 Get to (n1,s1):1
              [client=127.0.0.1:38578,user=root,txn=acdd2ae4,n1] sending request to local server
               === SPAN START: /cockroach.roachpb.Internal/Batch ===
              [n1] 1 Get
              [n1,s1] executing 1 requests
              [n1,s1,r7/1:/Table/{SystemCon…-11}] read-only path
              [n1,s1,r7/1:/Table/{SystemCon…-11}] command queue
              [n1,s1,r7/1:/Table/{SystemCon…-11}] waiting for read lock
              [n1,s1,r7/1:/Table/{SystemCon…-11}] read completed
               === SPAN START: txn coordinator send ===
              [n1,client=127.0.0.1:38578,user=root] client.Txn did AutoCommit. err: <nil>

              [n1,client=127.0.0.1:38578,user=root] added table 'data.public."single col list partitioning"' to table collection
              [n1,client=127.0.0.1:38578,user=root] optimizer plan succeeded
              [n1,client=127.0.0.1:38578,user=root] planning ends
              [n1,client=127.0.0.1:38578,user=root] checking distributability
              [n1,client=127.0.0.1:38578,user=root] distributable plan: true
              [n1,client=127.0.0.1:38578,user=root] execution starts: distributed
               === SPAN START: consuming rows ===
              [n1,client=127.0.0.1:38578,user=root] creating DistSQL plan with distributedMode=true
              [n1,client=127.0.0.1:38578,user=root] querying next range at /Table/88/1/4
              [n1,client=127.0.0.1:38578,user=root] running DistSQL plan
               === SPAN START: /cockroach.sql.distsqlrun.DistSQL/SetupFlow ===
               === SPAN START: outbox ===
            cockroach.stat.outbox.bytes_sent: ߀
            cockroach.streamid: 1
               === SPAN START: [async] drain ===
               === SPAN START: table reader ===
            cockroach.stat.tablereader.input.rows: 0
            cockroach.stat.tablereader.stalltime: 2µs
            cockroach.processorid: 0
              [n3] Scan /Table/88/1/4{-/#}
               === SPAN START: txn coordinator send ===
               === SPAN START: dist sender send ===
              [txn=cec3a39b,n3] querying next range at /Table/88/1/4
              [txn=cec3a39b,n3] r184: sending batch 1 Scan to (n3,s3):2
              [txn=cec3a39b,n3] sending request to local server
               === SPAN START: /cockroach.roachpb.Internal/Batch ===
              [n3] 1 Scan
              [n3,s3] executing 1 requests
              [n3,s3,r184/2:/Table/88/1/{4-5}] read-only path
              [n3,s3,r184/2:/Table/88/1/{4-5}] command queue
              [n3,s3,r184/2:/Table/88/1/{4-5}] waiting for read lock
              [n3,s3,r184/2:/Table/88/1/{4-5}] read completed
               === SPAN START: ordered aggregator ===
            cockroach.stat.aggregator.input.rows: 0
            cockroach.stat.aggregator.mem.max: 10 KiB
            cockroach.stat.aggregator.stalltime: 343µs
            cockroach.processorid: 1
              [n3] accumulation complete
               === SPAN START: flow ===
              [n1,client=127.0.0.1:38578,user=root] starting (1 processors, 0 startables)
               === SPAN START: noop ===
            cockroach.processorid: 2
              [n1,client=127.0.0.1:38578,user=root] execution ends
              [n1,client=127.0.0.1:38578,user=root] rows affected: 1
               === SPAN START: txn coordinator send ===
              [n1,client=127.0.0.1:38578,user=root] AutoCommit. err: <nil>
              [n1,client=127.0.0.1:38578,user=root] releasing 1 tables
              [n1,client=127.0.0.1:38578,user=root] [NoTxn pos:5018] executing ExecStmt: SET TRACING = off
              [n1,client=127.0.0.1:38578,user=root] executing: SET TRACING = off in state: NoTxn
            goroutine 82190 [running]:
            runtime/debug.Stack(0xa7a358200, 0xc4232c0750, 0x3da27e0)
                /usr/local/go/src/runtime/debug/stack.go:24 +0xb5
            github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon(0x3e03c80, 0xc4225322d0, 0xc4232c0720)
                /go/src/github.com/cockroachdb/cockroach/pkg/testutils/soon.go:50 +0x172
            github.com/cockroachdb/cockroach/pkg/ccl/partitionccl.TestRepartitioning.func1(0xc4225322d0)
                /go/src/github.com/cockroachdb/cockroach/pkg/ccl/partitionccl/partition_test.go:1261 +0x35f
            testing.tRunner(0xc4225322d0, 0xc4206edf50)
                /usr/local/go/src/testing/testing.go:777 +0x16e
            created by testing.(*T).Run
                /usr/local/go/src/testing/testing.go:824 +0x565
cockroach-teamcity commented 6 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/8bd6caec6eed07ecff29801f3f11993352544387

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=846134&tab=buildLog

cockroach-teamcity commented 6 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/88fd66dea46c2fbdb24c3f225a817ea806808ffa

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=847805&tab=buildLog

cockroach-teamcity commented 6 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/c74df43ba18345c48c685e899e2f4ef9785aef34

Parameters:

TAGS=
GOFLAGS=-race

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=849178&tab=buildLog

danhhz commented 6 years ago

I can only assume that this will have the same issue as #28789 if unskipped. I'll double check though

danhhz commented 6 years ago

Yup. Verified that it hits what seems like the same issue as #28789