cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.98k stars 3.79k forks source link

sql: TestTruncateWhileColumnBackfill failed #43990

Closed cockroach-teamcity closed 2 years ago

cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@c0c1013368c9a7bfefdfc1bf5d3d204bcfa769fc:

I200115 01:42:29.119398 345459 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:40372,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 01:42:29.203891 345459 sql/event_log.go:132  [n1,client=127.0.0.1:40372,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}
I200115 01:42:29.243432 345915 sql/lease.go:921  new lease: 53("test") ver=3:1579052860.760902611,0, refcount=0
I200115 01:42:29.254961 344747 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 2}]
I200115 01:42:29.325636 344747 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 01:42:29.424543 344747 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 3}]
I200115 01:42:29.426672 346043 sql/lease.go:921  new lease: 53("test") ver=4:1579052862.745639680,0, refcount=0
I200115 01:42:29.509084 344747 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
I200115 01:42:29.709607 344745 server/status/runtime.go:498  [n1] runtime stats: 1.9 GiB RSS, 188 goroutines, 50 MiB/130 MiB/98 MiB GO alloc/idle/total, 29 MiB/73 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (1067x), 80 KiB/80 KiB (r/w)net
W200115 01:42:29.793334 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 01:42:30.126594 344747 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 01:42:30.224588 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 01:42:30.531203 344747 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 01:42:30.617321 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 01:42:30.778026 345459 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:40372,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 01:42:30.972713 345459 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:40372,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 01:42:31.089031 345459 sql/event_log.go:132  [n1,client=127.0.0.1:40372,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200115 01:42:31.109421 346617 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200115 01:42:31.182587 344465 util/stop/stopper.go:539  quiescing
W200115 01:42:31.184835 346558 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Table/SystemConfigSpan/Start: node unavailable; try another peer
W200115 01:42:31.190902 344747 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200115 01:42:31.192404 344747 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200115 01:42:31.193174 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 01:42:31.220425 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 01:42:31.236057 346617 internal/client/txn.go:558  [n1,split,s1,r29/1:/{Table/53-Max}] failure aborting transaction: node unavailable; try another peer; abort caused by: result is ambiguous (server shutdown)
E200115 01:42:31.236781 346617 storage/queue.go:1035  [n1,split,s1,r29/1:/{Table/53-Max}] unable to split [n1,s1,r29/1:/{Table/53-Max}] at key "/Table/54": split at key /Table/54 failed: result is ambiguous (server shutdown)
W200115 01:42:31.241486 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 01:42:31.243351 344673 storage/store.go:1715  [n1,s1,r29/1:/Table/5{3-4}] unable to gossip on capacity change: node unavailable; try another peer
I200115 01:42:31.244070 344673 storage/queue.go:524  [n1,s1,r29/1:/Table/5{3-4}] rate limited in MaybeAdd (split): node unavailable; try another peer
I200115 01:42:31.244184 344673 storage/queue.go:524  [n1,s1,r29/1:/Table/5{3-4}] rate limited in MaybeAdd (replicate): node unavailable; try another peer
I200115 01:42:31.247612 344675 storage/queue.go:524  [n1,s1,r30/1:/{Table/54-Max}] rate limited in MaybeAdd (merge): node unavailable; try another peer
W200115 01:42:31.262420 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 01:42:31.282990 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 01:42:31.303770 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 01:42:31.324604 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 01:42:31.345095 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 01:42:31.365783 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 01:42:31.386451 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 01:42:31.407174 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 01:42:31.427846 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 01:42:31.448468 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 01:42:31.469182 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 01:42:31.489821 344747 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
I200115 01:42:31.493007 344714 kv/transport_race.go:108  transport race promotion: ran 97 iterations on up to 1115 requests
--- FAIL: TestTruncateWhileColumnBackfill (12.04s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@0046d62b00aa9e74489d1f44125948c9af729e2b:

I200115 03:44:01.375079 326194 storage/stores.go:240  [n1] read 0 node addresses from persistent storage
I200115 03:44:01.375776 326194 server/node.go:655  [n1] connecting to gossip network to verify cluster ID...
I200115 03:44:01.375839 326194 server/node.go:675  [n1] node connected via gossip and verified as part of cluster "be300262-d3e6-4f0c-8701-870bdd17e886"
I200115 03:44:01.376303 326194 server/node.go:522  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I200115 03:44:01.376515 326194 server/server.go:2102  [n1] Could not start goroutine dumper worker due to: directory to store dumps could not be determined
I200115 03:44:01.382513 326194 server/server.go:1562  [n1] starting https server at 127.0.0.1:40075 (use: 127.0.0.1:40075)
I200115 03:44:01.382623 326194 server/server.go:1567  [n1] starting postgres server at 127.0.0.1:42119 (use: 127.0.0.1:42119)
I200115 03:44:01.382728 326194 server/server.go:1569  [n1] starting grpc server at 127.0.0.1:45755
I200115 03:44:01.382797 326194 server/server.go:1570  [n1] advertising CockroachDB node at 127.0.0.1:45755
I200115 03:44:01.436789 326369 sql/event_log.go:132  [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I200115 03:44:01.445997 325891 sql/event_log.go:132  [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:19.2-9 User:root}
I200115 03:44:01.458608 326842 sql/event_log.go:132  [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:04f53849-3b39-42a7-8e8c-86d43536bb71 User:root}
I200115 03:44:01.466655 326316 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}
I200115 03:44:01.472020 326830 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}
I200115 03:44:01.490564 326194 server/server.go:1626  [n1] done ensuring all necessary migrations have run
I200115 03:44:01.490618 326194 server/server.go:1907  [n1] serving sql connections
I200115 03:44:01.493405 327093 server/server_update.go:53  [n1] no need to upgrade, cluster already at the newest version
I200115 03:44:01.494001 327095 sql/event_log.go:132  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:45755 Attrs: Locality: ServerVersion:19.2-9 BuildTag:v20.1.0-alpha.20191118-1249-g0046d62 StartedAt:1579059841367827468 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:42119} ClusterID:be300262-d3e6-4f0c-8701-870bdd17e886 StartedAt:1579059841367827468 LastUp:1579059841367827468}
I200115 03:44:01.514194 326572 sql/event_log.go:132  [n1,client=127.0.0.1:36146,user=root] Event: "create_database", target: 52, info: {DatabaseName:t Statement:CREATE DATABASE t User:root}
I200115 03:44:01.527216 326572 sql/event_log.go:132  [n1,client=127.0.0.1:36146,user=root] Event: "create_table", target: 53, info: {TableName:t.public.test Statement:CREATE TABLE t.public.test (k INT8 PRIMARY KEY, v INT8) User:root}
I200115 03:44:01.529221 327070 storage/replica_command.go:406  [n1,split,s1,r28/1:/{Table/32-Max}] initiating a split of this range at key /Table/53 [r29] (zone config)
I200115 03:44:01.742780 326572 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:36146,user=root] publish: descID=53 (test) version=2 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 03:44:01.749469 326572 sql/event_log.go:132  [n1,client=127.0.0.1:36146,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test ADD COLUMN x DECIMAL NOT NULL DEFAULT 1.4::DECIMAL, ADD CHECK (x >= 0) User:root MutationID:1 CascadeDroppedViews:[]}
I200115 03:44:01.753782 327104 sql/lease.go:921  new lease: 53("test") ver=2:1579060146.798060490,0, refcount=0
I200115 03:44:01.763081 326572 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:36146,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 03:44:01.778626 326572 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:36146,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 03:44:01.808075 326572 sql/event_log.go:132  [n1,client=127.0.0.1:36146,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}
I200115 03:44:01.816211 326361 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 2}]
I200115 03:44:01.816715 327363 sql/lease.go:921  new lease: 53("test") ver=3:1579060146.799060490,0, refcount=0
I200115 03:44:01.870913 326361 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 03:44:01.878989 327511 sql/lease.go:921  new lease: 53("test") ver=4:1579060150.380369336,0, refcount=0
I200115 03:44:01.891343 326361 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 03:44:01.925111 326361 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 03:44:01.938666 326572 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:36146,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 03:44:02.001273 326572 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:36146,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 03:44:02.047424 326572 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:36146,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 03:44:02.056506 326572 sql/event_log.go:132  [n1,client=127.0.0.1:36146,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200115 03:44:02.058469 327846 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200115 03:44:02.079462 326361 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=6 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 03:44:02.079680 326194 util/stop/stopper.go:539  quiescing
W200115 03:44:02.079952 326322 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Table/SystemConfigSpan/Start: node unavailable; try another peer
W200115 03:44:02.081757 326361 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200115 03:44:02.081893 326361 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200115 03:44:02.081957 326361 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
--- FAIL: TestTruncateWhileColumnBackfill (0.81s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@0046d62b00aa9e74489d1f44125948c9af729e2b:

I200115 04:14:37.915257 345199 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:34354,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 04:14:38.064494 345199 sql/event_log.go:132  [n1,client=127.0.0.1:34354,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}
I200115 04:14:38.111587 345848 sql/lease.go:921  new lease: 53("test") ver=3:1579061977.065810814,0, refcount=0
I200115 04:14:38.160940 344573 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 2}]
I200115 04:14:38.227783 344573 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 04:14:38.344925 345890 sql/lease.go:921  new lease: 53("test") ver=4:1579061982.380952399,0, refcount=0
I200115 04:14:38.355387 344573 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 3}]
I200115 04:14:38.444608 344573 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 04:14:38.776931 344573 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 04:14:39.099673 344573 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 04:14:39.213408 344573 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 04:14:39.484155 344573 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 04:14:39.568804 344573 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 04:14:39.934689 344573 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 04:14:40.071861 344573 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 04:14:40.408750 344573 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 04:14:40.491540 344573 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 04:14:40.816316 344573 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 04:14:40.911848 344573 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 04:14:41.048566 345199 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:34354,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 04:14:41.255921 345199 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:34354,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 04:14:41.376253 345199 sql/event_log.go:132  [n1,client=127.0.0.1:34354,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200115 04:14:41.439219 346879 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200115 04:14:41.560469 344380 util/stop/stopper.go:539  quiescing
W200115 04:14:41.561848 347004 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Table/SystemConfigSpan/Start: node unavailable; try another peer
W200115 04:14:41.564510 344573 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200115 04:14:41.565744 344573 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200115 04:14:41.566214 344573 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
I200115 04:14:41.566862 344591 kv/transport_race.go:108  transport race promotion: ran 107 iterations on up to 1525 requests
W200115 04:14:41.587919 344573 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 04:14:41.608516 344573 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
I200115 04:14:41.615115 344589 storage/queue.go:524  [n1,s1] rate limited in MaybeAdd (gc): node unavailable; try another peer
I200115 04:14:41.615655 344589 storage/queue.go:524  [n1,s1] rate limited in MaybeAdd (merge): node unavailable; try another peer
I200115 04:14:41.615842 344589 storage/queue.go:524  [n1,s1] rate limited in MaybeAdd (split): node unavailable; try another peer
I200115 04:14:41.616044 344589 storage/queue.go:524  [n1,s1] rate limited in MaybeAdd (replicate): node unavailable; try another peer
I200115 04:14:41.616177 344589 storage/queue.go:524  [n1,s1] rate limited in MaybeAdd (replicaGC): node unavailable; try another peer
I200115 04:14:41.616263 344589 storage/queue.go:524  [n1,s1] rate limited in MaybeAdd (raftlog): node unavailable; try another peer
I200115 04:14:41.616347 344589 storage/queue.go:524  [n1,s1] rate limited in MaybeAdd (raftsnapshot): node unavailable; try another peer
I200115 04:14:41.616459 344589 storage/queue.go:524  [n1,s1] rate limited in MaybeAdd (consistencyChecker): node unavailable; try another peer
I200115 04:14:41.616532 344589 storage/queue.go:524  [n1,s1] rate limited in MaybeAdd (timeSeriesMaintenance): node unavailable; try another peer
W200115 04:14:41.627864 344524 storage/store.go:1715  [n1,s1,r29/1:/Table/5{3-4}] unable to gossip on capacity change: node unavailable; try another peer
W200115 04:14:41.629299 344573 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200115 04:14:41.631685 346879 internal/client/txn.go:558  [n1,split,s1,r29/1:/Table/5{3-4}] failure aborting transaction: node unavailable; try another peer; abort caused by: result is ambiguous (server shutdown)
E200115 04:14:41.632510 346879 storage/queue.go:1035  [n1,split,s1,r29/1:/Table/5{3-4}] unable to split [n1,s1,r29/1:/Table/5{3-4}] at key "/Table/54": split at key /Table/54 failed: result is ambiguous (server shutdown)
--- FAIL: TestTruncateWhileColumnBackfill (15.39s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
tbg commented 4 years ago

@lucy-zhang are you the right person to take a look at this? This is pretty flaky - it has prevented multiple merges that I know of today alone.

thoszhang commented 4 years ago

@spaskob Is this related to https://github.com/cockroachdb/cockroach/pull/43929?

cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@c8cc70f53643e6d06f61a26652da9440c03383e7:

I200115 17:31:51.985993 326802 storage/stores.go:240  [n1] read 0 node addresses from persistent storage
I200115 17:31:51.986089 326802 server/node.go:655  [n1] connecting to gossip network to verify cluster ID...
W200115 17:31:51.986929 326954 storage/store.go:1526  [n1,s1,r6/1:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r6: replica (n1,s1):1 not lease holder; lease holder unknown
I200115 17:31:51.987507 326802 server/node.go:675  [n1] node connected via gossip and verified as part of cluster "72624be7-4e9e-48dd-9707-56c1549a6c70"
I200115 17:31:51.987622 326802 server/node.go:522  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I200115 17:31:51.987687 326802 server/server.go:2102  [n1] Could not start goroutine dumper worker due to: directory to store dumps could not be determined
I200115 17:31:51.987742 326802 server/server.go:1562  [n1] starting https server at 127.0.0.1:36951 (use: 127.0.0.1:36951)
I200115 17:31:51.987768 326802 server/server.go:1567  [n1] starting postgres server at 127.0.0.1:44591 (use: 127.0.0.1:44591)
I200115 17:31:51.987786 326802 server/server.go:1569  [n1] starting grpc server at 127.0.0.1:36957
I200115 17:31:51.987803 326802 server/server.go:1570  [n1] advertising CockroachDB node at 127.0.0.1:36957
I200115 17:31:52.093783 326729 sql/event_log.go:132  [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I200115 17:31:52.130534 326731 sql/event_log.go:132  [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:19.2-9 User:root}
I200115 17:31:52.200261 327404 sql/event_log.go:132  [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:ec5a2220-8a8d-42f9-a0c0-f740136b6e4e User:root}
I200115 17:31:52.241913 327275 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}
I200115 17:31:52.260420 327538 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}
I200115 17:31:52.288326 326802 server/server.go:1626  [n1] done ensuring all necessary migrations have run
I200115 17:31:52.299109 326802 server/server.go:1907  [n1] serving sql connections
I200115 17:31:52.302600 327688 server/server_update.go:53  [n1] no need to upgrade, cluster already at the newest version
I200115 17:31:52.305515 327690 sql/event_log.go:132  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:36957 Attrs: Locality: ServerVersion:19.2-9 BuildTag:v20.1.0-alpha.20191118-1253-gc8cc70f StartedAt:1579109511977164898 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:44591} ClusterID:72624be7-4e9e-48dd-9707-56c1549a6c70 StartedAt:1579109511977164898 LastUp:1579109511977164898}
I200115 17:31:52.338613 327484 sql/event_log.go:132  [n1,client=127.0.0.1:34556,user=root] Event: "create_database", target: 52, info: {DatabaseName:t Statement:CREATE DATABASE t User:root}
I200115 17:31:52.348654 327484 sql/event_log.go:132  [n1,client=127.0.0.1:34556,user=root] Event: "create_table", target: 53, info: {TableName:t.public.test Statement:CREATE TABLE t.public.test (k INT8 PRIMARY KEY, v INT8) User:root}
I200115 17:31:52.350890 327824 storage/replica_command.go:406  [n1,split,s1,r28/1:/{Table/32-Max}] initiating a split of this range at key /Table/53 [r29] (zone config)
I200115 17:31:52.605371 327484 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:34556,user=root] publish: descID=53 (test) version=2 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 17:31:52.614682 327777 sql/lease.go:921  new lease: 53("test") ver=2:1579109827.303253904,0, refcount=0
I200115 17:31:52.617185 327484 sql/event_log.go:132  [n1,client=127.0.0.1:34556,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test ADD COLUMN x DECIMAL NOT NULL DEFAULT 1.4::DECIMAL, ADD CHECK (x >= 0) User:root MutationID:1 CascadeDroppedViews:[]}
I200115 17:31:52.621707 327484 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:34556,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 17:31:52.626028 327484 sql/event_log.go:132  [n1,client=127.0.0.1:34556,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}
I200115 17:31:52.629627 327852 sql/lease.go:921  new lease: 53("test") ver=3:1579109827.304253904,0, refcount=0
I200115 17:31:52.673106 326878 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 17:31:52.683066 326878 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 3}]
I200115 17:31:52.687046 328063 sql/lease.go:921  new lease: 53("test") ver=4:1579109827.305253904,0, refcount=0
I200115 17:31:52.707400 326878 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 17:31:52.747367 326878 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 17:31:52.747703 327484 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:34556,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 17:31:52.772722 327484 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:34556,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 17:31:52.793395 327484 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:34556,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 17:31:52.807185 328379 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200115 17:31:52.826473 326878 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=6 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 17:31:52.830671 327484 sql/event_log.go:132  [n1,client=127.0.0.1:34556,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200115 17:31:52.834393 326802 util/stop/stopper.go:539  quiescing
W200115 17:31:52.834516 328157 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Table/SystemConfigSpan/Start: node unavailable; try another peer
W200115 17:31:52.847559 326878 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200115 17:31:52.848075 326878 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200115 17:31:52.848350 326878 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
--- FAIL: TestTruncateWhileColumnBackfill (0.97s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
spaskob commented 4 years ago

yes, https://github.com/cockroachdb/cockroach/pull/44018 The problem comes from https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/schema_changer.go#L1008 where the Started is sometimes called on succeeded job.

On Wed, Jan 15, 2020 at 12:41 PM cockroach-teamcity < notifications@github.com> wrote:

(sql).TestTruncateWhileColumnBackfill failed https://viewLog.html?buildId=1692225&tab=buildLog on master@c8cc70f53643e6d06f61a26652da9440c03383e7 https://github.com/cockroachdb/cockroach/commits/c8cc70f53643e6d06f61a26652da9440c03383e7 :

I200115 17:31:51.985993 326802 storage/stores.go:240 [n1] read 0 node addresses from persistent storage

I200115 17:31:51.986089 326802 server/node.go:655 [n1] connecting to gossip network to verify cluster ID...

W200115 17:31:51.986929 326954 storage/store.go:1526 [n1,s1,r6/1:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r6: replica (n1,s1):1 not lease holder; lease holder unknown

I200115 17:31:51.987507 326802 server/node.go:675 [n1] node connected via gossip and verified as part of cluster "72624be7-4e9e-48dd-9707-56c1549a6c70"

I200115 17:31:51.987622 326802 server/node.go:522 [n1] node=1: started with [=] engine(s) and attributes []

I200115 17:31:51.987687 326802 server/server.go:2102 [n1] Could not start goroutine dumper worker due to: directory to store dumps could not be determined

I200115 17:31:51.987742 326802 server/server.go:1562 [n1] starting https server at 127.0.0.1:36951 (use: 127.0.0.1:36951)

I200115 17:31:51.987768 326802 server/server.go:1567 [n1] starting postgres server at 127.0.0.1:44591 (use: 127.0.0.1:44591)

I200115 17:31:51.987786 326802 server/server.go:1569 [n1] starting grpc server at 127.0.0.1:36957

I200115 17:31:51.987803 326802 server/server.go:1570 [n1] advertising CockroachDB node at 127.0.0.1:36957

I200115 17:31:52.093783 326729 sql/event_log.go:132 [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}

I200115 17:31:52.130534 326731 sql/event_log.go:132 [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:19.2-9 User:root}

I200115 17:31:52.200261 327404 sql/event_log.go:132 [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:ec5a2220-8a8d-42f9-a0c0-f740136b6e4e User:root}

I200115 17:31:52.241913 327275 sql/event_log.go:132 [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}

I200115 17:31:52.260420 327538 sql/event_log.go:132 [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}

I200115 17:31:52.288326 326802 server/server.go:1626 [n1] done ensuring all necessary migrations have run

I200115 17:31:52.299109 326802 server/server.go:1907 [n1] serving sql connections

I200115 17:31:52.302600 327688 server/server_update.go:53 [n1] no need to upgrade, cluster already at the newest version

I200115 17:31:52.305515 327690 sql/event_log.go:132 [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:36957 Attrs: Locality: ServerVersion:19.2-9 BuildTag:v20.1.0-alpha.20191118-1253-gc8cc70f StartedAt:1579109511977164898 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:44591} ClusterID:72624be7-4e9e-48dd-9707-56c1549a6c70 StartedAt:1579109511977164898 LastUp:1579109511977164898}

I200115 17:31:52.338613 327484 sql/event_log.go:132 [n1,client=127.0.0.1:34556,user=root] Event: "create_database", target: 52, info: {DatabaseName:t Statement:CREATE DATABASE t User:root}

I200115 17:31:52.348654 327484 sql/event_log.go:132 [n1,client=127.0.0.1:34556,user=root] Event: "create_table", target: 53, info: {TableName:t.public.test Statement:CREATE TABLE t.public.test (k INT8 PRIMARY KEY, v INT8) User:root}

I200115 17:31:52.350890 327824 storage/replica_command.go:406 [n1,split,s1,r28/1:/{Table/32-Max}] initiating a split of this range at key /Table/53 [r29] (zone config)

I200115 17:31:52.605371 327484 sql/sqlbase/structured.go:1475 [n1,client=127.0.0.1:34556,user=root] publish: descID=53 (test) version=2 mtime=1970-01-01 00:00:00 +0000 UTC

I200115 17:31:52.614682 327777 sql/lease.go:921 new lease: 53("test") ver=2:1579109827.303253904,0, refcount=0

I200115 17:31:52.617185 327484 sql/event_log.go:132 [n1,client=127.0.0.1:34556,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test ADD COLUMN x DECIMAL NOT NULL DEFAULT 1.4::DECIMAL, ADD CHECK (x >= 0) User:root MutationID:1 CascadeDroppedViews:[]}

I200115 17:31:52.621707 327484 sql/sqlbase/structured.go:1475 [n1,client=127.0.0.1:34556,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC

I200115 17:31:52.626028 327484 sql/event_log.go:132 [n1,client=127.0.0.1:34556,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}

I200115 17:31:52.629627 327852 sql/lease.go:921 new lease: 53("test") ver=3:1579109827.304253904,0, refcount=0

I200115 17:31:52.673106 326878 sql/sqlbase/structured.go:1475 [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC

I200115 17:31:52.683066 326878 sql/lease.go:326 [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 3}]

I200115 17:31:52.687046 328063 sql/lease.go:921 new lease: 53("test") ver=4:1579109827.305253904,0, refcount=0

I200115 17:31:52.707400 326878 sql/backfill.go:143 [n1,scExec] Running backfill for "test", v=4, m=1

W200115 17:31:52.747367 326878 sql/schema_changer.go:2033 [n1] Error executing schema change: context deadline exceeded

I200115 17:31:52.747703 327484 sql/sqlbase/structured.go:1475 [n1,client=127.0.0.1:34556,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC

I200115 17:31:52.772722 327484 sql/sqlbase/structured.go:1475 [n1,client=127.0.0.1:34556,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC

I200115 17:31:52.793395 327484 sql/sqlbase/structured.go:1475 [n1,client=127.0.0.1:34556,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC

I200115 17:31:52.807185 328379 storage/replica_command.go:406 [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)

I200115 17:31:52.826473 326878 sql/sqlbase/structured.go:1475 [n1,scExec] publish: descID=53 (test) version=6 mtime=1970-01-01 00:00:00 +0000 UTC

I200115 17:31:52.830671 327484 sql/event_log.go:132 [n1,client=127.0.0.1:34556,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}

I200115 17:31:52.834393 326802 util/stop/stopper.go:539 quiescing

W200115 17:31:52.834516 328157 storage/intentresolver/intent_resolver.go:820 failed to gc transaction record: could not GC completed transaction anchored at /Table/SystemConfigSpan/Start: node unavailable; try another peer

W200115 17:31:52.847559 326878 sql/schema_changer.go:955 [n1,scExec] waiting to update leases: node unavailable; try another peer

W200115 17:31:52.848075 326878 sql/schema_changer.go:982 [n1,scExec] while reasing schema change lease: node unavailable; try another peer

W200115 17:31:52.848350 326878 sql/schema_changer.go:2033 [n1] Error executing schema change: node unavailable; try another peer

--- FAIL: TestTruncateWhileColumnBackfill (0.97s)

schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee:

old friend vivekmenezes is friendless```

Repro

Parameters: - GOFLAGS=-json make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cockroachdb/cockroach/issues/43990?email_source=notifications&email_token=AANL63ZDWX3AVBOTBM3XCBDQ55DDFA5CNFSM4KG4OLAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJBFHXQ#issuecomment-574772190, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANL636DUP2W6UNWCLF7COLQ55DDFANCNFSM4KG4OLAA .

cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@f899d1c5b31a594f32a71ec9fb059ebcf5b9976a:

I200115 18:09:54.507323 326291 storage/stores.go:240  [n1] read 0 node addresses from persistent storage
I200115 18:09:54.507425 326291 server/node.go:655  [n1] connecting to gossip network to verify cluster ID...
I200115 18:09:54.508122 326291 server/node.go:675  [n1] node connected via gossip and verified as part of cluster "05f9d714-5f2d-4d0e-bd6e-036f1b73494c"
I200115 18:09:54.508293 326291 server/node.go:522  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I200115 18:09:54.508405 326291 server/server.go:2102  [n1] Could not start goroutine dumper worker due to: directory to store dumps could not be determined
I200115 18:09:54.508501 326291 server/server.go:1562  [n1] starting https server at 127.0.0.1:42121 (use: 127.0.0.1:42121)
I200115 18:09:54.508543 326291 server/server.go:1567  [n1] starting postgres server at 127.0.0.1:42893 (use: 127.0.0.1:42893)
I200115 18:09:54.508561 326291 server/server.go:1569  [n1] starting grpc server at 127.0.0.1:33145
I200115 18:09:54.508577 326291 server/server.go:1570  [n1] advertising CockroachDB node at 127.0.0.1:33145
I200115 18:09:54.550996 326394 sql/event_log.go:132  [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I200115 18:09:54.559448 326708 sql/event_log.go:132  [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:19.2-9 User:root}
I200115 18:09:54.595759 326967 sql/event_log.go:132  [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:1afc327c-82fb-47c7-8878-71a458adea08 User:root}
I200115 18:09:54.612585 327034 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}
I200115 18:09:54.621199 327041 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}
I200115 18:09:54.659537 326291 server/server.go:1626  [n1] done ensuring all necessary migrations have run
I200115 18:09:54.659893 326291 server/server.go:1907  [n1] serving sql connections
I200115 18:09:54.662684 327238 server/server_update.go:53  [n1] no need to upgrade, cluster already at the newest version
I200115 18:09:54.663576 327240 sql/event_log.go:132  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:33145 Attrs: Locality: ServerVersion:19.2-9 BuildTag:v20.1.0-alpha.20191118-1257-gf899d1c StartedAt:1579111794502511846 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:42893} ClusterID:05f9d714-5f2d-4d0e-bd6e-036f1b73494c StartedAt:1579111794502511846 LastUp:1579111794502511846}
I200115 18:09:54.677130 327228 sql/event_log.go:132  [n1,client=127.0.0.1:34168,user=root] Event: "create_database", target: 52, info: {DatabaseName:t Statement:CREATE DATABASE t User:root}
I200115 18:09:54.686975 327228 sql/event_log.go:132  [n1,client=127.0.0.1:34168,user=root] Event: "create_table", target: 53, info: {TableName:t.public.test Statement:CREATE TABLE t.public.test (k INT8 PRIMARY KEY, v INT8) User:root}
I200115 18:09:54.699648 327379 storage/replica_command.go:406  [n1,split,s1,r28/1:/{Table/32-Max}] initiating a split of this range at key /Table/53 [r29] (zone config)
I200115 18:09:55.059944 327228 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:34168,user=root] publish: descID=53 (test) version=2 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 18:09:55.066578 327228 sql/event_log.go:132  [n1,client=127.0.0.1:34168,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test ADD COLUMN x DECIMAL NOT NULL DEFAULT 1.4::DECIMAL, ADD CHECK (x >= 0) User:root MutationID:1 CascadeDroppedViews:[]}
I200115 18:09:55.069885 327431 sql/lease.go:921  new lease: 53("test") ver=2:1579112105.157301416,0, refcount=0
I200115 18:09:55.074825 327228 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:34168,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 18:09:55.082708 327228 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:34168,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 18:09:55.086614 327228 sql/event_log.go:132  [n1,client=127.0.0.1:34168,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}
I200115 18:09:55.089340 327542 sql/lease.go:921  new lease: 53("test") ver=3:1579112105.158301416,0, refcount=0
I200115 18:09:55.112206 326556 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 18:09:55.120740 327737 sql/lease.go:921  new lease: 53("test") ver=4:1579112109.295856090,0, refcount=0
I200115 18:09:55.128906 326556 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 18:09:55.178434 326556 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 18:09:55.178882 327228 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:34168,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 18:09:55.226086 327228 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:34168,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 18:09:55.302378 327228 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:34168,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 18:09:55.320740 327984 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200115 18:09:55.334260 327228 sql/event_log.go:132  [n1,client=127.0.0.1:34168,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200115 18:09:55.338572 326556 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=6 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 18:09:55.341602 326291 util/stop/stopper.go:539  quiescing
W200115 18:09:55.341929 326556 internal/client/txn.go:558  [n1,scExec] failure aborting transaction: node unavailable; try another peer; abort caused by: failed to mark job 521146843318222849 as successful: job-update: node unavailable; try another peer
W200115 18:09:55.342055 326556 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200115 18:09:55.342166 326556 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200115 18:09:55.342342 326556 sql/schema_changer.go:2033  [n1] Error executing schema change: failed to mark job 521146843318222849 as successful: job-update: node unavailable; try another peer
W200115 18:09:55.342631 327999 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Local/Range/Table/53/RangeDescriptor: node unavailable; try another peer
--- FAIL: TestTruncateWhileColumnBackfill (0.98s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@f899d1c5b31a594f32a71ec9fb059ebcf5b9976a:

I200115 18:44:31.644374 347745 sql/event_log.go:132  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:45671 Attrs: Locality: ServerVersion:19.2-9 BuildTag:v20.1.0-alpha.20191118-1257-gf899d1c StartedAt:1579113869897543818 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:39551} ClusterID:05ab76af-43ec-4e63-bc92-08ff19437bbb StartedAt:1579113869897543818 LastUp:1579113869897543818}
I200115 18:44:31.728120 347958 sql/event_log.go:132  [n1,client=127.0.0.1:58258,user=root] Event: "create_database", target: 52, info: {DatabaseName:t Statement:CREATE DATABASE t User:root}
I200115 18:44:31.839914 347958 sql/event_log.go:132  [n1,client=127.0.0.1:58258,user=root] Event: "create_table", target: 53, info: {TableName:t.public.test Statement:CREATE TABLE t.public.test (k INT8 PRIMARY KEY, v INT8) User:root}
I200115 18:44:31.920862 348054 storage/replica_command.go:406  [n1,split,s1,r28/1:/{Table/32-Max}] initiating a split of this range at key /Table/53 [r29] (zone config)
I200115 18:44:31.954307 347024 gossip/gossip.go:1532  [n1] node has connected to cluster via gossip
I200115 18:44:31.955259 347024 storage/stores.go:259  [n1] wrote 0 node addresses to persistent storage
I200115 18:44:36.097194 348142 storage/replica_consistency.go:256  [n1,consistencyChecker,s1,r4/1:/System{/tsd-tse}] triggering stats recomputation to resolve delta of {ContainsEstimates:1350 LastUpdateNanos:1579113870237197300 IntentAge:0 GCBytesAge:0 LiveBytes:-32675 LiveCount:-648 KeyBytes:-31352 KeyCount:-648 ValBytes:-1323 ValCount:-648 IntentBytes:0 IntentCount:0 SysBytes:0 SysCount:0}
I200115 18:44:40.056412 347215 server/status/runtime.go:498  [n1] runtime stats: 1.9 GiB RSS, 186 goroutines, 37 MiB/141 MiB/82 MiB GO alloc/idle/total, 29 MiB/76 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (1107x), 365 KiB/365 KiB (r/w)net
I200115 18:44:41.878935 347958 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:58258,user=root] publish: descID=53 (test) version=2 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 18:44:41.951419 347958 sql/event_log.go:132  [n1,client=127.0.0.1:58258,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test ADD COLUMN x DECIMAL NOT NULL DEFAULT 1.4::DECIMAL, ADD CHECK (x >= 0) User:root MutationID:1 CascadeDroppedViews:[]}
I200115 18:44:42.073840 348315 sql/lease.go:921  new lease: 53("test") ver=2:1579114169.175160635,0, refcount=0
I200115 18:44:42.167092 347958 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:58258,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 18:44:42.253867 347958 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:58258,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 18:44:42.372711 347958 sql/event_log.go:132  [n1,client=127.0.0.1:58258,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}
I200115 18:44:42.423573 348442 sql/lease.go:921  new lease: 53("test") ver=3:1579114181.823388797,0, refcount=0
I200115 18:44:42.443577 347217 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 2}]
I200115 18:44:42.519452 347217 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 18:44:42.632220 347217 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 3}]
I200115 18:44:42.633850 348552 sql/lease.go:921  new lease: 53("test") ver=4:1579114196.174717235,0, refcount=0
I200115 18:44:42.735542 347217 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 18:44:43.051471 347217 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 18:44:43.428459 347217 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 18:44:43.538718 347217 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 18:44:43.868805 347217 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 18:44:43.968755 347217 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 18:44:44.271504 347217 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 18:44:44.379748 347217 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 18:44:44.674924 347217 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 18:44:44.773481 347217 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 18:44:44.931825 347958 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:58258,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 18:44:45.170226 347958 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:58258,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 18:44:45.281997 347958 sql/event_log.go:132  [n1,client=127.0.0.1:58258,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200115 18:44:45.310015 349396 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200115 18:44:45.441352 346977 util/stop/stopper.go:539  quiescing
I200115 18:44:45.443662 347119 kv/transport_race.go:108  transport race promotion: ran 46 iterations on up to 1374 requests
W200115 18:44:45.444374 349218 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Table/SystemConfigSpan/Start: node unavailable; try another peer
W200115 18:44:45.451377 349396 internal/client/txn.go:558  [n1,split,s1,r29/1:/{Table/53-Max}] failure aborting transaction: node unavailable; try another peer; abort caused by: result is ambiguous (server shutdown)
E200115 18:44:45.453549 349396 storage/queue.go:1035  [n1,split,s1,r29/1:/{Table/53-Max}] unable to split [n1,s1,r29/1:/{Table/53-Max}] at key "/Table/54": split at key /Table/54 failed: result is ambiguous (server shutdown)
W200115 18:44:45.453749 347217 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200115 18:44:45.454758 347217 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200115 18:44:45.455030 347217 sql/schema_changer.go:2033  [n1] Error executing schema change: count-leases: failed to send RPC: sending to all 1 replicas failed; last error: (err: node unavailable; try another peer) <nil>
W200115 18:44:45.466428 347140 storage/store.go:1715  [n1,s1,r29/1:/Table/5{3-4}] unable to gossip on capacity change: node unavailable; try another peer
I200115 18:44:45.467384 347140 storage/queue.go:524  [n1,s1,r29/1:/Table/5{3-4}] rate limited in MaybeAdd (split): node unavailable; try another peer
I200115 18:44:45.467743 347140 storage/queue.go:524  [n1,s1,r29/1:/Table/5{3-4}] rate limited in MaybeAdd (replicate): node unavailable; try another peer
--- FAIL: TestTruncateWhileColumnBackfill (15.86s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@a8878e79b11321d88a9e297cd5820a43ca1b65a2:

I200115 19:01:57.561754 327117 storage/stores.go:240  [n1] read 0 node addresses from persistent storage
I200115 19:01:57.561879 327117 server/node.go:655  [n1] connecting to gossip network to verify cluster ID...
I200115 19:01:57.561938 327117 server/node.go:675  [n1] node connected via gossip and verified as part of cluster "73a3a40b-9de7-423b-b086-45c83748e5ea"
I200115 19:01:57.562098 327117 server/node.go:522  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I200115 19:01:57.562477 327117 server/server.go:2102  [n1] Could not start goroutine dumper worker due to: directory to store dumps could not be determined
I200115 19:01:57.568468 327117 server/server.go:1562  [n1] starting https server at 127.0.0.1:34307 (use: 127.0.0.1:34307)
I200115 19:01:57.568582 327117 server/server.go:1567  [n1] starting postgres server at 127.0.0.1:36415 (use: 127.0.0.1:36415)
I200115 19:01:57.568674 327117 server/server.go:1569  [n1] starting grpc server at 127.0.0.1:38923
I200115 19:01:57.568754 327117 server/server.go:1570  [n1] advertising CockroachDB node at 127.0.0.1:38923
I200115 19:01:57.623555 327170 sql/event_log.go:132  [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I200115 19:01:57.634422 327379 sql/event_log.go:132  [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:19.2-9 User:root}
I200115 19:01:57.658378 327710 sql/event_log.go:132  [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:9f055d45-1436-4d1f-ad9e-748015e0b34b User:root}
I200115 19:01:57.664828 327689 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}
I200115 19:01:57.668383 327696 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}
I200115 19:01:57.686584 327117 server/server.go:1626  [n1] done ensuring all necessary migrations have run
I200115 19:01:57.686785 327117 server/server.go:1907  [n1] serving sql connections
I200115 19:01:57.690784 327923 server/server_update.go:53  [n1] no need to upgrade, cluster already at the newest version
I200115 19:01:57.693679 327973 sql/event_log.go:132  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:38923 Attrs: Locality: ServerVersion:19.2-9 BuildTag:v20.1.0-alpha.20191118-1261-ga8878e7 StartedAt:1579114917540570139 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:36415} ClusterID:73a3a40b-9de7-423b-b086-45c83748e5ea StartedAt:1579114917540570139 LastUp:1579114917540570139}
I200115 19:01:57.722065 327983 sql/event_log.go:132  [n1,client=127.0.0.1:58570,user=root] Event: "create_database", target: 52, info: {DatabaseName:t Statement:CREATE DATABASE t User:root}
I200115 19:01:57.732166 327983 sql/event_log.go:132  [n1,client=127.0.0.1:58570,user=root] Event: "create_table", target: 53, info: {TableName:t.public.test Statement:CREATE TABLE t.public.test (k INT8 PRIMARY KEY, v INT8) User:root}
I200115 19:01:57.734630 328104 storage/replica_command.go:406  [n1,split,s1,r28/1:/{Table/32-Max}] initiating a split of this range at key /Table/53 [r29] (zone config)
I200115 19:01:57.930539 327983 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:58570,user=root] publish: descID=53 (test) version=2 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 19:01:57.939199 327983 sql/event_log.go:132  [n1,client=127.0.0.1:58570,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test ADD COLUMN x DECIMAL NOT NULL DEFAULT 1.4::DECIMAL, ADD CHECK (x >= 0) User:root MutationID:1 CascadeDroppedViews:[]}
I200115 19:01:57.947068 328144 sql/lease.go:921  new lease: 53("test") ver=2:1579115212.101631975,0, refcount=0
I200115 19:01:57.953024 327983 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:58570,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 19:01:57.988174 327983 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:58570,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 19:01:57.997934 327983 sql/event_log.go:132  [n1,client=127.0.0.1:58570,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}
I200115 19:01:57.998089 328166 sql/lease.go:921  new lease: 53("test") ver=3:1579115212.102631975,0, refcount=0
I200115 19:01:58.017447 327261 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 19:01:58.027318 328468 sql/lease.go:921  new lease: 53("test") ver=4:1579115218.770221873,0, refcount=0
I200115 19:01:58.036873 327261 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 19:01:58.062178 327261 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 19:01:58.065828 327983 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:58570,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 19:01:58.123962 327983 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:58570,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 19:01:58.148734 327983 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:58570,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 19:01:58.159605 327983 sql/event_log.go:132  [n1,client=127.0.0.1:58570,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200115 19:01:58.161509 328170 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200115 19:01:58.189893 327261 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=6 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 19:01:58.195030 327117 util/stop/stopper.go:539  quiescing
W200115 19:01:58.195412 328720 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Local/Range/Table/53/RangeDescriptor: node unavailable; try another peer
W200115 19:01:58.198845 327261 internal/client/txn.go:558  [n1,scExec] failure aborting transaction: node unavailable; try another peer; abort caused by: failed to mark job 521157076346503169 as successful: job-update: node unavailable; try another peer
W200115 19:01:58.199054 327261 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200115 19:01:58.199237 327261 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200115 19:01:58.199440 327261 sql/schema_changer.go:2033  [n1] Error executing schema change: failed to mark job 521157076346503169 as successful: job-update: node unavailable; try another peer
--- FAIL: TestTruncateWhileColumnBackfill (0.75s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@1ee57925403cf698c1e7d9505fbbca35bbc63b2a:

I200115 20:02:16.910416 326135 server/node.go:438  [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=13 KiB), ranges=28, leases=0, queries=0.00, writes=0.00, bytesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=950.00 pMax=11033.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00 pMax=0.00}
I200115 20:02:16.910528 326135 storage/stores.go:240  [n1] read 0 node addresses from persistent storage
I200115 20:02:16.910614 326135 server/node.go:655  [n1] connecting to gossip network to verify cluster ID...
I200115 20:02:16.910635 326135 server/node.go:675  [n1] node connected via gossip and verified as part of cluster "ab46c2a3-9c21-4a4a-bace-b066ee2d658f"
I200115 20:02:16.910686 326135 server/node.go:522  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I200115 20:02:16.910761 326135 server/server.go:2102  [n1] Could not start goroutine dumper worker due to: directory to store dumps could not be determined
I200115 20:02:16.910846 326135 server/server.go:1562  [n1] starting https server at 127.0.0.1:35559 (use: 127.0.0.1:35559)
I200115 20:02:16.910876 326135 server/server.go:1567  [n1] starting postgres server at 127.0.0.1:38219 (use: 127.0.0.1:38219)
I200115 20:02:16.910892 326135 server/server.go:1569  [n1] starting grpc server at 127.0.0.1:32989
I200115 20:02:16.910909 326135 server/server.go:1570  [n1] advertising CockroachDB node at 127.0.0.1:32989
I200115 20:02:16.944821 326415 sql/event_log.go:132  [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I200115 20:02:16.954607 326497 sql/event_log.go:132  [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:19.2-9 User:root}
I200115 20:02:16.973058 326765 sql/event_log.go:132  [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:2bd44036-3b30-47e9-bb98-97306fe12aa7 User:root}
I200115 20:02:16.980290 326817 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}
I200115 20:02:16.983937 326636 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}
I200115 20:02:17.000001 326135 server/server.go:1626  [n1] done ensuring all necessary migrations have run
I200115 20:02:17.000219 326135 server/server.go:1907  [n1] serving sql connections
I200115 20:02:17.003806 327034 server/server_update.go:53  [n1] no need to upgrade, cluster already at the newest version
I200115 20:02:17.006023 327036 sql/event_log.go:132  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:32989 Attrs: Locality: ServerVersion:19.2-9 BuildTag:v20.1.0-alpha.20191118-1263-g1ee5792 StartedAt:1579118536899498061 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:38219} ClusterID:ab46c2a3-9c21-4a4a-bace-b066ee2d658f StartedAt:1579118536899498061 LastUp:1579118536899498061}
I200115 20:02:17.018175 326748 sql/event_log.go:132  [n1,client=127.0.0.1:46570,user=root] Event: "create_database", target: 52, info: {DatabaseName:t Statement:CREATE DATABASE t User:root}
I200115 20:02:17.028376 326748 sql/event_log.go:132  [n1,client=127.0.0.1:46570,user=root] Event: "create_table", target: 53, info: {TableName:t.public.test Statement:CREATE TABLE t.public.test (k INT8 PRIMARY KEY, v INT8) User:root}
I200115 20:02:17.030244 326823 storage/replica_command.go:406  [n1,split,s1,r28/1:/{Table/32-Max}] initiating a split of this range at key /Table/53 [r29] (zone config)
I200115 20:02:17.214130 326748 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:46570,user=root] publish: descID=53 (test) version=2 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 20:02:17.225234 326748 sql/event_log.go:132  [n1,client=127.0.0.1:46570,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test ADD COLUMN x DECIMAL NOT NULL DEFAULT 1.4::DECIMAL, ADD CHECK (x >= 0) User:root MutationID:1 CascadeDroppedViews:[]}
I200115 20:02:17.228417 326449 sql/lease.go:921  new lease: 53("test") ver=2:1579118847.527624692,0, refcount=0
I200115 20:02:17.232016 326748 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:46570,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 20:02:17.284927 326748 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:46570,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 20:02:17.292818 326748 sql/event_log.go:132  [n1,client=127.0.0.1:46570,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}
I200115 20:02:17.295520 327398 sql/lease.go:921  new lease: 53("test") ver=3:1579118851.844657809,0, refcount=0
I200115 20:02:17.346559 326211 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 20:02:17.362857 326211 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 3}]
I200115 20:02:17.363269 327111 sql/lease.go:921  new lease: 53("test") ver=4:1579118851.845657809,0, refcount=0
I200115 20:02:17.391485 326211 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 20:02:17.413366 326211 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 20:02:17.424801 326748 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:46570,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 20:02:17.440485 326748 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:46570,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 20:02:17.458143 326748 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:46570,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 20:02:17.471804 326748 sql/event_log.go:132  [n1,client=127.0.0.1:46570,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200115 20:02:17.473292 327758 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200115 20:02:17.490286 326135 util/stop/stopper.go:539  quiescing
W200115 20:02:17.490426 327813 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Local/Range/Table/53/RangeDescriptor: node unavailable; try another peer
W200115 20:02:17.492534 326211 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200115 20:02:17.492613 326211 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200115 20:02:17.492644 326211 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
--- FAIL: TestTruncateWhileColumnBackfill (0.68s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@e7f83bf05d032e374e80f03c2355fa855d62e285:

I200115 21:26:56.073988 335106 storage/stores.go:234  [n1] read 0 node addresses from persistent storage
I200115 21:26:56.074069 335106 server/node.go:655  [n1] connecting to gossip network to verify cluster ID...
I200115 21:26:56.074114 335106 server/node.go:675  [n1] node connected via gossip and verified as part of cluster "15a0002b-dfe1-430b-9a6e-f9907c1b6c4d"
I200115 21:26:56.074598 335106 server/node.go:522  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I200115 21:26:56.074759 335106 server/server.go:2102  [n1] Could not start goroutine dumper worker due to: directory to store dumps could not be determined
I200115 21:26:56.080693 335106 server/server.go:1562  [n1] starting https server at 127.0.0.1:42011 (use: 127.0.0.1:42011)
I200115 21:26:56.081243 335106 server/server.go:1567  [n1] starting postgres server at 127.0.0.1:33989 (use: 127.0.0.1:33989)
I200115 21:26:56.081322 335106 server/server.go:1569  [n1] starting grpc server at 127.0.0.1:33313
I200115 21:26:56.081406 335106 server/server.go:1570  [n1] advertising CockroachDB node at 127.0.0.1:33313
I200115 21:26:56.127200 335021 sql/event_log.go:132  [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I200115 21:26:56.147207 335309 sql/event_log.go:132  [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:19.2-9 User:root}
I200115 21:26:56.174799 335693 sql/event_log.go:132  [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:150a96ca-b84c-4205-82a0-9a9025ba5115 User:root}
I200115 21:26:56.181628 335700 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}
I200115 21:26:56.187127 335723 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}
I200115 21:26:56.200737 335106 server/server.go:1626  [n1] done ensuring all necessary migrations have run
I200115 21:26:56.200797 335106 server/server.go:1907  [n1] serving sql connections
I200115 21:26:56.204445 335946 server/server_update.go:53  [n1] no need to upgrade, cluster already at the newest version
I200115 21:26:56.204956 335948 sql/event_log.go:132  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:33313 Attrs: Locality: ServerVersion:19.2-9 BuildTag:v20.1.0-alpha.20191118-1281-ge7f83bf StartedAt:1579123616063940978 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:33989} ClusterID:15a0002b-dfe1-430b-9a6e-f9907c1b6c4d StartedAt:1579123616063940978 LastUp:1579123616063940978}
I200115 21:26:56.242896 335811 sql/event_log.go:132  [n1,client=127.0.0.1:56074,user=root] Event: "create_database", target: 52, info: {DatabaseName:t Statement:CREATE DATABASE t User:root}
I200115 21:26:56.254714 335811 sql/event_log.go:132  [n1,client=127.0.0.1:56074,user=root] Event: "create_table", target: 53, info: {TableName:t.public.test Statement:CREATE TABLE t.public.test (k INT8 PRIMARY KEY, v INT8) User:root}
I200115 21:26:56.257887 336078 storage/replica_command.go:406  [n1,split,s1,r28/1:/{Table/32-Max}] initiating a split of this range at key /Table/53 [r29] (zone config)
I200115 21:26:56.422762 335811 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:56074,user=root] publish: descID=53 (test) version=2 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 21:26:56.426616 335811 sql/event_log.go:132  [n1,client=127.0.0.1:56074,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test ADD COLUMN x DECIMAL NOT NULL DEFAULT 1.4::DECIMAL, ADD CHECK (x >= 0) User:root MutationID:1 CascadeDroppedViews:[]}
I200115 21:26:56.428164 336140 sql/lease.go:921  new lease: 53("test") ver=2:1579123926.293183538,0, refcount=0
I200115 21:26:56.432002 335811 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:56074,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 21:26:56.439473 335811 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:56074,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 21:26:56.443115 335811 sql/event_log.go:132  [n1,client=127.0.0.1:56074,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}
I200115 21:26:56.444991 336251 sql/lease.go:921  new lease: 53("test") ver=3:1579123926.294183538,0, refcount=0
I200115 21:26:56.487461 335267 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 21:26:56.499241 336384 sql/lease.go:921  new lease: 53("test") ver=4:1579123926.295183538,0, refcount=0
I200115 21:26:56.512785 335267 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 21:26:56.545822 335267 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 21:26:56.644754 335267 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200115 21:26:56.667915 335267 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200115 21:26:56.678396 335811 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:56074,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 21:26:56.699177 335811 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:56074,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 21:26:56.714816 335811 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:56074,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200115 21:26:56.729212 336835 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200115 21:26:56.742171 335811 sql/event_log.go:132  [n1,client=127.0.0.1:56074,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200115 21:26:56.749340 335106 util/stop/stopper.go:539  quiescing
W200115 21:26:56.749684 335595 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Table/SystemConfigSpan/Start: node unavailable; try another peer
W200115 21:26:56.754956 335267 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200115 21:26:56.755135 335267 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200115 21:26:56.755368 335267 sql/schema_changer.go:2033  [n1] Error executing schema change: count-leases: node unavailable; try another peer
--- FAIL: TestTruncateWhileColumnBackfill (0.78s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@7fa19362c937ed922c1f7a6ec24c445e05cec180:

W200116 01:31:56.065412 327888 storage/store.go:1526  [n1,s1,r6/1:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r6: replica (n1,s1):1 not lease holder; lease holder unknown
I200116 01:31:56.066131 327646 storage/stores.go:234  [n1] read 0 node addresses from persistent storage
I200116 01:31:56.066613 327646 server/node.go:655  [n1] connecting to gossip network to verify cluster ID...
I200116 01:31:56.066973 327646 server/node.go:675  [n1] node connected via gossip and verified as part of cluster "c6366f3f-2fc4-49e6-8740-12a9f3377215"
I200116 01:31:56.067458 327646 server/node.go:522  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I200116 01:31:56.067926 327646 server/server.go:2102  [n1] Could not start goroutine dumper worker due to: directory to store dumps could not be determined
I200116 01:31:56.068369 327646 server/server.go:1562  [n1] starting https server at 127.0.0.1:43225 (use: 127.0.0.1:43225)
I200116 01:31:56.068717 327646 server/server.go:1567  [n1] starting postgres server at 127.0.0.1:42579 (use: 127.0.0.1:42579)
I200116 01:31:56.069050 327646 server/server.go:1569  [n1] starting grpc server at 127.0.0.1:45027
I200116 01:31:56.069402 327646 server/server.go:1570  [n1] advertising CockroachDB node at 127.0.0.1:45027
I200116 01:31:56.114014 327957 sql/event_log.go:132  [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I200116 01:31:56.134174 328108 sql/event_log.go:132  [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:19.2-9 User:root}
I200116 01:31:56.159310 328113 sql/event_log.go:132  [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:65363f23-83b1-487b-8c61-abeef0394b9b User:root}
I200116 01:31:56.177477 328421 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}
I200116 01:31:56.194275 328436 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}
I200116 01:31:56.218759 327646 server/server.go:1626  [n1] done ensuring all necessary migrations have run
I200116 01:31:56.219044 327646 server/server.go:1907  [n1] serving sql connections
I200116 01:31:56.222358 328511 sql/event_log.go:132  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:45027 Attrs: Locality: ServerVersion:19.2-9 BuildTag:v20.1.0-alpha.20191118-1287-g7fa1936 StartedAt:1579138316059577199 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:42579} ClusterID:c6366f3f-2fc4-49e6-8740-12a9f3377215 StartedAt:1579138316059577199 LastUp:1579138316059577199}
I200116 01:31:56.224744 328509 server/server_update.go:53  [n1] no need to upgrade, cluster already at the newest version
I200116 01:31:56.239565 328439 sql/event_log.go:132  [n1,client=127.0.0.1:50294,user=root] Event: "create_database", target: 52, info: {DatabaseName:t Statement:CREATE DATABASE t User:root}
I200116 01:31:56.254259 328439 sql/event_log.go:132  [n1,client=127.0.0.1:50294,user=root] Event: "create_table", target: 53, info: {TableName:t.public.test Statement:CREATE TABLE t.public.test (k INT8 PRIMARY KEY, v INT8) User:root}
I200116 01:31:56.257470 328662 storage/replica_command.go:406  [n1,split,s1,r28/1:/{Table/32-Max}] initiating a split of this range at key /Table/53 [r29] (zone config)
I200116 01:31:56.555700 328439 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:50294,user=root] publish: descID=53 (test) version=2 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 01:31:56.561512 328439 sql/event_log.go:132  [n1,client=127.0.0.1:50294,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test ADD COLUMN x DECIMAL NOT NULL DEFAULT 1.4::DECIMAL, ADD CHECK (x >= 0) User:root MutationID:1 CascadeDroppedViews:[]}
I200116 01:31:56.564365 328809 sql/lease.go:921  new lease: 53("test") ver=2:1579138617.168511304,0, refcount=0
I200116 01:31:56.569949 328439 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:50294,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 01:31:56.580162 328439 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:50294,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 01:31:56.585570 328439 sql/event_log.go:132  [n1,client=127.0.0.1:50294,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}
I200116 01:31:56.588046 328932 sql/lease.go:921  new lease: 53("test") ver=3:1579138617.169511304,0, refcount=0
I200116 01:31:56.610160 327903 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 01:31:56.623475 328827 sql/lease.go:921  new lease: 53("test") ver=4:1579138617.170511304,0, refcount=0
I200116 01:31:56.626268 327903 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 3}]
I200116 01:31:56.655455 327903 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 01:31:56.683126 327903 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 01:31:56.696978 328439 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:50294,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 01:31:56.745772 328439 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:50294,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 01:31:56.769955 328439 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:50294,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 01:31:56.791452 329347 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200116 01:31:56.794913 328439 sql/event_log.go:132  [n1,client=127.0.0.1:50294,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200116 01:31:56.815263 327646 util/stop/stopper.go:539  quiescing
W200116 01:31:56.815399 328835 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Local/Range/Table/53/RangeDescriptor: node unavailable; try another peer
W200116 01:31:56.833822 327903 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200116 01:31:56.833920 327903 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200116 01:31:56.833979 327903 sql/schema_changer.go:2033  [n1] Error executing schema change: count-leases: node unavailable; try another peer
--- FAIL: TestTruncateWhileColumnBackfill (0.88s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@e2c2be6cbef07c7f2ee6341816f81f72a206c32a:

I200116 02:34:33.248550 326366 server/node.go:438  [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=13 KiB), ranges=28, leases=0, queries=0.00, writes=0.00, bytesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=950.00 pMax=11033.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00 pMax=0.00}
I200116 02:34:33.248648 326366 storage/stores.go:234  [n1] read 0 node addresses from persistent storage
I200116 02:34:33.248727 326366 server/node.go:655  [n1] connecting to gossip network to verify cluster ID...
I200116 02:34:33.248748 326366 server/node.go:675  [n1] node connected via gossip and verified as part of cluster "c3e38ecb-e97c-4bf0-a986-07b7b8e9363e"
I200116 02:34:33.248810 326366 server/node.go:522  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I200116 02:34:33.248868 326366 server/server.go:2102  [n1] Could not start goroutine dumper worker due to: directory to store dumps could not be determined
I200116 02:34:33.248923 326366 server/server.go:1562  [n1] starting https server at 127.0.0.1:39129 (use: 127.0.0.1:39129)
I200116 02:34:33.248947 326366 server/server.go:1567  [n1] starting postgres server at 127.0.0.1:33139 (use: 127.0.0.1:33139)
I200116 02:34:33.248963 326366 server/server.go:1569  [n1] starting grpc server at 127.0.0.1:33227
I200116 02:34:33.248980 326366 server/server.go:1570  [n1] advertising CockroachDB node at 127.0.0.1:33227
I200116 02:34:33.293507 326579 sql/event_log.go:132  [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I200116 02:34:33.314717 326689 sql/event_log.go:132  [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:19.2-9 User:root}
I200116 02:34:33.330704 327112 sql/event_log.go:132  [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:f76baccf-64cf-45fe-8145-6e98cb3f5786 User:root}
I200116 02:34:33.339736 327042 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}
I200116 02:34:33.346195 327193 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}
I200116 02:34:33.366029 326366 server/server.go:1626  [n1] done ensuring all necessary migrations have run
I200116 02:34:33.366089 326366 server/server.go:1907  [n1] serving sql connections
I200116 02:34:33.368506 327377 sql/event_log.go:132  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:33227 Attrs: Locality: ServerVersion:19.2-9 BuildTag:v20.1.0-alpha.20191118-1289-ge2c2be6 StartedAt:1579142073234374612 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:33139} ClusterID:c3e38ecb-e97c-4bf0-a986-07b7b8e9363e StartedAt:1579142073234374612 LastUp:1579142073234374612}
I200116 02:34:33.368654 327375 server/server_update.go:53  [n1] no need to upgrade, cluster already at the newest version
I200116 02:34:33.410986 327201 sql/event_log.go:132  [n1,client=127.0.0.1:46060,user=root] Event: "create_database", target: 52, info: {DatabaseName:t Statement:CREATE DATABASE t User:root}
I200116 02:34:33.420515 327201 sql/event_log.go:132  [n1,client=127.0.0.1:46060,user=root] Event: "create_table", target: 53, info: {TableName:t.public.test Statement:CREATE TABLE t.public.test (k INT8 PRIMARY KEY, v INT8) User:root}
I200116 02:34:33.422034 327483 storage/replica_command.go:406  [n1,split,s1,r28/1:/{Table/32-Max}] initiating a split of this range at key /Table/53 [r29] (zone config)
I200116 02:34:33.607783 327201 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:46060,user=root] publish: descID=53 (test) version=2 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 02:34:33.615975 327201 sql/event_log.go:132  [n1,client=127.0.0.1:46060,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test ADD COLUMN x DECIMAL NOT NULL DEFAULT 1.4::DECIMAL, ADD CHECK (x >= 0) User:root MutationID:1 CascadeDroppedViews:[]}
I200116 02:34:33.619779 327473 sql/lease.go:921  new lease: 53("test") ver=2:1579142375.215623063,0, refcount=0
I200116 02:34:33.652761 327201 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:46060,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 02:34:33.659414 327201 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:46060,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 02:34:33.667435 327201 sql/event_log.go:132  [n1,client=127.0.0.1:46060,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}
I200116 02:34:33.669968 327534 sql/lease.go:921  new lease: 53("test") ver=3:1579142384.111744633,0, refcount=0
I200116 02:34:33.679061 326385 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 02:34:33.684946 326385 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 3}]
I200116 02:34:33.688086 326853 sql/lease.go:921  new lease: 53("test") ver=4:1579142384.112744633,0, refcount=0
I200116 02:34:33.710048 326385 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 02:34:33.728789 326385 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 02:34:33.737832 327201 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:46060,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 02:34:33.759317 327201 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:46060,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 02:34:33.786634 327201 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:46060,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 02:34:33.795301 327201 sql/event_log.go:132  [n1,client=127.0.0.1:46060,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200116 02:34:33.796888 327710 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200116 02:34:33.808431 326366 util/stop/stopper.go:539  quiescing
W200116 02:34:33.808518 326385 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200116 02:34:33.808557 326385 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200116 02:34:33.808580 326385 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 02:34:33.808523 328044 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Table/SystemConfigSpan/Start: node unavailable; try another peer
--- FAIL: TestTruncateWhileColumnBackfill (0.66s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@7fa19362c937ed922c1f7a6ec24c445e05cec180:

I200116 02:18:51.226764 352168 sql/lease.go:921  new lease: 53("test") ver=2:1579141432.965203113,0, refcount=0
I200116 02:18:51.294984 351862 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:54044,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 02:18:51.365333 351862 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:54044,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 02:18:51.456324 351862 sql/event_log.go:132  [n1,client=127.0.0.1:54044,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}
I200116 02:18:51.504641 352381 sql/lease.go:921  new lease: 53("test") ver=3:1579141436.632906368,0, refcount=0
I200116 02:18:51.510618 351110 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 2}]
I200116 02:18:51.583766 351110 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 02:18:51.679167 351110 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 3}]
I200116 02:18:51.689520 352469 sql/lease.go:921  new lease: 53("test") ver=4:1579141436.633906368,0, refcount=0
I200116 02:18:51.849092 351110 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 02:18:52.153956 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 02:18:52.239427 352433 storage/replica_consistency.go:256  [n1,consistencyChecker,s1,r4/1:/System{/tsd-tse}] triggering stats recomputation to resolve delta of {ContainsEstimates:2700 LastUpdateNanos:1579141129959302785 IntentAge:0 GCBytesAge:0 LiveBytes:-7484 LiveCount:-648 KeyBytes:-31352 KeyCount:-648 ValBytes:23868 ValCount:-648 IntentBytes:0 IntentCount:0 SysBytes:0 SysCount:0}
I200116 02:18:52.446240 351110 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 02:18:52.534327 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 02:18:52.854320 351110 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 02:18:52.930269 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 02:18:53.222844 351110 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 02:18:53.297338 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 02:18:53.584152 351110 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 02:18:53.663499 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 02:18:53.918272 351862 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:54044,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 02:18:54.108409 351862 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:54044,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 02:18:54.240748 351862 sql/event_log.go:132  [n1,client=127.0.0.1:54044,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200116 02:18:54.289266 353345 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200116 02:18:54.464579 350815 util/stop/stopper.go:539  quiescing
W200116 02:18:54.466572 353387 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Table/15/1/521242971598520321/0: node unavailable; try another peer
W200116 02:18:54.473737 351110 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200116 02:18:54.475374 351110 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200116 02:18:54.477456 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: count-leases: node unavailable; try another peer
W200116 02:18:54.499821 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 02:18:54.521451 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 02:18:54.542312 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 02:18:54.563198 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 02:18:54.584054 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 02:18:54.605109 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 02:18:54.625963 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 02:18:54.647296 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 02:18:54.667954 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 02:18:54.688843 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 02:18:54.709871 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 02:18:54.730716 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 02:18:54.751776 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 02:18:54.772519 351110 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
I200116 02:18:54.786283 350938 kv/transport_race.go:108  transport race promotion: ran 40 iterations on up to 1349 requests
--- FAIL: TestTruncateWhileColumnBackfill (16.13s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@e2c2be6cbef07c7f2ee6341816f81f72a206c32a:

W200116 06:20:33.197828 326499 storage/store.go:1526  [n1,s1,r6/1:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r6: replica (n1,s1):1 not lease holder; lease holder unknown
I200116 06:20:33.198074 326276 storage/stores.go:234  [n1] read 0 node addresses from persistent storage
I200116 06:20:33.198354 326276 server/node.go:655  [n1] connecting to gossip network to verify cluster ID...
I200116 06:20:33.198640 326276 server/node.go:675  [n1] node connected via gossip and verified as part of cluster "e14a632f-0038-4881-be2d-de370f8e82e8"
I200116 06:20:33.198996 326276 server/node.go:522  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I200116 06:20:33.199361 326276 server/server.go:2102  [n1] Could not start goroutine dumper worker due to: directory to store dumps could not be determined
I200116 06:20:33.199673 326276 server/server.go:1562  [n1] starting https server at 127.0.0.1:34535 (use: 127.0.0.1:34535)
I200116 06:20:33.199705 326276 server/server.go:1567  [n1] starting postgres server at 127.0.0.1:40517 (use: 127.0.0.1:40517)
I200116 06:20:33.199721 326276 server/server.go:1569  [n1] starting grpc server at 127.0.0.1:41711
I200116 06:20:33.199736 326276 server/server.go:1570  [n1] advertising CockroachDB node at 127.0.0.1:41711
W200116 06:20:33.249865 326499 storage/store.go:1526  [n1,s1,r6/1:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r6: replica (n1,s1):1 not lease holder; lease holder unknown
I200116 06:20:33.312984 326522 sql/event_log.go:132  [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I200116 06:20:33.324000 326664 sql/event_log.go:132  [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:19.2-9 User:root}
I200116 06:20:33.340604 326936 sql/event_log.go:132  [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:e302a628-ae5c-45b4-b898-b6d9a52931ce User:root}
I200116 06:20:33.352315 327002 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}
I200116 06:20:33.357632 327009 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}
I200116 06:20:33.376791 326276 server/server.go:1626  [n1] done ensuring all necessary migrations have run
I200116 06:20:33.377004 326276 server/server.go:1907  [n1] serving sql connections
I200116 06:20:33.380119 327200 server/server_update.go:53  [n1] no need to upgrade, cluster already at the newest version
I200116 06:20:33.383260 327202 sql/event_log.go:132  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:41711 Attrs: Locality: ServerVersion:19.2-9 BuildTag:v20.1.0-alpha.20191118-1289-ge2c2be6 StartedAt:1579155633193009508 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:40517} ClusterID:e14a632f-0038-4881-be2d-de370f8e82e8 StartedAt:1579155633193009508 LastUp:1579155633193009508}
I200116 06:20:33.402011 326712 sql/event_log.go:132  [n1,client=127.0.0.1:47634,user=root] Event: "create_database", target: 52, info: {DatabaseName:t Statement:CREATE DATABASE t User:root}
I200116 06:20:33.413206 326712 sql/event_log.go:132  [n1,client=127.0.0.1:47634,user=root] Event: "create_table", target: 53, info: {TableName:t.public.test Statement:CREATE TABLE t.public.test (k INT8 PRIMARY KEY, v INT8) User:root}
I200116 06:20:33.415830 327338 storage/replica_command.go:406  [n1,split,s1,r28/1:/{Table/32-Max}] initiating a split of this range at key /Table/53 [r29] (zone config)
I200116 06:20:33.718691 326712 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:47634,user=root] publish: descID=53 (test) version=2 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 06:20:33.723993 326712 sql/event_log.go:132  [n1,client=127.0.0.1:47634,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test ADD COLUMN x DECIMAL NOT NULL DEFAULT 1.4::DECIMAL, ADD CHECK (x >= 0) User:root MutationID:1 CascadeDroppedViews:[]}
I200116 06:20:33.727635 327445 sql/lease.go:921  new lease: 53("test") ver=2:1579155944.572251187,0, refcount=0
I200116 06:20:33.745136 326712 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:47634,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 06:20:33.751030 326712 sql/event_log.go:132  [n1,client=127.0.0.1:47634,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}
I200116 06:20:33.753911 327371 sql/lease.go:921  new lease: 53("test") ver=3:1579155944.573251187,0, refcount=0
I200116 06:20:33.768806 326387 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 06:20:33.774570 326387 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 3}]
I200116 06:20:33.776010 327552 sql/lease.go:921  new lease: 53("test") ver=4:1579155947.663078073,0, refcount=0
I200116 06:20:33.808906 326387 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 06:20:33.845016 326387 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 06:20:33.854562 326712 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:47634,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 06:20:33.879908 326712 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:47634,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 06:20:33.928339 326712 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:47634,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 06:20:33.941248 327933 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200116 06:20:33.947594 326712 sql/event_log.go:132  [n1,client=127.0.0.1:47634,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200116 06:20:33.973121 326276 util/stop/stopper.go:539  quiescing
W200116 06:20:33.973299 327836 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Local/Range/Table/53/RangeDescriptor: node unavailable; try another peer
W200116 06:20:33.974150 326387 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200116 06:20:33.974717 326387 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200116 06:20:33.975165 326387 sql/schema_changer.go:2033  [n1] Error executing schema change: count-leases: node unavailable; try another peer
--- FAIL: TestTruncateWhileColumnBackfill (0.88s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@e2c2be6cbef07c7f2ee6341816f81f72a206c32a:

I200116 06:51:21.931657 346430 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:36498,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 06:51:22.006059 346430 sql/event_log.go:132  [n1,client=127.0.0.1:36498,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}
I200116 06:51:22.024644 345684 server/status/runtime.go:498  [n1] runtime stats: 1.8 GiB RSS, 199 goroutines, 42 MiB/137 MiB/81 MiB GO alloc/idle/total, 30 MiB/76 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (1113x), 448 KiB/448 KiB (r/w)net
I200116 06:51:22.065142 346818 sql/lease.go:921  new lease: 53("test") ver=3:1579157775.295926374,0, refcount=0
I200116 06:51:22.072676 345702 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 2}]
I200116 06:51:22.167076 345702 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 06:51:22.272624 345702 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 3}]
I200116 06:51:22.282671 347013 sql/lease.go:921  new lease: 53("test") ver=4:1579157796.502083577,0, refcount=0
I200116 06:51:22.429756 345702 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 06:51:22.724013 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 06:51:23.102832 345702 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 06:51:23.176866 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 06:51:23.419016 345702 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 06:51:23.497080 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 06:51:23.649482 346430 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:36498,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 06:51:23.835435 346430 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:36498,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 06:51:23.932198 346430 sql/event_log.go:132  [n1,client=127.0.0.1:36498,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200116 06:51:23.946576 347592 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200116 06:51:24.251072 345502 util/stop/stopper.go:539  quiescing
W200116 06:51:24.252044 347393 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Table/SystemConfigSpan/Start: node unavailable; try another peer
W200116 06:51:24.257935 345702 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200116 06:51:24.258786 345702 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200116 06:51:24.259054 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: count-leases: node unavailable; try another peer
W200116 06:51:24.268398 345661 storage/store.go:1715  [n1,s1,r29/1:/Table/5{3-4}] unable to gossip on capacity change: node unavailable; try another peer
I200116 06:51:24.269018 345661 storage/queue.go:524  [n1,s1,r29/1:/Table/5{3-4}] rate limited in MaybeAdd (split): node unavailable; try another peer
I200116 06:51:24.269209 345661 storage/queue.go:524  [n1,s1,r29/1:/Table/5{3-4}] rate limited in MaybeAdd (replicate): node unavailable; try another peer
I200116 06:51:24.273785 345659 storage/queue.go:524  [n1,s1,r30/1:/{Table/54-Max}] rate limited in MaybeAdd (merge): node unavailable; try another peer
W200116 06:51:24.274733 347592 internal/client/txn.go:558  [n1,split,s1,r29/1:/Table/5{3-4}] failure aborting transaction: node unavailable; try another peer; abort caused by: result is ambiguous (server shutdown)
E200116 06:51:24.275388 347592 storage/queue.go:1035  [n1,split,s1,r29/1:/Table/5{3-4}] unable to split [n1,s1,r29/1:/Table/5{3-4}] at key "/Table/54": split at key /Table/54 failed: result is ambiguous (server shutdown)
W200116 06:51:24.280567 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 06:51:24.301364 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 06:51:24.322231 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 06:51:24.343109 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 06:51:24.364250 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 06:51:24.384690 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 06:51:24.405649 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 06:51:24.426335 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 06:51:24.447411 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 06:51:24.468154 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 06:51:24.488783 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 06:51:24.509515 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 06:51:24.530186 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 06:51:24.550950 345702 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
I200116 06:51:24.561917 345704 kv/transport_race.go:108  transport race promotion: ran 24 iterations on up to 1002 requests
--- FAIL: TestTruncateWhileColumnBackfill (12.85s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@16d4f36a9d94eeec5a69fa715ee409af0575e5f9:

I200116 07:54:56.480964 325997 server/node.go:655  [n1] connecting to gossip network to verify cluster ID...
I200116 07:54:56.480984 325997 server/node.go:675  [n1] node connected via gossip and verified as part of cluster "505c10ee-a7fe-4d18-9cc0-c35f1a42cfb9"
I200116 07:54:56.481041 325997 server/node.go:522  [n1] node=1: started with [<no-attributes>=<in-mem>] engine(s) and attributes []
I200116 07:54:56.481094 325997 server/server.go:2102  [n1] Could not start goroutine dumper worker due to: directory to store dumps could not be determined
I200116 07:54:56.481153 325997 server/server.go:1562  [n1] starting https server at 127.0.0.1:45043 (use: 127.0.0.1:45043)
I200116 07:54:56.481177 325997 server/server.go:1567  [n1] starting postgres server at 127.0.0.1:45001 (use: 127.0.0.1:45001)
I200116 07:54:56.481192 325997 server/server.go:1569  [n1] starting grpc server at 127.0.0.1:35017
I200116 07:54:56.481206 325997 server/server.go:1570  [n1] advertising CockroachDB node at 127.0.0.1:35017
W200116 07:54:56.482395 326125 storage/store.go:1526  [n1,s1,r6/1:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r6: replica (n1,s1):1 not lease holder; lease holder unknown
I200116 07:54:56.594811 326253 sql/event_log.go:132  [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root}
I200116 07:54:56.603829 326153 sql/event_log.go:132  [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:19.2-9 User:root}
I200116 07:54:56.627249 326581 sql/event_log.go:132  [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:b127e587-98bf-4ffb-b8ab-f67b595e4f1d User:root}
I200116 07:54:56.636506 325894 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root}
I200116 07:54:56.642170 325898 sql/event_log.go:132  [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root}
I200116 07:54:56.680146 325997 server/server.go:1626  [n1] done ensuring all necessary migrations have run
I200116 07:54:56.680254 325997 server/server.go:1907  [n1] serving sql connections
I200116 07:54:56.684803 326653 sql/event_log.go:132  [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:35017 Attrs: Locality: ServerVersion:19.2-9 BuildTag:v20.1.0-alpha.20191118-1292-g16d4f36 StartedAt:1579161296477643131 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:45001} ClusterID:505c10ee-a7fe-4d18-9cc0-c35f1a42cfb9 StartedAt:1579161296477643131 LastUp:1579161296477643131}
I200116 07:54:56.693270 326651 server/server_update.go:53  [n1] no need to upgrade, cluster already at the newest version
I200116 07:54:56.704460 326853 sql/event_log.go:132  [n1,client=127.0.0.1:48542,user=root] Event: "create_database", target: 52, info: {DatabaseName:t Statement:CREATE DATABASE t User:root}
I200116 07:54:56.715378 326853 sql/event_log.go:132  [n1,client=127.0.0.1:48542,user=root] Event: "create_table", target: 53, info: {TableName:t.public.test Statement:CREATE TABLE t.public.test (k INT8 PRIMARY KEY, v INT8) User:root}
I200116 07:54:56.717668 326972 storage/replica_command.go:406  [n1,split,s1,r28/1:/{Table/32-Max}] initiating a split of this range at key /Table/53 [r29] (zone config)
I200116 07:54:56.939927 326853 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:48542,user=root] publish: descID=53 (test) version=2 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 07:54:56.945254 326853 sql/event_log.go:132  [n1,client=127.0.0.1:48542,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test ADD COLUMN x DECIMAL NOT NULL DEFAULT 1.4::DECIMAL, ADD CHECK (x >= 0) User:root MutationID:1 CascadeDroppedViews:[]}
I200116 07:54:56.947635 326812 sql/lease.go:921  new lease: 53("test") ver=2:1579161587.577699179,0, refcount=0
I200116 07:54:56.956758 326853 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:48542,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 07:54:56.968600 326853 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:48542,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 07:54:56.976357 326853 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:48542,user=root] publish: descID=53 (test) version=3 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 07:54:56.984975 326853 sql/event_log.go:132  [n1,client=127.0.0.1:48542,user=root] Event: "alter_table", target: 53, info: {TableName:t.public.test Statement:ALTER TABLE t.public.test DROP COLUMN v User:root MutationID:2 CascadeDroppedViews:[]}
I200116 07:54:56.989585 327214 sql/lease.go:921  new lease: 53("test") ver=3:1579161611.419860217,0, refcount=0
I200116 07:54:56.996917 326140 sql/sqlbase/structured.go:1475  [n1,scExec] publish: descID=53 (test) version=4 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 07:54:57.016512 327127 sql/lease.go:921  new lease: 53("test") ver=4:1579161611.420860217,0, refcount=0
I200116 07:54:57.021999 326140 sql/lease.go:326  [n1,scExec] waiting for 1 leases to expire: desc=[{test 53 3}]
I200116 07:54:57.048350 326140 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 07:54:57.064270 326140 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 07:54:57.084405 326853 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:48542,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 07:54:57.126468 326853 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:48542,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 07:54:57.184603 326853 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:48542,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 07:54:57.195160 326853 sql/event_log.go:132  [n1,client=127.0.0.1:48542,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200116 07:54:57.196211 327297 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200116 07:54:57.219972 325997 util/stop/stopper.go:539  quiescing
W200116 07:54:57.220081 327588 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Table/SystemConfigSpan/Start: node unavailable; try another peer
W200116 07:54:57.220944 326140 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200116 07:54:57.220998 326140 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200116 07:54:57.221023 326140 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
--- FAIL: TestTruncateWhileColumnBackfill (0.87s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
tbg commented 4 years ago

Ok this is flaky as hell, I'm going to skip it. @spaskob could you provide a fix?

cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@16d4f36a9d94eeec5a69fa715ee409af0575e5f9:

W200116 08:32:23.392900 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 08:32:23.598479 351015 storage/replica_consistency.go:256  [n1,consistencyChecker,s1,r4/1:/System{/tsd-tse}] triggering stats recomputation to resolve delta of {ContainsEstimates:2700 LastUpdateNanos:1579163534014985143 IntentAge:0 GCBytesAge:0 LiveBytes:-7430 LiveCount:-648 KeyBytes:-31352 KeyCount:-648 ValBytes:23922 ValCount:-648 IntentBytes:0 IntentCount:0 SysBytes:0 SysCount:0}
I200116 08:32:23.906041 346672 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 08:32:24.047122 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 08:32:24.467784 346672 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 08:32:24.593076 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 08:32:24.982522 346672 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 08:32:25.078590 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 08:32:25.562863 346672 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 08:32:25.657088 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 08:32:26.028768 346672 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 08:32:26.161299 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 08:32:26.560566 346672 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 08:32:26.641080 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 08:32:26.954827 346672 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 08:32:27.035247 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 08:32:27.315469 346672 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 08:32:27.383653 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 08:32:27.658067 346672 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 08:32:27.797506 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 08:32:28.144302 346672 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 08:32:28.253999 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 08:32:28.579867 346672 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 08:32:28.680128 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 08:32:28.956554 346672 sql/backfill.go:143  [n1,scExec] Running backfill for "test", v=4, m=1
W200116 08:32:29.053619 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: context deadline exceeded
I200116 08:32:29.205076 347261 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:45798,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 08:32:29.378787 347261 sql/sqlbase/structured.go:1475  [n1,client=127.0.0.1:45798,user=root] publish: descID=53 (test) version=5 mtime=1970-01-01 00:00:00 +0000 UTC
I200116 08:32:29.483175 347261 sql/event_log.go:132  [n1,client=127.0.0.1:45798,user=root] Event: "truncate_table", target: 53, info: {TableName:t.public.test Statement:TRUNCATE TABLE t.public.test User:root}
I200116 08:32:29.577972 353060 storage/replica_command.go:406  [n1,split,s1,r29/1:/{Table/53-Max}] initiating a split of this range at key /Table/54 [r30] (zone config)
I200116 08:32:29.689020 346461 util/stop/stopper.go:539  quiescing
W200116 08:32:29.690232 353002 storage/intentresolver/intent_resolver.go:820  failed to gc transaction record: could not GC completed transaction anchored at /Table/SystemConfigSpan/Start: node unavailable; try another peer
W200116 08:32:29.694105 346672 sql/schema_changer.go:955  [n1,scExec] waiting to update leases: node unavailable; try another peer
W200116 08:32:29.695180 346672 sql/schema_changer.go:982  [n1,scExec] while reasing schema change lease: node unavailable; try another peer
W200116 08:32:29.695638 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
I200116 08:32:29.711712 346695 kv/transport_race.go:108  transport race promotion: ran 124 iterations on up to 6127 requests
W200116 08:32:29.716971 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 08:32:29.736261 346616 storage/store.go:1715  [n1,s1,r29/1:/Table/5{3-4}] unable to gossip on capacity change: node unavailable; try another peer
I200116 08:32:29.736982 346616 storage/queue.go:524  [n1,s1,r29/1:/Table/5{3-4}] rate limited in MaybeAdd (split): node unavailable; try another peer
I200116 08:32:29.737193 346616 storage/queue.go:524  [n1,s1,r29/1:/Table/5{3-4}] rate limited in MaybeAdd (replicate): node unavailable; try another peer
W200116 08:32:29.737745 346672 sql/schema_changer.go:2033  [n1] Error executing schema change: node unavailable; try another peer
W200116 08:32:29.740341 353060 internal/client/txn.go:558  [n1,split,s1,r29/1:/Table/5{3-4}] failure aborting transaction: node unavailable; try another peer; abort caused by: result is ambiguous (server shutdown)
I200116 08:32:29.740911 346602 storage/queue.go:524  [n1,s1,r30/1:/{Table/54-Max}] rate limited in MaybeAdd (merge): node unavailable; try another peer
E200116 08:32:29.741122 353060 storage/queue.go:1035  [n1,split,s1,r29/1:/Table/5{3-4}] unable to split [n1,s1,r29/1:/Table/5{3-4}] at key "/Table/54": split at key /Table/54 failed: result is ambiguous (server shutdown)
--- FAIL: TestTruncateWhileColumnBackfill (26.72s)
    schema_changer_test.go:3662: job 0: expected status succeeded, got running

Failed to find issue assignee: 
old friend vivekmenezes is friendless```

<details><summary>Repro</summary><p>
Parameters:

- GOFLAGS=-json

make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1



<sub>powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)</sub></p></details>
otan commented 4 years ago

change is being reverted

cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@d89d18bd2db88034463f2d1379190e6915309939:

Fatal error:

panic: runtime error: index out of range [0] with length 0

Stack:

goroutine 585339 [running]:
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.(*ImmutableTableDescriptor).MakeFirstMutationPublic(0xc0045de500, 0x0, 0x0, 0xc0042c11e0, 0x162cc14ae0939842)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/structured.go:3513 +0x424
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).validateConstraints.func2(0x59f4000, 0xc001cce640, 0xc004938880, 0x59f40c0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/backfill.go:569 +0x51
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1(0xc002e4ff68, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x3a
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc0093b3b90, 0xc0093b3bf0)
    /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
    /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
Log preceding fatal error

``` === RUN TestTruncateWhileColumnBackfill ```

More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestTruncateWhileColumnBackfill.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@d89d18bd2db88034463f2d1379190e6915309939:

=== RUN   TestTruncateWhileColumnBackfill
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTruncateWhileColumnBackfill830180493
--- FAIL: TestTruncateWhileColumnBackfill (6.60s)
    test_log_scope.go:154: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTruncateWhileColumnBackfill830180493
    test_log_scope.go:63: use -show-logs to present logs inline
    schema_changer_test.go:4278: expected 0 key value pairs, but got 301

Failed to find issue assignee: 
old friend vivekmenezes is friendless
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestTruncateWhileColumnBackfill.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@2022bd0431f62b4b73c5377cac638bfa61bfb01d:

Fatal error:

panic: runtime error: index out of range [0] with length 0

Stack:

goroutine 878035 [running]:
github.com/cockroachdb/cockroach/pkg/sql/sqlbase.(*ImmutableTableDescriptor).MakeFirstMutationPublic(0xc0023d0000, 0x0, 0x0, 0xc0006fef30, 0x162cc2e115ff585c)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/structured.go:3513 +0x424
github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChanger).validateConstraints.func2(0x59f4d00, 0xc0072d1880, 0xc009acc380, 0x59f4dc0)
    /go/src/github.com/cockroachdb/cockroach/pkg/sql/backfill.go:569 +0x51
github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx.func1(0xc000994f68, 0x0)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:166 +0x3a
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc00634e2d0, 0xc00634e330)
    /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
    /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66
Log preceding fatal error

``` === RUN TestTruncateWhileColumnBackfill ```

More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestTruncateWhileColumnBackfill.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@f89dbd928242862863858483f6970854a7cc912a:

=== RUN   TestTruncateWhileColumnBackfill
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTruncateWhileColumnBackfill042715419
--- FAIL: TestTruncateWhileColumnBackfill (2.12s)
    test_log_scope.go:154: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTruncateWhileColumnBackfill042715419
    test_log_scope.go:63: use -show-logs to present logs inline
    schema_changer_test.go:4278: expected 0 key value pairs, but got 4001

Failed to find issue assignee: 
old friend vivekmenezes is friendless
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestTruncateWhileColumnBackfill.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@f89dbd928242862863858483f6970854a7cc912a:

=== RUN   TestTruncateWhileColumnBackfill
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTruncateWhileColumnBackfill081824596
--- FAIL: TestTruncateWhileColumnBackfill (6.54s)
    test_log_scope.go:154: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTruncateWhileColumnBackfill081824596
    test_log_scope.go:63: use -show-logs to present logs inline
    schema_changer_test.go:4278: expected 0 key value pairs, but got 301

Failed to find issue assignee: 
old friend vivekmenezes is friendless
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestTruncateWhileColumnBackfill.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 4 years ago

(sql).TestTruncateWhileColumnBackfill failed on master@2022bd0431f62b4b73c5377cac638bfa61bfb01d:

=== RUN   TestTruncateWhileColumnBackfill
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTruncateWhileColumnBackfill471520250
--- FAIL: TestTruncateWhileColumnBackfill (6.64s)
    test_log_scope.go:154: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestTruncateWhileColumnBackfill471520250
    test_log_scope.go:63: use -show-logs to present logs inline
    schema_changer_test.go:4278: expected 0 key value pairs, but got 301

Failed to find issue assignee: 
old friend vivekmenezes is friendless
More

Parameters: - GOFLAGS=-json ``` make stressrace TESTS=TestTruncateWhileColumnBackfill PKG=./pkg/sql TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2ATestTruncateWhileColumnBackfill.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

ajwerner commented 2 years ago

@ajwerner to delete this test.