search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.09k stars 3.8k forks source link

roachtest: tpccbench/nodes=9/cpu=4/multi-region failed [replica inconsistency] #69414

Closed cockroach-teamcity closed 3 years ago

cockroach-teamcity commented 3 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ ab1fc343c9a1140191f96353995258e609a84d02:

          |   |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:412
          |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func1
          |   |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:166
          |   | github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).ParallelE.func1.1
          |   |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1709
          |   | runtime.goexit
          |   |     /usr/local/go/src/runtime/asm_amd64.s:1371
          | Wraps: (2) ~ ./cockroach.sh
          |   | Job for cockroach.service failed because the control process exited with error code.
          |   | See "systemctl status cockroach.service" and "journalctl -xe" for details.
          | Wraps: (3) exit status 1
          | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *exec.ExitError: 
          | I210826 12:54:50.179613 1 (gostd) cluster_synced.go:1677  [-] 1  command failed
        Wraps: (2) exit status 1
        Error types: (1) *cluster.WithCommandDetails (2) *exec.ExitError

    cluster.go:1249,context.go:89,cluster.go:1237,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3359107-1629959017-35-n12cpu4-geo --oneshot --ignore-empty-nodes: exit status 1 1: 1287
        3: 1312
        2: 1418
        4: skipped
        7: dead (exit status 1)
        6: 1334
        5: 1311
        10: 1470
        11: 1440
        8: skipped
        9: 1435
        12: skipped
        Error: UNCLASSIFIED_PROBLEM: 7: dead (exit status 1)
        (1) UNCLASSIFIED_PROBLEM
        Wraps: (2) attached stack trace
          -- stack trace:
          | main.glob..func14
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
          | main.wrap.func1
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
          | github.com/spf13/cobra.(*Command).execute
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
          | github.com/spf13/cobra.(*Command).ExecuteC
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
          | github.com/spf13/cobra.(*Command).Execute
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
          | main.main
          |     /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
          | runtime.main
          |     /usr/local/go/src/runtime/proc.go:225
          | runtime.goexit
          |     /usr/local/go/src/runtime/asm_amd64.s:1371
        Wraps: (3) 7: dead (exit status 1)
        Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
Reproduce

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

tbg commented 3 years ago

on n7 (not n5 as indicated by the message - it's getting confused because of the gaps in the node list)

* ERROR: ERROR: startup forbidden by prior critical alert
* DETAIL: From /mnt/data1/cockroach/auxiliary/_CRITICAL_ALERT.txt:
*
ERROR: startup forbidden by prior critical alert
DETAIL: From /mnt/data1/cockroach/auxiliary/_CRITICAL_ALERT.txt:
Failed running "start"
cockroach exited with code 1: Thu Aug 26 12:54:31 UTC 2021

We've seen this before, in https://github.com/cockroachdb/cockroach/issues/67471.

tbg commented 3 years ago

Uh-oh:

(n6,s6):3: checksum 7c32651636a3930317d12e4d006802fefd1cfac60fe9cfe32dac898b5ac75d76c899fb4b908b13395f92cb2f1b2897a7dbbdec995f455c2c8a32d1102b99bf1e [minority]
- stats: contains_estimates:0 last_update_nanos:1629982307391576058 intent_age:0 gc_bytes_age:69662266620 live_bytes:133029913 live_count:396189 key_bytes:9276126 key_count:396189 val_bytes:135241850 val_count:431486 intent_bytes:4923 intent_count:15 separated_intent_count:15 sys_bytes:13323 sys_count:138 abort_span_bytes:9701 
- stats.Sub(recomputation): last_update_nanos:1629982307391576058 intent_age:-2994 live_bytes:-84 val_bytes:-84 intent_bytes:-318 intent_count:-1 separated_intent_count:-1 
(n4,s4):8: checksum f44a7c890d7c01c844b4eae75bf9690d1c6072f2b6f9f3ad09a450b738315239afe21df3bb29f170778cf236d6db4b8d544dfdfb7835940869be19af034deb87
- stats: contains_estimates:0 last_update_nanos:1629982307391576058 intent_age:0 gc_bytes_age:69662266620 live_bytes:133029913 live_count:396189 key_bytes:9276126 key_count:396189 val_bytes:135241850 val_count:431486 intent_bytes:4923 intent_count:15 separated_intent_count:15 sys_bytes:13323 sys_count:138 abort_span_bytes:9701 
- stats.Sub(recomputation): last_update_nanos:1629982307391576058 
(n5,s5):7: checksum f44a7c890d7c01c844b4eae75bf9690d1c6072f2b6f9f3ad09a450b738315239afe21df3bb29f170778cf236d6db4b8d544dfdfb7835940869be19af034deb87
- stats: contains_estimates:0 last_update_nanos:1629982307391576058 intent_age:0 gc_bytes_age:69662266620 live_bytes:133029913 live_count:396189 key_bytes:9276126 key_count:396189 val_bytes:135241850 val_count:431486 intent_bytes:4923 intent_count:15 separated_intent_count:15 sys_bytes:13323 sys_count:138 abort_span_bytes:9701 
- stats.Sub(recomputation): last_update_nanos:1629982307391576058 
consistency check failed; fetching details and shutting down minority (n6,s6):3
t/data1/cockroach/auxiliary/checkpoints/r734_at_14189›

(n6,s6):3: checksum ad260ec0efec305ff8a3a2601ba19ea7276820ef270e602151cf9dcbb22cfe38d1e0407759c29938475d157b3e27776d7d0d0d3a7c7504da112fa886b9fe3497 [minority]
- stats: contains_estimates:0 last_update_nanos:1629982316710611497 intent_age:0 gc_bytes_age:69765807974 live_bytes:133028679 live_count:396189 key_bytes:9277338 key_count:396189 val_bytes:135272256 val_count:431587 intent_bytes:0 intent_count:0 separated_intent_count:0 sys_bytes:13323 sys_count:138 abort_span_bytes:9701 
- stats.Sub(recomputation): last_update_nanos:1629982316710611497 intent_age:-3003 live_bytes:-84 val_bytes:-84 intent_bytes:-318 intent_count:-1 separated_intent_count:-1 
(n4,s4):8: checksum f8449b7052dcb7045aefe6309381393c6715d3408fb571e9dfd30d88ec7a7f07819b997ada537108433b50daaec786f8fb7dc22109e1ffc1267b189faeb85c54
- stats: contains_estimates:0 last_update_nanos:1629982316710611497 intent_age:0 gc_bytes_age:69765807974 live_bytes:133028679 live_count:396189 key_bytes:9277338 key_count:396189 val_bytes:135272256 val_count:431587 intent_bytes:0 intent_count:0 separated_intent_count:0 sys_bytes:13323 sys_count:138 abort_span_bytes:9701 
- stats.Sub(recomputation): last_update_nanos:1629982316710611497 
(n5,s5):7: checksum f8449b7052dcb7045aefe6309381393c6715d3408fb571e9dfd30d88ec7a7f07819b997ada537108433b50daaec786f8fb7dc22109e1ffc1267b189faeb85c54
- stats: contains_estimates:0 last_update_nanos:1629982316710611497 intent_age:0 gc_bytes_age:69765807974 live_bytes:133028679 live_count:396189 key_bytes:9277338 key_count:396189 val_bytes:135272256 val_count:431587 intent_bytes:0 intent_count:0 separated_intent_count:0 sys_bytes:13323 sys_count:138 abort_span_bytes:9701 
- stats.Sub(recomputation): last_update_nanos:1629982316710611497 
====== diff(f8449b7052dcb7045aefe6309381393c6715d3408fb571e9dfd30d88ec7a7f07819b997ada537108433b50daaec786f8fb7dc22109e1ffc1267b189faeb85c54, [minority]) ======
--- leaseholder
+++ follower
+0,0 ‹/Table/61/1/1686/4881/0›
+    ts:1970-01-01 00:00:00 +0000 UTC
+    value:‹1629979313.521491293,2 {Txn:id=a181b3d4 key=/Table/55/1/1686/2/0 pri=0.02746475 epo=4 ts=1629979313.521491293,2 min=1629979155.112560857,0 seq=5 Timestamp:1629979313.521491293,2 Deleted:false KeyBytes:12 ValBytes:306 RawBytes:[] IntentHistory:[] MergeTimestamp:<nil> TxnDidNotUpdateMeta:<nil>}›
+    raw mvcc_key/value: ‹c589f70696f713118800› ‹0a3d0a10a181b3d4ff05487885ec309606895b5e1a07bf89f706968a8820042a0c08ddfabbadcbb1b6cf16100230e9ff2338054a0a08d9919a9efdacb6cf16120c08ddfabbadcbb1b6cf1610021800200c28b202›
consistency check failed

I'm uploading the full artifacts to https://drive.google.com/drive/folders/1z_gpHX39QwKMC4x7aYWMriRxV6mV5cTC?usp=sharing (CRL only).

tbg commented 3 years ago

$ cockroach debug merge-logs --format crdb-v1 --program-filter '^cockroach$' --filter 'r734[^0-9]' logs/ | gh gist create

https://gist.github.com/4d458af340e47fa9c1f549297eb747f9

tbg commented 3 years ago

The timestamp of the separated intent missing on the leaseholder n6 is

Thursday, August 26, 2021 12:01:53 PM

The log timestamps at the time of the consistency check are close to an hour later, around 12:52. So whatever went wrong likely went wrong around 12:0[01] on r723.

Unfortunately, there's radio silence around this time:

teamcity-3359107-1629959017-35-n12cpu4-geo-0007> I210826 09:42:11.211904 170 kv/kvserver/store_remove_replica.go:133 ⋮ [n6,s6,r734/3:‹/Table/61/1/168{4/7433-5/634…}›] 117349  removing replica r732/3
teamcity-3359107-1629959017-35-n12cpu4-geo-0006> W210826 11:53:31.483979 263 kv/kvserver/store_raft.go:525 ⋮ [n5,s5,r734/7:‹/Table/61/1/168{4/7433-8/3622}›] 2129  handle raft ready: 0.5s [applied=0, batches=0, state_assertions=0]
teamcity-3359107-1629959017-35-n12cpu4-geo-0006> W210826 11:54:53.437996 264 kv/kvserver/store_raft.go:525 ⋮ [n5,s5,r734/7:‹/Table/61/1/168{4/7433-8/3622}›] 4151  handle raft ready: 1.7s [applied=1, batches=1, state_assertions=0]
teamcity-3359107-1629959017-35-n12cpu4-geo-0006> W210826 11:55:56.079618 1368314 kv/kvserver/spanlatch/manager.go:526 ⋮ [n5,s5,r734/7:‹/Table/61/1/168{4/7433-8/3622}›] 5926  have been waiting 15s to acquire ‹read› latch ‹/Table/61/1/1686/15985/0@1629978933.679786651,0›, held by ‹write› la
tch ‹/Table/61/1/1686/15985/0@1629978810.646836154,0›
teamcity-3359107-1629959017-35-n12cpu4-geo-0006> W210826 12:03:30.615608 254 kv/kvserver/store_raft.go:525 ⋮ [n5,s5,r734/7:‹/Table/61/1/168{4/7433-8/3622}›] 32283  handle raft ready: 0.7s [applied=0, batches=0, state_assertions=0]
teamcity-3359107-1629959017-35-n12cpu4-geo-0006> W210826 12:04:09.733580 2575922 kv/kvserver/spanlatch/manager.go:526 ⋮ [n5,s5,r734/7:‹/Table/61/1/168{4/7433-8/3622}›] 33541  have been waiting 15s to acquire ‹write› latch ‹/Table/61/1/1685/25393/0@1629979312.161687797,0›, held by ‹read› latch ‹/Table/61/1/1685/25393/0@1629979379.140708674,0›
teamcity-3359107-1629959017-35-n12cpu4-geo-0005> W210826 12:04:26.445723 1749274 kv/kvclient/kvcoord/dist_sender.go:1537 ⋮ [n4] 6706  slow range RPC: have been waiting 87.18s (1 attempts) for RPC Get

I'm also not seeing anything obviously wrong around the 12:00 mark:

cockroach debug merge-logs --format crdb-v1 --program-filter '^cockroach$' --from '210826 11:58:00' --to '210826 12:03:00' logs/ > 1200.log
grep -vE 'have been waiting|finished waiting|handle raft|slow RPC|runtime_stats|health alerts|gossip.go' 1200.log  | less -S
[...]
tbg commented 3 years ago

The txnid also makes no appearance in the logs. Neither does the full pretty-printed key.

sumeerbhola commented 3 years ago

If this is reproducible, then one thing to try would be to change the code to disable the SINGLEDEL optimization.

btw, there is a case where SINGLEDEL can have 2 SETs under it.

tbg commented 3 years ago

@sumeerbhola how ironclad is the testing of single deletions on the pebble side? I expect this to be very very hard to reproduce, so if we can do any work on the pebble side to make sure that single deletes are very well tested in the presence of concurrent compactions, ingestions, etc that would be helpful.

tbg commented 3 years ago

@AlexTalks will start trying to reproduce this tomorrow, with help from yours truly. I'll need to take another look at the test failure to get an idea of whether we can skip parts of the test or specialize it somehow, but we probably want to run with something like https://github.com/cockroachdb/cockroach/pull/62791/commits/fd9aa5a5a0ae15f55623d45acfd6ffe7b4baa48f and also more stringent consistency checks, perhaps even specialized checks on txn records we know have been single-del'ed.

sumeerbhola commented 3 years ago

@sumeerbhola how ironclad is the testing of single deletions on the pebble side? I expect this to be very very hard to reproduce, so if we can do any work on the pebble side to make sure that single deletes are very well tested in the presence of concurrent compactions, ingestions, etc that would be helpful.

It is quite thoroughly tested using the Pebble metamorphic test which generates single delete just like other operations to be used in batches and file ingestion. It does not test the particular case that I mentioned earlier, for which I was claiming correctness based on fundamental Pebble invariants. I'll look into adding testing for it.

sumeerbhola commented 3 years ago

It does not test the particular case that I mentioned earlier, for which I was claiming correctness based on fundamental Pebble invariants.

@nicktrav will start working on enhancing the Pebble metamorphic test for this case + make it behave randomly in terms of sometimes using Del instead of SingleDel for keys that are eligible for SingleDel.

tbg commented 3 years ago

The timeline in the test is:

- 8:13am test starts
- 8:37 starting to wait for rebalancing
- 9:42: here we see the last range event for r734. It does not split, merge, or really log anything interesting past this point
- 10:03: line search at 2000 warehouses...
- 10:24: pass,  trying 2015 warehouses
- 10:44: pass, trying 2045
- 11:04: pass, trying 2105
- 11:25: pass, trying 2225
- 11:45: pass, trying 2465
- 11:53: we see log activity on r734, it's not terribly interesting (slow raft ready etc), so this is just an artifact of the CRDB cluster being overloaded.
- 12:01: around this time, the intent should be written
- 12:26: failed due to efficiency, trying 2285
- 12:46 failed due to efficiency, trying 2255
- 12:52: failed due to n7 dead (this is going to be the consistency check failure, since the exit code is 7=FatalError)

Given all of this, I think we can make some assumptions:

In the past, such inconsistencies would usually come down to a problem at the storage layer (i.e. the "singledelete didn't do its job" theory the storage team is looking into by adding more testing) and the kv replication layer (for example, we once had a bug where cached raft entries at the leader were not reliably evicted when an uncommitted part of the raft log was discarded, so we'd sometimes send the wrong entry to a follower and it would thus diverge. https://github.com/cockroachdb/cockroach/issues/61990#issuecomment-814827717 is the most recent example of this).

The import phase of the test takes around 30 minutes only, and we're interested in overload. I would thus say we try a repro cycle where we stay multi-region (in case that exacerbated anything), and we run the import but skip the rebalancing phase. When we then hit the cluster with 2000 warehouse TPCC, it will most likely overload the cluster (if that proves to be false, we start at a higher warehouse count like 2200). We run with an aggressive consistency check interval, too, as it is very valuable to stop the test close to where the issue happens. One interesting thing to note is about the line search and how it reacts to dead nodes: it doesn't fail the test! This is because by design this test verges into overload territory and nodes may crash (at least today). The reason the above test failed is because when an inconsistency is detected, we error out during the next attempt at starting the node, and that will terminate the line search. So the line search behavior is already exactly what we want for the repro cycle we're after; we will get failures where the node refused to start (likely due to the inconsistency death rattle), but we don't have to worry about nodes ooming and such, as this will be suppressed by the line search.

Since I expect that it will take a while to get a repro, we'll also put in the pebble ArchiveCleaner commit referenced above. We need to check that we're not running out of disk space; if that becomes an issue maybe we can hack around it by provisioning larger disks, or by catching disk-full conditions and ignoring the outcome.

tbg commented 3 years ago

Going to see how many of the clusters for this test we can put into andrei-jepsen

for i in $(seq 1 10); do GCE_PROJECT=andrei-jepsen roachprod create $USER-geo$i -n 12 --clouds=gce --local-ssd=true --gce-machine-type=n1-standard-4 --gce-zones=us-east1-b,us-west1-b,europe-west2-b --geo --lifetime=12h0m0s --os-volume-size=32 --local-ssd-no-ext4-barrier; done
tbg commented 3 years ago

^-- 10 clusters worked fine, so going to try making a few more. 💸

tbg commented 3 years ago

@AlexTalks has what I think is the "final first version" of our repro machinery running, at commit "roachtest: stop tpcc iteration on single passing test" of https://github.com/cockroachdb/cockroach/pull/69646/commits. The minutiae are in (internal link) slack, lurkers welcome. We'll provide condensed updates here.

cockroach-teamcity commented 3 years ago

roachtest.tpccbench/nodes=9/cpu=4/multi-region failed with artifacts on master @ 63679e7bf4a0244c6e46381751543cfe3f32388c:


The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1
    monitor.go:128,tpcc.go:1121,tpcc.go:955,test_runner.go:777: monitor failure: monitor task failed: failed with output "I210902 08:01:47.894392 1 workload/cli/run.go:408  [-] 1  creating load generator...\nInitializing 3000 connections...\nInitializing 0 idle connections...\nInitializing 3000 workers and preparing statements...\nfatal error: concurrent map writes\nfatal error: concurrent map writes\n\ngoroutine 9174 [running]:\nruntime.throw(0x5249fe4, 0x15)\n\t/usr/local/go/src/runtime/panic.go:1117 +0x72 fp=0xc00009ec60 sp=0xc00009ec30 pc=0x48b672\nruntime.mapassign_faststr(0x4a822a0, 0xc0019107e0, 0xc00680e500, 0xb, 0x2)\n\t/usr/local/go/src/runtime/map_faststr.go:291 +0x3d8 fp=0xc00009ecc8 sp=0xc00009ec60 pc=0x467bb8\ngithub.com/cockroachdb/cockroach/pkg/workload.(*SQLRunner).Init(0xc000ae0b70, 0x8a99a38, 0xc0001106c0, 0x5218701, 0x9, 0xc0019107b0, 0xc000c025a0, 0x8, 0x0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/sql_runner.go:122 +0x20b fp=0xc00009ed80 sp=0xc00009ecc8 pc=0x3aa030b\ngithub.com/cockroachdb/cockroach/pkg/workload/tpcc.createNewOrder(0x8a99a38, 0xc0001106c0, 0xc00000d2c0, 0xc0019107b0, 0x17, 0xc0017e20c0, 0x0, 0x18)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/tpcc/new_order.go:126 +0x38a fp=0xc00009ee18 sp=0xc00009ed80 pc=0x4204a2a\ngithub.com/cockroachdb/cockroach/pkg/workload/tpcc.newWorker(0x8a99a38, 0xc0001106c0, 0xc00000d2c0, 0xc0019107b0, 0xc005e852c0, 0xc000ad4030, 0x65, 0xc006718000, 0xc006150508, 0xc00120af94)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/tpcc/worker.go:193 +0x21b fp=0xc00009eea8 sp=0xc00009ee18 pc=0x421159b\ngithub.com/cockroachdb/cockroach/pkg/workload/tpcc.(*tpcc).Ops.func3(0x495801, 0x0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/tpcc/tpcc.go:862 +0x145 fp=0xc00009ef78 sp=0xc00009eea8 pc=0x421cfc5\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc00028eea0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x59 fp=0xc00009efd0 sp=0xc00009ef78 pc=0x12dd439\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00009efd8 sp=0xc00009efd0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 1 [runnable]:\nruntime.gopark(0x7aa4308, 0xc0063067d8, 0x160f, 0x2)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000cc4d18 sp=0xc000cc4cf8 pc=0x48e2e5\nruntime.chansend(0xc006306780, 0xc000cc4e30, 0xc000cc4d01, 0x420f89e, 0xf3af843e77348937)\n\t/usr/local/go/src/runtime/chan.go:257 +0x29a fp=0xc000cc4da0 sp=0xc000cc4d18 pc=0x458cba\nruntime.chansend1(0xc006306780, 0xc000cc4e30)\n\t/usr/local/go/src/runtime/chan.go:143 +0x35 fp=0xc000cc4dd8 sp=0xc000cc4da0 pc=0x458a15\ngithub.com/cockroachdb/cockroach/pkg/workload/tpcc.(*tpcc).Ops(0xc00000d2c0, 0x8a99a38, 0xc0001106c0, 0xc000692900, 0x9, 0x11, 0xc000110660, 0x0, 0x0, 0x0, ...)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/tpcc/tpcc.go:860 +0xc7e fp=0xc000cc50a0 sp=0xc000cc4dd8 pc=0x420f89e\ngithub.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func2(0x7f859c862ff8, 0xc00000d2c0, 0xc000692900, 0x9, 0x11, 0xc000110660, 0xc0010c8040, 0x8a99a38, 0xc0001106c0, 0x1, ...)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:419 +0x207 fp=0xc000cc5218 sp=0xc000cc50a0 pc=0x3b19967\ngithub.com/cockroachdb/cockroach/pkg/workload/cli.runRun(0x8a501d0, 0xc00000d2c0, 0xc000692900, 0x9, 0x11, 0x5208372, 0x4, 0x0, 0x0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:436 +0x685 fp=0xc000cc5710 sp=0xc000cc5218 pc=0x3b16425\ngithub.com/cockroachdb/cockroach/pkg/workload/cli.CmdHelper.func1(0xc000c0a500, 0xc000692900, 0x9, 0x11, 0x4316377, 0x8a99a00)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:223 +0x1e4 fp=0xc000cc5c58 sp=0xc000cc5710 pc=0x3b18d24\ngithub.com/cockroachdb/cockroach/pkg/workload/cli.HandleErrs.func1(0xc000c0a500, 0xc000692900, 0x9, 0x11)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/cli.go:87 +0x5d fp=0xc000cc5cd8 sp=0xc000cc5c58 pc=0x3b17f1d\ngithub.com/spf13/cobra.(*Command).execute(0xc000c0a500, 0xc0006927e0, 0x11, 0x12, 0xc000c0a500, 0xc0006927e0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856 +0x2c2 fp=0xc000cc5d98 sp=0xc000cc5cd8 pc=0x3accf82\ngithub.com/spf13/cobra.(*Command).ExecuteC(0xb0b0ac0, 0xc000074100, 0xc0003a0eaa, 0x11)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960 +0x375 fp=0xc000cc5e78 sp=0xc000cc5d98 pc=0x3acdcb5\ngithub.com/spf13/cobra.(*Command).Execute(...)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897\ngithub.com/cockroachdb/cockroach/pkg/cli.Run(...)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:278\ngithub.com/cockroachdb/cockroach/pkg/cli.doMain(0xc000c0a500, 0xc0003a0eaa, 0x11, 0x0, 0x0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:125 +0x11a fp=0xc000cc5ed8 sp=0xc000cc5e78 pc=0x42b9bda\ngithub.com/cockroachdb/cockroach/pkg/cli.Main()\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cli/cli.go:59 +0x13a fp=0xc000cc5f78 sp=0xc000cc5ed8 pc=0x42b98ba\nmain.main()\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/cockroach/main.go:26 +0x25 fp=0xc000cc5f88 sp=0xc000cc5f78 pc=0x4462ba5\nruntime.main()\n\t/usr/local/go/src/runtime/proc.go:225 +0x256 fp=0xc000cc5fe0 sp=0xc000cc5f88 pc=0x48deb6\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000cc5fe8 sp=0xc000cc5fe0 pc=0x4c5cc1\n\ngoroutine 2 [force gc (idle), 3 minutes]:\nruntime.gopark(0x7aa4510, 0xb44e2b0, 0x1411, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000090fb0 sp=0xc000090f90 pc=0x48e2e5\nruntime.goparkunlock(...)\n\t/usr/local/go/src/runtime/proc.go:342\nruntime.forcegchelper()\n\t/usr/local/go/src/runtime/proc.go:276 +0xc5 fp=0xc000090fe0 sp=0xc000090fb0 pc=0x48e145\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000090fe8 sp=0xc000090fe0 pc=0x4c5cc1\ncreated by runtime.init.6\n\t/usr/local/go/src/runtime/proc.go:264 +0x35\n\ngoroutine 3 [GC sweep wait]:\nruntime.gopark(0x7aa4510, 0xb456ce0, 0x140c, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0000917a8 sp=0xc000091788 pc=0x48e2e5\nruntime.goparkunlock(...)\n\t/usr/local/go/src/runtime/proc.go:342\nruntime.bgsweep(0xc00007c070)\n\t/usr/local/go/src/runtime/mgcsweep.go:182 +0x13b fp=0xc0000917d8 sp=0xc0000917a8 pc=0x4781fb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000917e0 sp=0xc0000917d8 pc=0x4c5cc1\ncreated by runtime.gcenable\n\t/usr/local/go/src/runtime/mgc.go:217 +0x5c\n\ngoroutine 4 [GC scavenge wait]:\nruntime.gopark(0x7aa4510, 0xb457de0, 0x140d, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000091f78 sp=0xc000091f58 pc=0x48e2e5\nruntime.goparkunlock(...)\n\t/usr/local/go/src/runtime/proc.go:342\nruntime.bgscavenge(0xc00007c070)\n\t/usr/local/go/src/runtime/mgcscavenge.go:314 +0x29c fp=0xc000091fd8 sp=0xc000091f78 pc=0x47637c\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000091fe0 sp=0xc000091fd8 pc=0x4c5cc1\ncreated by runtime.gcenable\n\t/usr/local/go/src/runtime/mgc.go:218 +0x7e\n\ngoroutine 5 [finalizer wait, 27 minutes]:\nruntime.gopark(0x7aa4510, 0xb4b19b8, 0xc000121410, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000090758 sp=0xc000090738 pc=0x48e2e5\nruntime.goparkunlock(...)\n\t/usr/local/go/src/runtime/proc.go:342\nruntime.runfinq()\n\t/usr/local/go/src/runtime/mfinal.go:175 +0xa9 fp=0xc0000907e0 sp=0xc000090758 pc=0x46d309\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000907e8 sp=0xc0000907e0 pc=0x4c5cc1\ncreated by runtime.createfing\n\t/usr/local/go/src/runtime/mfinal.go:156 +0x65\n\ngoroutine 19 [syscall, 27 minutes]:\nruntime.notetsleepg(0xb4b2cc0, 0xffffffffffffffff, 0x0)\n\t/usr/local/go/src/runtime/lock_futex.go:235 +0x34 fp=0xc0002a8798 sp=0xc0002a8768 pc=0x45fa54\nos/signal.signal_recv(0x0)\n\t/usr/local/go/src/runtime/sigqueue.go:168 +0xa5 fp=0xc0002a87c0 sp=0xc0002a8798 pc=0x4c21e5\nos/signal.loop()\n\t/usr/local/go/src/os/signal/signal_unix.go:23 +0x25 fp=0xc0002a87e0 sp=0xc0002a87c0 pc=0xf217c5\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0002a87e8 sp=0xc0002a87e0 pc=0x4c5cc1\ncreated by os/signal.Notify.func1.1\n\t/usr/local/go/src/os/signal/signal.go:151 +0x45\n\ngoroutine 18 [select, 27 minutes, locked to thread]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000092e10 sp=0xc000092df0 pc=0x48e2e5\nruntime.selectgo(0xc000092fa8, 0xc000092fa0, 0x0, 0x0, 0x2, 0x1, 0x1, 0x1)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc000092f48 sp=0xc000092e10 pc=0x49f557\nruntime.ensureSigM.func1()\n\t/usr/local/go/src/runtime/signal_unix.go:889 +0x1cd fp=0xc000092fe0 sp=0xc000092f48 pc=0x4be08d\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000092fe8 sp=0xc000092fe0 pc=0x4c5cc1\ncreated by runtime.ensureSigM\n\t/usr/local/go/src/runtime/signal_unix.go:872 +0xd5\n\ngoroutine 8 [chan receive, 27 minutes]:\nruntime.gopark(0x7aa4308, 0xc0000c65f8, 0x170e, 0x2)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0000935f8 sp=0xc0000935d8 pc=0x48e2e5\nruntime.chanrecv(0xc0000c65a0, 0xc0000936df, 0x1, 0x0)\n\t/usr/local/go/src/runtime/chan.go:576 +0x36f fp=0xc000093688 sp=0xc0000935f8 pc=0x459b8f\nruntime.chanrecv2(0xc0000c65a0, 0xc0000936df, 0x0)\n\t/usr/local/go/src/runtime/chan.go:444 +0x2b fp=0xc0000936b8 sp=0xc000093688 pc=0x45980b\ngithub.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00025b930)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149 fp=0xc0000937d8 sp=0xc0000936b8 pc=0xcf17e9\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000937e0 sp=0xc0000937d8 pc=0x4c5cc1\ncreated by github.com/klauspost/compress/zstd.newBlockDec\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173\n\ngoroutine 9 [chan receive, 27 minutes]:\nruntime.gopark(0x7aa4308, 0xc0000c6658, 0x170e, 0x2)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000290df8 sp=0xc000290dd8 pc=0x48e2e5\nruntime.chanrecv(0xc0000c6600, 0xc000290edf, 0x1, 0x0)\n\t/usr/local/go/src/runtime/chan.go:576 +0x36f fp=0xc000290e88 sp=0xc000290df8 pc=0x459b8f\nruntime.chanrecv2(0xc0000c6600, 0xc000093edf, 0x0)\n\t/usr/local/go/src/runtime/chan.go:444 +0x2b fp=0xc000290eb8 sp=0xc000290e88 pc=0x45980b\ngithub.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00025ba00)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149 fp=0xc000290fd8 sp=0xc000290eb8 pc=0xcf17e9\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000290fe0 sp=0xc000290fd8 pc=0x4c5cc1\ncreated by github.com/klauspost/compress/zstd.newBlockDec\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173\n\ngoroutine 10 [chan receive, 27 minutes]:\nruntime.gopark(0x7aa4308, 0xc0000c66b8, 0x170e, 0x2)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc00008c5f8 sp=0xc00008c5d8 pc=0x48e2e5\nruntime.chanrecv(0xc0000c6660, 0xc00008c6df, 0x1, 0x0)\n\t/usr/local/go/src/runtime/chan.go:576 +0x36f fp=0xc00008c688 sp=0xc00008c5f8 pc=0x459b8f\nruntime.chanrecv2(0xc0000c6660, 0xc00008c6df, 0x0)\n\t/usr/local/go/src/runtime/chan.go:444 +0x2b fp=0xc00008c6b8 sp=0xc00008c688 pc=0x45980b\ngithub.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00025bad0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149 fp=0xc00008c7d8 sp=0xc00008c6b8 pc=0xcf17e9\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00008c7e0 sp=0xc00008c7d8 pc=0x4c5cc1\ncreated by github.com/klauspost/compress/zstd.newBlockDec\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173\n\ngoroutine 11 [chan receive, 27 minutes]:\nruntime.gopark(0x7aa4308, 0xc0000c6718, 0x170e, 0x2)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc00008cdf8 sp=0xc00008cdd8 pc=0x48e2e5\nruntime.chanrecv(0xc0000c66c0, 0xc00008cedf, 0x1, 0x0)\n\t/usr/local/go/src/runtime/chan.go:576 +0x36f fp=0xc00008ce88 sp=0xc00008cdf8 pc=0x459b8f\nruntime.chanrecv2(0xc0000c66c0, 0xc00008cedf, 0x0)\n\t/usr/local/go/src/runtime/chan.go:444 +0x2b fp=0xc00008ceb8 sp=0xc00008ce88 pc=0x45980b\ngithub.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00025bba0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:215 +0x149 fp=0xc00008cfd8 sp=0xc00008ceb8 pc=0xcf17e9\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00008cfe0 sp=0xc00008cfd8 pc=0x4c5cc1\ncreated by github.com/klauspost/compress/zstd.newBlockDec\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/klauspost/compress/zstd/blockdec.go:118 +0x173\n\ngoroutine 12 [chan receive]:\nruntime.gopark(0x7aa4308, 0xc000110058, 0x49170e, 0x2)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc00008d6a8 sp=0xc00008d688 pc=0x48e2e5\nruntime.chanrecv(0xc000110000, 0xc00008d7b8, 0xf46601, 0xc00008d7a0)\n\t/usr/local/go/src/runtime/chan.go:576 +0x36f fp=0xc00008d738 sp=0xc00008d6a8 pc=0x459b8f\nruntime.chanrecv2(0xc000110000, 0xc00008d7b8, 0x0)\n\t/usr/local/go/src/runtime/chan.go:444 +0x2b fp=0xc00008d768 sp=0xc00008d738 pc=0x45980b\ngithub.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:75 +0x74 fp=0xc00008d7e0 sp=0xc00008d768 pc=0xf3eed4\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00008d7e8 sp=0xc00008d7e0 pc=0x4c5cc1\ncreated by github.com/cockroachdb/cockroach/pkg/util/log.init.6\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:41 +0x35\n\ngoroutine 13 [chan receive, 27 minutes]:\nruntime.gopark(0x7aa4308, 0xc00028e178, 0xb4b170e, 0x2)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc00008de90 sp=0xc00008de70 pc=0x48e2e5\nruntime.chanrecv(0xc00028e120, 0xc00008dfc0, 0xb45e601, 0x7aa46d8)\n\t/usr/local/go/src/runtime/chan.go:576 +0x36f fp=0xc00008df20 sp=0xc00008de90 pc=0x459b8f\nruntime.chanrecv2(0xc00028e120, 0xc00008dfc0, 0x1)\n\t/usr/local/go/src/runtime/chan.go:444 +0x2b fp=0xc00008df50 sp=0xc00008df20 pc=0x45980b\ngithub.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:98 +0x12c fp=0xc00008dfe0 sp=0xc00008df50 pc=0xf3f16c\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00008dfe8 sp=0xc00008dfe0 pc=0x4c5cc1\ncreated by github.com/cockroachdb/cockroach/pkg/util/log.init.6\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/log/log_flush.go:42 +0x4d\n\ngoroutine 15 [GC worker (idle)]:\nruntime.gopark(0x7aa43a8, 0xc000486e20, 0x1418, 0x0)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc00008e760 sp=0xc00008e740 pc=0x48e2e5\nruntime.gcBgMarkWorker()\n\t/usr/local/go/src/runtime/mgc.go:1911 +0xfb fp=0xc00008e7e0 sp=0xc00008e760 pc=0x47105b\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00008e7e8 sp=0xc00008e7e0 pc=0x4c5cc1\ncreated by runtime.gcBgMarkStartWorkers\n\t/usr/local/go/src/runtime/mgc.go:1835 +0x37\n\ngoroutine 16 [GC worker (idle)]:\nruntime.gopark(0x7aa43a8, 0xc000394260, 0x1418, 0x0)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc00008ef60 sp=0xc00008ef40 pc=0x48e2e5\nruntime.gcBgMarkWorker()\n\t/usr/local/go/src/runtime/mgc.go:1911 +0xfb fp=0xc00008efe0 sp=0xc00008ef60 pc=0x47105b\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00008efe8 sp=0xc00008efe0 pc=0x4c5cc1\ncreated by runtime.gcBgMarkStartWorkers\n\t/usr/local/go/src/runtime/mgc.go:1835 +0x37\n\ngoroutine 35 [GC worker (idle)]:\nruntime.gopark(0x7aa43a8, 0xc00012a000, 0x1418, 0x0)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000093f60 sp=0xc000093f40 pc=0x48e2e5\nruntime.gcBgMarkWorker()\n\t/usr/local/go/src/runtime/mgc.go:1911 +0xfb fp=0xc000093fe0 sp=0xc000093f60 pc=0x47105b\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000093fe8 sp=0xc000093fe0 pc=0x4c5cc1\ncreated by runtime.gcBgMarkStartWorkers\n\t/usr/local/go/src/runtime/mgc.go:1835 +0x37\n\ngoroutine 50 [GC worker (idle)]:\nruntime.gopark(0x7aa43a8, 0xc000486e40, 0x1418, 0x0)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000092760 sp=0xc000092740 pc=0x48e2e5\nruntime.gcBgMarkWorker()\n\t/usr/local/go/src/runtime/mgc.go:1911 +0xfb fp=0xc0000927e0 sp=0xc000092760 pc=0x47105b\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000927e8 sp=0xc0000927e0 pc=0x4c5cc1\ncreated by runtime.gcBgMarkStartWorkers\n\t/usr/local/go/src/runtime/mgc.go:1835 +0x37\n\ngoroutine 36 [select]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0002a9600 sp=0xc0002a95e0 pc=0x48e2e5\nruntime.selectgo(0xc0002a9798, 0xc0002a977c, 0x0, 0x0, 0x3, 0x1, 0x1, 0x459601)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc0002a9738 sp=0xc0002a9600 pc=0x49f557\ngo.opencensus.io/stats/view.(*worker).start(0xc0003e0880)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:276 +0xcd fp=0xc0002a97d8 sp=0xc0002a9738 pc=0x39af94d\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0002a97e0 sp=0xc0002a97d8 pc=0x4c5cc1\ncreated by go.opencensus.io/stats/view.init.0\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:34 +0x68\n\ngoroutine 20 [chan receive]:\nruntime.gopark(0x7aa4308, 0xc0008d22f8, 0x45170e, 0x2)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0002a4630 sp=0xc0002a4610 pc=0x48e2e5\nruntime.chanrecv(0xc0008d22a0, 0xc0002a4770, 0xf4201, 0xc0008d0101)\n\t/usr/local/go/src/runtime/chan.go:576 +0x36f fp=0xc0002a46c0 sp=0xc0002a4630 pc=0x459b8f\nruntime.chanrecv1(0xc0008d22a0, 0xc0002a4770)\n\t/usr/local/go/src/runtime/chan.go:439 +0x2b fp=0xc0002a46f0 sp=0xc0002a46c0 pc=0x4597cb\ngithub.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/goschedstats/runnable.go:165 +0x16b fp=0xc0002a47e0 sp=0xc0002a46f0 pc=0x3b5ba6b\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0002a47e8 sp=0xc0002a47e0 pc=0x4c5cc1\ncreated by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/goschedstats/runnable.go:157 +0x35\n\ngoroutine 52 [IO wait, 27 minutes]:\nruntime.gopark(0x7aa44e8, 0x7f859c879f58, 0x1b02, 0x5)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000297b50 sp=0xc000297b30 pc=0x48e2e5\nruntime.netpollblock(0x7f859c879f30, 0x72, 0xd0)\n\t/usr/local/go/src/runtime/netpoll.go:438 +0x9b fp=0xc000297b88 sp=0xc000297b50 pc=0x48687b\ninternal/poll.runtime_pollWait(0x7f859c879f30, 0x72, 0x0)\n\t/usr/local/go/src/runtime/netpoll.go:222 +0x55 fp=0xc000297bb0 sp=0xc000297b88 pc=0x4bfe35\ninternal/poll.(*pollDesc).wait(0xc000883598, 0x72, 0x0, 0x0, 0x5210cbf)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 fp=0xc000297be0 sp=0xc000297bb0 pc=0x539dc5\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Accept(0xc000883580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_unix.go:401 +0x212 fp=0xc000297c60 sp=0xc000297be0 pc=0x53cc32\nnet.(*netFD).accept(0xc000883580, 0x30, 0x30, 0x7f85c516c5b8)\n\t/usr/local/go/src/net/fd_unix.go:172 +0x45 fp=0xc000297d28 sp=0xc000297c60 pc=0x5ee805\nnet.(*TCPListener).accept(0xc0009262e8, 0xc000297da0, 0x461658, 0x30)\n\t/usr/local/go/src/net/tcpsock_posix.go:139 +0x32 fp=0xc000297d68 sp=0xc000297d28 pc=0x60d7b2\nnet.(*TCPListener).Accept(0xc0009262e8, 0x4ee40a0, 0xc000a47590, 0x4a4b980, 0xb066520)\n\t/usr/local/go/src/net/tcpsock.go:261 +0x65 fp=0xc000297da8 sp=0xc000297d68 pc=0x60c5a5\nnet/http.(*onceCloseListener).Accept(0xc000a474a0, 0xc000074100, 0x4a4b980, 0xb066520, 0x517c340)\n\t<autogenerated>:1 +0x3c fp=0xc000297de0 sp=0xc000297da8 pc=0x95d8bc\nnet/http.(*Server).Serve(0xc0008d5260, 0x8a73d40, 0xc0009262e8, 0x0, 0x0)\n\t/usr/local/go/src/net/http/server.go:2961 +0x285 fp=0xc000297f08 sp=0xc000297de0 pc=0x9355e5\nnet/http.(*Server).ListenAndServe(0xc0008d5260, 0xc0008d5260, 0x1)\n\t/usr/local/go/src/net/http/server.go:2890 +0xba fp=0xc000297f58 sp=0xc000297f08 pc=0x9352fa\nnet/http.ListenAndServe(...)\n\t/usr/local/go/src/net/http/server.go:3144\ngithub.com/cockroachdb/cockroach/pkg/workload/cli.startPProfEndPoint.func1(0x8a99a00, 0xc000074100)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:341 +0xca fp=0xc000297fd0 sp=0xc000297f58 pc=0x3b1944a\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000297fd8 sp=0xc000297fd0 pc=0x4c5cc1\ncreated by github.com/cockroachdb/cockroach/pkg/workload/cli.startPProfEndPoint\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:340 +0xbf\n\ngoroutine 21 [select, 27 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0010d3618 sp=0xc0010d35f8 pc=0x48e2e5\nruntime.selectgo(0xc0010d3798, 0xc0010d3790, 0x0, 0x0, 0x2, 0x1, 0x0, 0x0)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc0010d3750 sp=0xc0010d3618 pc=0x49f557\ndatabase/sql.(*DB).connectionOpener(0xc001296000, 0x8a999c8, 0xc0010c8000)\n\t/usr/local/go/src/database/sql/sql.go:1133 +0xb5 fp=0xc0010d37c8 sp=0xc0010d3750 pc=0xfb1ad5\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0010d37d0 sp=0xc0010d37c8 pc=0x4c5cc1\ncreated by database/sql.OpenDB\n\t/usr/local/go/src/database/sql/sql.go:740 +0x12a\n\ngoroutine 22 [IO wait, 27 minutes]:\nruntime.gopark(0x7aa44e8, 0x7f859c879e70, 0x1b02, 0x5)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0000a5b18 sp=0xc0000a5af8 pc=0x48e2e5\nruntime.netpollblock(0x7f859c879e48, 0x72, 0xd0)\n\t/usr/local/go/src/runtime/netpoll.go:438 +0x9b fp=0xc0000a5b50 sp=0xc0000a5b18 pc=0x48687b\ninternal/poll.runtime_pollWait(0x7f859c879e48, 0x72, 0x0)\n\t/usr/local/go/src/runtime/netpoll.go:222 +0x55 fp=0xc0000a5b78 sp=0xc0000a5b50 pc=0x4bfe35\ninternal/poll.(*pollDesc).wait(0xc0003e0098, 0x72, 0x0, 0x0, 0x5210cbf)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45 fp=0xc0000a5ba8 sp=0xc0000a5b78 pc=0x539dc5\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Accept(0xc0003e0080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_unix.go:401 +0x212 fp=0xc0000a5c28 sp=0xc0000a5ba8 pc=0x53cc32\nnet.(*netFD).accept(0xc0003e0080, 0x30, 0x30, 0x7f85c516c108)\n\t/usr/local/go/src/net/fd_unix.go:172 +0x45 fp=0xc0000a5cf0 sp=0xc0000a5c28 pc=0x5ee805\nnet.(*TCPListener).accept(0xc0017eb548, 0xc0000a5d68, 0x461658, 0x30)\n\t/usr/local/go/src/net/tcpsock_posix.go:139 +0x32 fp=0xc0000a5d30 sp=0xc0000a5cf0 pc=0x60d7b2\nnet.(*TCPListener).Accept(0xc0017eb548, 0x4ee40a0, 0xc0019106c0, 0x4a4b980, 0xb066520)\n\t/usr/local/go/src/net/tcpsock.go:261 +0x65 fp=0xc0000a5d70 sp=0xc0000a5d30 pc=0x60c5a5\nnet/http.(*onceCloseListener).Accept(0xc001910600, 0xc000074100, 0x4a4b980, 0xb066520, 0x517c340)\n\t<autogenerated>:1 +0x3c fp=0xc0000a5da8 sp=0xc0000a5d70 pc=0x95d8bc\nnet/http.(*Server).Serve(0xc00087c380, 0x8a73d40, 0xc0017eb548, 0x0, 0x0)\n\t/usr/local/go/src/net/http/server.go:2961 +0x285 fp=0xc0000a5ed0 sp=0xc0000a5da8 pc=0x9355e5\nnet/http.(*Server).ListenAndServe(0xc00087c380, 0xc00087c380, 0x0)\n\t/usr/local/go/src/net/http/server.go:2890 +0xba fp=0xc0000a5f20 sp=0xc0000a5ed0 pc=0x9352fa\nnet/http.ListenAndServe(...)\n\t/usr/local/go/src/net/http/server.go:3144\ngithub.com/cockroachdb/cockroach/pkg/workload/cli.runRun.func1(0xc000110660)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:398 +0x179 fp=0xc0000a5fd8 sp=0xc0000a5f20 pc=0x3b19679\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000a5fe0 sp=0xc0000a5fd8 pc=0x4c5cc1\ncreated by github.com/cockroachdb/cockroach/pkg/workload/cli.runRun\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/workload/cli/run.go:397 +0x4ef\n\ngoroutine 53 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0002a4e20 sp=0xc0002a4e00 pc=0x48e2e5\nruntime.selectgo(0xc0002a4fa8, 0xc0002a4f98, 0x0, 0x0, 0x2, 0x1, 0x4958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc0002a4f58 sp=0xc0002a4e20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000be6000)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc0002a4fd8 sp=0xc0002a4f58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0002a4fe0 sp=0xc0002a4fd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 33 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0002abe20 sp=0xc0002abe00 pc=0x48e2e5\nruntime.selectgo(0xc0002abfa8, 0xc0002abf98, 0x0, 0x0, 0x2, 0x1, 0x100000000000000, 0x0)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc0002abf58 sp=0xc0002abe20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000d440e0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc0002abfd8 sp=0xc0002abf58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0002abfe0 sp=0xc0002abfd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 9259 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc0003809c0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc0062a7fd0 sp=0xc0062a7fc8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0062a7fd8 sp=0xc0062a7fd0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 37 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0002a8e20 sp=0xc0002a8e00 pc=0x48e2e5\nruntime.selectgo(0xc0002a8fa8, 0xc0002a8f98, 0x0, 0x0, 0x2, 0x1, 0x100000000000000, 0x547f1de)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc0002a8f58 sp=0xc0002a8e20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000c680e0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc0002a8fd8 sp=0xc0002a8f58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0002a8fe0 sp=0xc0002a8fd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 9141 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc006e9a360)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc002a7afd0 sp=0xc002a7afc8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc002a7afd8 sp=0xc002a7afd0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 86 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc00008fe20 sp=0xc00008fe00 pc=0x48e2e5\nruntime.selectgo(0xc00008ffa8, 0xc00008ff98, 0x0, 0x0, 0x2, 0x1, 0x100000000000000, 0x0)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc00008ff58 sp=0xc00008fe20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc0004fad90)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc00008ffd8 sp=0xc00008ff58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00008ffe0 sp=0xc00008ffd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 87 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0002a7620 sp=0xc0002a7600 pc=0x48e2e5\nruntime.selectgo(0xc0002a77a8, 0xc0002a7798, 0x0, 0x0, 0x2, 0x1, 0x100000000000000, 0x0)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc0002a7758 sp=0xc0002a7620 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc0004fb110)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc0002a77d8 sp=0xc0002a7758 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0002a77e0 sp=0xc0002a77d8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 54 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0002a9e20 sp=0xc0002a9e00 pc=0x48e2e5\nruntime.selectgo(0xc0002a9fa8, 0xc0002a9f98, 0x0, 0x0, 0x2, 0xc0012f0901, 0x1000000004958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc0002a9f58 sp=0xc0002a9e20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000be6230)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc0002a9fd8 sp=0xc0002a9f58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0002a9fe0 sp=0xc0002a9fd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 3137 [select]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0010d2e20 sp=0xc0010d2e00 pc=0x48e2e5\nruntime.selectgo(0xc0010d2fa8, 0xc0010d2f98, 0x0, 0x0, 0x2, 0x1, 0x4958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc0010d2f58 sp=0xc0010d2e20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc0041b5a40)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc0010d2fd8 sp=0xc0010d2f58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0010d2fe0 sp=0xc0010d2fd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 56 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0002aa620 sp=0xc0002aa600 pc=0x48e2e5\nruntime.selectgo(0xc0002aa7a8, 0xc0002aa798, 0x0, 0x0, 0x2, 0x1, 0x100000000000000, 0x0)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc0002aa758 sp=0xc0002aa620 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000be64d0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc0002aa7d8 sp=0xc0002aa758 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0002aa7e0 sp=0xc0002aa7d8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 89 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc00008f620 sp=0xc00008f600 pc=0x48e2e5\nruntime.selectgo(0xc00008f7a8, 0xc00008f798, 0x0, 0x0, 0x2, 0x1, 0x100000000000000, 0x0)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc00008f758 sp=0xc00008f620 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc0004fb490)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc00008f7d8 sp=0xc00008f758 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00008f7e0 sp=0xc00008f7d8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 100 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0002aae20 sp=0xc0002aae00 pc=0x48e2e5\nruntime.selectgo(0xc0002aafa8, 0xc0002aaf98, 0x0, 0x0, 0x2, 0x1, 0x100000000000000, 0x0)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc0002aaf58 sp=0xc0002aae20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000d44380)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc0002aafd8 sp=0xc0002aaf58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0002aafe0 sp=0xc0002aafd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 93 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0010cfe20 sp=0xc0010cfe00 pc=0x48e2e5\nruntime.selectgo(0xc0010cffa8, 0xc0010cff98, 0x0, 0x0, 0x2, 0x1, 0x100000000000000, 0x0)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc0010cff58 sp=0xc0010cfe20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc0004fb7a0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc0010cffd8 sp=0xc0010cff58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0010cffe0 sp=0xc0010cffd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 102 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0002a6620 sp=0xc0002a6600 pc=0x48e2e5\nruntime.selectgo(0xc0002a67a8, 0xc0002a6798, 0x0, 0x0, 0x2, 0x1, 0x100000000000000, 0x0)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc0002a6758 sp=0xc0002a6620 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000d44620)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc0002a67d8 sp=0xc0002a6758 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0002a67e0 sp=0xc0002a67d8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 104 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0002a5620 sp=0xc0002a5600 pc=0x48e2e5\nruntime.selectgo(0xc0002a57a8, 0xc0002a5798, 0x0, 0x0, 0x2, 0x1, 0x100000000000000, 0x0)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc0002a5758 sp=0xc0002a5620 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000d447e0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc0002a57d8 sp=0xc0002a5758 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0002a57e0 sp=0xc0002a57d8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 106 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc0002a7e20 sp=0xc0002a7e00 pc=0x48e2e5\nruntime.selectgo(0xc0002a7fa8, 0xc0002a7f98, 0x0, 0x0, 0x2, 0xc00092d201, 0x6, 0x0)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc0002a7f58 sp=0xc0002a7e20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000d44af0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc0002a7fd8 sp=0xc0002a7f58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0002a7fe0 sp=0xc0002a7fd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 9216 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc006e9bd40)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc0066bb7d0 sp=0xc0066bb7c8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0066bb7d8 sp=0xc0066bb7d0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 108 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000b94e20 sp=0xc000b94e00 pc=0x48e2e5\nruntime.selectgo(0xc000b94fa8, 0xc000b94f98, 0x0, 0x0, 0x2, 0x1, 0x0, 0x0)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc000b94f58 sp=0xc000b94e20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000d44cb0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc000b94fd8 sp=0xc000b94f58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000b94fe0 sp=0xc000b94fd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 110 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000b95e20 sp=0xc000b95e00 pc=0x48e2e5\nruntime.selectgo(0xc000b95fa8, 0xc000b95f98, 0x0, 0x0, 0x2, 0x1, 0x0, 0x0)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc000b95f58 sp=0xc000b95e20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000d44e70)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc000b95fd8 sp=0xc000b95f58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000b95fe0 sp=0xc000b95fd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 9218 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc006e9be00)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc0066bbfd0 sp=0xc0066bbfc8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0066bbfd8 sp=0xc0066bbfd0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 3185 [select]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc002abe620 sp=0xc002abe600 pc=0x48e2e5\nruntime.selectgo(0xc002abe7a8, 0xc002abe798, 0x0, 0x0, 0x2, 0x1, 0x4958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc002abe758 sp=0xc002abe620 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000ad8700)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc002abe7d8 sp=0xc002abe758 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc002abe7e0 sp=0xc002abe7d8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 3135 [select]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc002abde20 sp=0xc002abde00 pc=0x48e2e5\nruntime.selectgo(0xc002abdfa8, 0xc002abdf98, 0x0, 0x0, 0x2, 0x1, 0x4958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc002abdf58 sp=0xc002abde20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc0041b57a0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc002abdfd8 sp=0xc002abdf58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc002abdfe0 sp=0xc002abdfd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 2043 [select]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc001210e20 sp=0xc001210e00 pc=0x48e2e5\nruntime.selectgo(0xc001210fa8, 0xc001210f98, 0x0, 0x0, 0x2, 0x1, 0x4958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc001210f58 sp=0xc001210e20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000d45b90)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc001210fd8 sp=0xc001210f58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc001210fe0 sp=0xc001210fd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 9152 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc006e9b140)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc0021217d0 sp=0xc0021217c8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0021217d8 sp=0xc0021217d0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 9243 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc004a45a40)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc00211cfd0 sp=0xc00211cfc8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00211cfd8 sp=0xc00211cfd0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 3147 [select]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc002105620 sp=0xc002105600 pc=0x48e2e5\nruntime.selectgo(0xc0021057a8, 0xc002105798, 0x0, 0x0, 0x2, 0x1, 0x4958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc002105758 sp=0xc002105620 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc00095c7e0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc0021057d8 sp=0xc002105758 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0021057e0 sp=0xc0021057d8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 1757 [select]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000cbde20 sp=0xc000cbde00 pc=0x48e2e5\nruntime.selectgo(0xc000cbdfa8, 0xc000cbdf98, 0x0, 0x0, 0x2, 0x1, 0x4958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc000cbdf58 sp=0xc000cbde20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000ae08c0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc000cbdfd8 sp=0xc000cbdf58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000cbdfe0 sp=0xc000cbdfd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 9247 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc004a45bc0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc002116fd0 sp=0xc002116fc8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc002116fd8 sp=0xc002116fd0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 9242 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc004a459e0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc00629d7d0 sp=0xc00629d7c8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00629d7d8 sp=0xc00629d7d0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 1525 [select]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000cace20 sp=0xc000cace00 pc=0x48e2e5\nruntime.selectgo(0xc000cacfa8, 0xc000cacf98, 0x0, 0x0, 0x2, 0x1, 0x4958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc000cacf58 sp=0xc000cace20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000ad30a0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc000cacfd8 sp=0xc000cacf58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000cacfe0 sp=0xc000cacfd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 9253 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc004a45e00)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc0012117d0 sp=0xc0012117c8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0012117d8 sp=0xc0012117d0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 9226 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc004a44300)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc000b93fd0 sp=0xc000b93fc8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000b93fd8 sp=0xc000b93fd0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 9220 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc006e9bec0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc000bf37d0 sp=0xc000bf37c8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000bf37d8 sp=0xc000bf37d0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 576 [select]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000cc8620 sp=0xc000cc8600 pc=0x48e2e5\nruntime.selectgo(0xc000cc87a8, 0xc000cc8798, 0x0, 0x0, 0x2, 0x1, 0x4958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc000cc8758 sp=0xc000cc8620 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000941180)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc000cc87d8 sp=0xc000cc8758 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000cc87e0 sp=0xc000cc87d8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 297 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000b92620 sp=0xc000b92600 pc=0x48e2e5\nruntime.selectgo(0xc000b927a8, 0xc000b92798, 0x0, 0x0, 0x2, 0x1, 0x4958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc000b92758 sp=0xc000b92620 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000c80930)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc000b927d8 sp=0xc000b92758 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000b927e0 sp=0xc000b927d8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 1123 [select]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000b90e20 sp=0xc000b90e00 pc=0x48e2e5\nruntime.selectgo(0xc000b90fa8, 0xc000b90f98, 0x0, 0x0, 0x2, 0x1, 0x4958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc000b90f58 sp=0xc000b90e20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000c98ee0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc000b90fd8 sp=0xc000b90f58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000b90fe0 sp=0xc000b90fd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 3132 [select]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc002a2b620 sp=0xc002a2b600 pc=0x48e2e5\nruntime.selectgo(0xc002a2b7a8, 0xc002a2b798, 0x0, 0x0, 0x2, 0x1, 0x4958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc002a2b758 sp=0xc002a2b620 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc0041b5570)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc002a2b7d8 sp=0xc002a2b758 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc002a2b7e0 sp=0xc002a2b7d8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 332 [select, 1 minutes]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000bf5620 sp=0xc000bf5600 pc=0x48e2e5\nruntime.selectgo(0xc000bf57a8, 0xc000bf5798, 0x0, 0x0, 0x2, 0x1, 0x4958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc000bf5758 sp=0xc000bf5620 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000c812d0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc000bf57d8 sp=0xc000bf5758 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000bf57e0 sp=0xc000bf57d8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 9223 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc004a440c0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc000bf67d0 sp=0xc000bf67c8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000bf67d8 sp=0xc000bf67d0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 879 [select]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000bf7e20 sp=0xc000bf7e00 pc=0x48e2e5\nruntime.selectgo(0xc000bf7fa8, 0xc000bf7f98, 0x0, 0x0, 0x2, 0x1, 0x4958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc000bf7f58 sp=0xc000bf7e20 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000be7110)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc000bf7fd8 sp=0xc000bf7f58 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000bf7fe0 sp=0xc000bf7fd8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 9264 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc000d3baa0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc000a617d0 sp=0xc000a617c8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000a617d8 sp=0xc000a617d0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 626 [select]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc000bef620 sp=0xc000bef600 pc=0x48e2e5\nruntime.selectgo(0xc000bef7a8, 0xc000bef798, 0x0, 0x0, 0x2, 0x1, 0x4958f6, 0x7aa4400)\n\t/usr/local/go/src/runtime/select.go:327 +0xef7 fp=0xc000bef758 sp=0xc000bef620 pc=0x49f557\ngithub.com/jackc/pgx/v4/pgxpool.(*Pool).backgroundHealthCheck(0xc000a92000)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:342 +0xbb fp=0xc000bef7d8 sp=0xc000bef758 pc=0x3a95fdb\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000bef7e0 sp=0xc000bef7d8 pc=0x4c5cc1\ncreated by github.com/jackc/pgx/v4/pgxpool.ConnectConfig\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/jackc/pgx/v4/pgxpool/pool.go:223 +0x27f\n\ngoroutine 9228 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc004a446c0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc000bf77d0 sp=0xc000bf77c8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000bf77d8 sp=0xc000bf77d0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 9207 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc006e9b9e0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc0066b97d0 sp=0xc0066b97c8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0066b97d8 sp=0xc0066b97d0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 9217 [runnable]:\ngolang.org/x/sync/errgroup.(*Group).Go.func1(0xc005f36b40, 0xc006e9bda0)\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 fp=0xc000b8efd0 sp=0xc000b8efc8 pc=0x12dd3e0\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000b8efd8 sp=0xc000b8efd0 pc=0x4c5cc1\ncreated by golang.org/x/sync/errgroup.(*Group).Go\n\t/home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66\n\ngoroutine 3186 [select]:\nruntime.gopark(0x7aa4560, 0x0, 0x1809, 0x1)\n\t/usr/local/go/src/runtime/proc.go:336 +0xe5 fp=0xc00
tbg commented 3 years ago

Alex got a reproduction the first night, which is very encouraging. We didn't get a diff (sadface) but we have the cluster, so the detective work can begin in earnest.

Here's the relevant logs from the range (removed some noisy messages):

https://gist.githubusercontent.com/tbg/54b036f9703c5d0ea5e68a7b07a13152/raw/982e2b629740283c1ddf574287a641d1391d08c8/gistfile0.txt

The important bit is that we have the stats recomputations for the three replicas and

One explanation could be that n1 and n6 both dropped a single delete (which made an inline intent erroneously visible). I don't know, that seems too synchronized, but the delta is again on a separated intent, and that lends further weight to it being something around separated intents. We will hopefully learn more as we dig into the data.

Here is the cluster, I just extended its lifetime "indefinitely". The checkpoints are on n1, n6, n7. I will post minutiae on the slack channel.

ROACHPROD_USER=sarkesian roachprod extend sarkesian-1630550505-05-n12cpu4-geo --lifetime 10000h
tbg commented 3 years ago

We are finding further evidence that the anomaly is related to the use of SingleDelete. This is because

a) the offending separated intent varies across all three stores - on n1, it is from epoch 1, on n6 it is from epoch 2, and on n7 it is absent. b) the intent has a nil TxnDidNotUpdateMeta, which qualifies it for use of SingleDelete (like the original failure did, too). This is a rare case in this workload, it seems, as most of the lock table has a boolean set there (which I assume is true, but not sure, maybe there are also code paths that set it to &false).

Our (KV's) current strategy is to reproduce the failure a few more times, and to then remove the use of SingleDelete and prove that the reproducability disappears. This should then conclusively blame SingleDelete. Whether that will be the outcome remains to be seen.

@sumeerbhola let's talk on https://cockroachlabs.slack.com/archives/C02DATMT9CH if you have ideas on what to look into on the stores for the above repro.

tbg commented 3 years ago

re b) this isn't as clear-cut as I made it out to be, the nil actually means no single delete, but I also don't understand these code paths well enough to really say. While the repro machine spins, will try to find out more about what operations applied on the diverging key. Hoping to see . Let's see if I still know how to use ./cockroach debug pebble find.

tbg commented 3 years ago

Looks like we are close to understanding this now. We appear to be laying down a SINGLEDEL in situations where it isn't the correct thing to do. This has to do with the fact that sometimes we replace intents (i.e. SET; DEL; SET) instead of adjust (SET; SET). I.e. the bug is not in SINGLEDEL itself, but in the optimization that decides when it is to be used in CRDB. The details are likely to emerge on slack around here.

nicktrav commented 3 years ago

I was able to add a metamorphic test to Pebble to reproduce the SET -> ... -> SET -> DELETE -> SET -> SINGLEDEL case.

The failure mode is appears to be the same - a resurrected key that was deleted prior to the GET.

Here's a gist with the op log from the failed run, along with the OPTIONS files for both DBs.

Still polishing up the metamorphic test additions. I'll push that up soon.

tbg commented 3 years ago

We got two more instances overnight. This doesn't seem too interesting to dig into more, but they confirm the same pattern and would thus presumably also be fixed by a fix for the metamorphic test failure in pebble Nick describes above.

consistency check failed; fetching details and shutting down minority (n7,s7):2
created checkpoint ‹/mnt/data1/cockroach/auxiliary/checkpoints/r251_at_195401›

(n3,s3):5: checksum f019300685b92f33bc14d58701c4dc673494c17c328fbd50fdf54eaa86a74b7711bbec796c3a902bfded9d0b318b8dfcc7efb6921185dec53853b0bebc235f3a
- stats: contains_estimates:0 last_update_nanos:1630611934333263781 intent_age:11892 gc_bytes_age:6291877160 live_bytes:83928903 live_count:2997287 key_bytes:71237381 key_count:3062995 val_bytes:15321522 val_count:3128703 intent_bytes:1260 intent_count:80 separated_intent_count:80 sys_bytes:545993 sys_count:7453 abort_span_bytes:544867 
- stats.Sub(recomputation): last_update_nanos:1630611934333263781 
(n6,s6):3: checksum f019300685b92f33bc14d58701c4dc673494c17c328fbd50fdf54eaa86a74b7711bbec796c3a902bfded9d0b318b8dfcc7efb6921185dec53853b0bebc235f3a
- stats: contains_estimates:0 last_update_nanos:1630611934333263781 intent_age:11892 gc_bytes_age:6291877160 live_bytes:83928903 live_count:2997287 key_bytes:71237381 key_count:3062995 val_bytes:15321522 val_count:3128703 intent_bytes:1260 intent_count:80 separated_intent_count:80 sys_bytes:545993 sys_count:7453 abort_span_bytes:544867 
- stats.Sub(recomputation): last_update_nanos:1630611934333263781 
====== diff(f019300685b92f33bc14d58701c4dc673494c17c328fbd50fdf54eaa86a74b7711bbec796c3a902bfded9d0b318b8dfcc7efb6921185dec53853b0bebc235f3a, [minority]) ======
--- leaseholder
+++ follower
+0,0 ‹/Table/61/1/352/6/3016/0›
+    ts:1970-01-01 00:00:00 +0000 UTC
+    value:‹1630610361.617540860,2 {Txn:id=e6b470d1 key=/Table/58/1/352/6/0 pri=0.09011640 epo=1 ts=1630610361.617540860,2 min=1630610327.007793507,0 seq=4 Timestamp:1630610361.617540860,2 Deleted:false KeyBytes:12 ValBytes:5 RawBytes:[] IntentHistory:[] MergeTimestamp:<nil> TxnDidNotUpdateMeta:<nil>}›
+    raw mvcc_key/value: ‹c589f701608ef70bc88800› ‹0a3d0a10e6b470d1d1f94a67a6f7fe8e186420171a07c289f701608e8820012a0c08fccda3dac0efc5d016100230838f7638044a0a08e3a288e3bfeec5d016120c08fccda3dac0efc5d01610021800200c2805›
(n7,s7):2: checksum 8e68bc224f174ba0362b74d0fd4261be1fd1abeaef572051a1b420ce31d210794615a23a8e5b7545cc9da4011c73d56c11d4dcff872ecd4266f11192748ff52b [minority]
- stats: contains_estimates:0 last_update_nanos:1630611934333263781 intent_age:11892 gc_bytes_age:6291877160 live_bytes:83928903 live_count:2997287 key_bytes:71237381 key_count:3062995 val_bytes:15321522 val_count:3128703 intent_bytes:1260 intent_count:80 separated_intent_count:80 sys_bytes:545993 sys_count:7453 abort_span_bytes:544867 
- stats.Sub(recomputation): last_update_nanos:1630611934333263781 intent_age:-1573 live_bytes:-83 val_bytes:-83 intent_bytes:-17 intent_count:-1 separated_intent_count:-1 
consistency check failed

and in the second one, we didn't get the diff:

(n5,s5):5: checksum ee6f6a1ea969985cd7d870e0d0cf5f5a7c289d95775d636a72a0911c1e9af958aa7c4a9bc951438d5ef4a34505762cb974b9dc0e92ebe6aa2f76956730ece9fb
- stats: contains_estimates:0 last_update_nanos:1630618577914061825 intent_age:35523 gc_bytes_age:2895486080 live_bytes:80749428 live_count:2999830 key_bytes:67419226 key_count:3050430 val_bytes:15312082 val_count:3101030 intent_bytes:11676 intent_count:728 separated_intent_count:728 sys_bytes:368822 sys_count:5090 abort_span_bytes:366662 
- stats.Sub(recomputation): last_update_nanos:1630618577914061825 intent_age:-1728 live_bytes:-81 val_bytes:-81 intent_bytes:-17 intent_count:-1 separated_intent_count:-1 
(n3,s3):6: checksum ef3af414590647b55327ba2c3ca5acad27b9194e00b3850d4628773de6d10b8f96edd9fe77e282d4fb3bfd065efb1aa1c1e87a1d6a370bd11d23453f81c30f96
- stats: contains_estimates:0 last_update_nanos:1630618577914061825 intent_age:35523 gc_bytes_age:2895486080 live_bytes:80749428 live_count:2999830 key_bytes:67419226 key_count:3050430 val_bytes:15312082 val_count:3101030 intent_bytes:11676 intent_count:728 separated_intent_count:728 sys_bytes:368822 sys_count:5090 abort_span_bytes:366662 
- stats.Sub(recomputation): last_update_nanos:1630618577914061825 intent_age:-1822 live_bytes:-81 val_bytes:-81 intent_bytes:-17 intent_count:-1 separated_intent_count:-1 
(n9,s9):8: checksum 7f5c4856d0417862ff9873671a80f56907138a3883c58af9ba84c07fee44a74d740691fa7e9d3b0b57614db7df586e0d6571ba33a28234952e0683152881ed33 [minority]
- stats: contains_estimates:0 last_update_nanos:1630618577914061825 intent_age:35523 gc_bytes_age:2895486080 live_bytes:80749428 live_count:2999830 key_bytes:67419226 key_count:3050430 val_bytes:15312082 val_count:3101030 intent_bytes:11676 intent_count:728 separated_intent_count:728 sys_bytes:368822 sys_count:5090 abort_span_bytes:366662 
- stats.Sub(recomputation): last_update_nanos:1630618577914061825 
consistency check failed; fetching details and shutting down minority (n9,s9):8

replica inconsistency detected; could not obtain actual diff: computing own checksum: ‹rpc error: code = DeadlineExceeded desc = context deadline exceeded›
tbg commented 3 years ago

Updating this to release-blocker, as discussed on the release triage meeting yesterday. This will mean no beta will be released until this issue is addressed.