search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.05k stars 3.8k forks source link

roachtest: tpccbench/nodes=3/cpu=16 failed #41912

Closed cockroach-teamcity closed 4 years ago

cockroach-teamcity commented 5 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/f9a102814bdce90d687f6215acadf10a9d784c29

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1555990&tab=artifacts#/tpccbench/nodes=3/cpu=16

The test failed on branch=master, cloud=aws:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191024-1555990/tpccbench/nodes=3/cpu=16/run_1
    test_runner.go:704: test timed out (10h0m0s)
ajwerner commented 4 years ago

This is another case of the import taking too long:

I191024 18:33:18.290805 1 ccl/workloadccl/fixture.go:325  imported 1.8 TiB bytes in 9 tables (took 9h44m5.532245739s, 53.05 MiB/s)
cockroach-teamcity commented 4 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/a57647381a4714b48f6ec6dec0bf766eaa6746dd

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1562380&tab=artifacts#/tpccbench/nodes=3/cpu=16

The test failed on branch=master, cloud=aws:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191029-1562380/tpccbench/nodes=3/cpu=16/run_1
    test_runner.go:712: test timed out (10h0m0s)
cockroach-teamcity commented 4 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/39e43fac9834b338e381641f0f573c060320d678

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1563589&tab=artifacts#/tpccbench/nodes=3/cpu=16

The test failed on branch=master, cloud=aws:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191030-1563589/tpccbench/nodes=3/cpu=16/run_1
    test_runner.go:712: test timed out (10h0m0s)
cockroach-teamcity commented 4 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/0f473848083559c8a98be032949df9428068c223

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1565995&tab=artifacts#/tpccbench/nodes=3/cpu=16

The test failed on branch=master, cloud=aws:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191031-1565995/tpccbench/nodes=3/cpu=16/run_1
    test_runner.go:712: test timed out (10h0m0s)
cockroach-teamcity commented 4 years ago

SHA: https://github.com/cockroachdb/cockroach/commits/4c52530a33367c58e434111324fcda8c3d73582a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1568297&tab=artifacts#/tpccbench/nodes=3/cpu=16

The test failed on branch=master, cloud=aws:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191101-1568297/tpccbench/nodes=3/cpu=16/run_1
    test_runner.go:712: test timed out (10h0m0s)
cockroach-teamcity commented 4 years ago

(roachtest).tpccbench/nodes=3/cpu=16 failed on master@e2c2be6cbef07c7f2ee6341816f81f72a206c32a:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200116-1693614/tpccbench/nodes=3/cpu=16/run_1
    cluster.go:1296,context.go:135,cluster.go:1285,test_runner.go:753: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1693614-1579160767-22-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
        2: dead
        1: dead
        3: dead
        Error:  2: dead, 1: dead, 3: dead
Repro

Artifacts: [/tpccbench/nodes=3/cpu=16](https://teamcity.cockroachdb.com/viewLog.html?buildId=1693614&tab=artifacts#/tpccbench/nodes=3/cpu=16) ``` make stressrace TESTS=tpccbench/nodes=3/cpu=16 PKG=./pkg/roachtest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1 ``` powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 4 years ago

(roachtest).tpccbench/nodes=3/cpu=16 failed on master@3bb183403b4bc75c25fbcfc69d7d35de76d2b984:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200119-1699769/tpccbench/nodes=3/cpu=16/run_1
    cluster.go:1296,context.go:135,cluster.go:1285,test_runner.go:753: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1699769-1579420121-23-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
        1: dead
        3: 22460
        2: 22280
        Error:  1: dead
Repro

Artifacts: [/tpccbench/nodes=3/cpu=16](https://teamcity.cockroachdb.com/viewLog.html?buildId=1699769&tab=artifacts#/tpccbench/nodes=3/cpu=16) [roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Atpccbench%2Fnodes%3D3%2Fcpu%3D16.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 4 years ago

(roachtest).tpccbench/nodes=3/cpu=16 failed on master@c76ad970e73e606bc55a372d93f9d2d6acb32c9c:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200122-1703406/tpccbench/nodes=3/cpu=16/run_1
    cluster.go:1296,context.go:135,cluster.go:1285,test_runner.go:753: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1703406-1579679704-20-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
        2: dead
        1: 22273
        3: 21475
        Error:  2: dead
Repro

Artifacts: [/tpccbench/nodes=3/cpu=16](https://teamcity.cockroachdb.com/viewLog.html?buildId=1703406&tab=artifacts#/tpccbench/nodes=3/cpu=16) [roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Atpccbench%2Fnodes%3D3%2Fcpu%3D16.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 4 years ago

(roachtest).tpccbench/nodes=3/cpu=16 failed on master@4ec5929a08687b82b05ef80231fddd0d53421973:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/tpccbench/nodes=3/cpu=16/run_1
    tpcc.go:742,tpcc.go:570,test_runner.go:741: mkdir /home/agent/temp/buildTmp/roachtest-tpcc748049937: no space left on device
        failed to create temp dir
        main.runTPCCBench
            /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:742
        main.registerTPCCBenchSpec.func1
            /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:570
        main.(*testRunner).runTest.func2
            /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:741
        runtime.goexit
            /usr/local/go/src/runtime/asm_amd64.s:1357
More

Artifacts: [/tpccbench/nodes=3/cpu=16](https://teamcity.cockroachdb.com/viewLog.html?buildId=1725352&tab=artifacts#/tpccbench/nodes=3/cpu=16) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Atpccbench%2Fnodes%3D3%2Fcpu%3D16.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 4 years ago

(roachtest).tpccbench/nodes=3/cpu=16 failed on master@2739821b911d777fa2a927295d699b559360a802:

          - monitor task failed:
          - error with attached stack trace:
            main.(*cluster).RunE
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2010
            main.loadTPCCBench
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:637
            main.runTPCCBench.func1
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:727
            main.(*monitor).Go.func1
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
            github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
            runtime.goexit
                /usr/local/go/src/runtime/asm_amd64.s:1357
          - error with embedded safe details: output in %s
            -- arg 1: <string>
          - output in run_131116.372_n4_workload_fixtures_load_tpcc:
          - error with attached stack trace:
            main.execCmd
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
            main.(*cluster).RunL
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
            main.(*cluster).RunE
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
            main.loadTPCCBench
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:637
            main.runTPCCBench.func1
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:727
            main.(*monitor).Go.func1
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
            github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
            runtime.goexit
                /usr/local/go/src/runtime/asm_amd64.s:1357
          - error with embedded safe details: %s returned:
            stderr:
            %s
            stdout:
            %s
            -- arg 1: <string>
            -- arg 2: <string>
            -- arg 3: <string>
          - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1747374-1581581035-101-n4cpu16:4 -- ./workload fixtures load tpcc --warehouses=2000 --scatter --checks=false {pgurl:1} returned:
            stderr:
            I200213 13:11:18.462580 1 ccl/workloadccl/cliccl/fixtures.go:279  starting restore of 9 tables
            Error: restoring fixture: sql: expected 6 destination arguments in Scan, not 7
            Error:  exit status 1

            stdout::
          - exit status 1
More

Artifacts: [/tpccbench/nodes=3/cpu=16](https://teamcity.cockroachdb.com/viewLog.html?buildId=1747374&tab=artifacts#/tpccbench/nodes=3/cpu=16) Related: - #44974 roachtest: tpccbench/nodes=3/cpu=16 failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-roachtest](https://api.github.com/repos/cockroachdb/cockroach/labels/O-roachtest) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-19.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-19.1) [release-blocker](https://api.github.com/repos/cockroachdb/cockroach/labels/release-blocker) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Atpccbench%2Fnodes%3D3%2Fcpu%3D16.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

cockroach-teamcity commented 4 years ago

(roachtest).tpccbench/nodes=3/cpu=16 failed on master@e671a4ef97cbc6cf5d22f8f322fd45733d302094:

                    /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
                github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
                    /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
                runtime.goexit
                    /usr/local/go/src/runtime/asm_amd64.s:1357
              - error with embedded safe details: output in %s
                -- arg 1: <string>
              - output in run_105821.001_n4_workload_run_tpcc:
              - error with attached stack trace:
                main.execCmd
                    /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:406
                main.(*cluster).RunL
                    /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2019
                main.(*cluster).RunE
                    /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2000
                main.runTPCCBench.func3.1
                    /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:815
                main.(*monitor).Go.func1
                    /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
                github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
                    /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
                runtime.goexit
                    /usr/local/go/src/runtime/asm_amd64.s:1357
              - error with embedded safe details: %s returned:
                stderr:
                %s
                stdout:
                %s
                -- arg 1: <string>
                -- arg 2: <string>
                -- arg 3: <string>
              - /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1749385-1581667748-22-n4cpu16:4 -- ./workload run tpcc --warehouses=2000 --active-warehouses=1670 --tolerate-errors --scatter --ramp=5m0s --duration=10m0s {pgurl:1-3} --histograms=perf/warehouses=1670/stats.json returned:
                stderr:
                 load: could not scatter ranges: Couldn't exec "ALTER TABLE warehouse SCATTER": dial tcp 10.128.0.39:26257: connect: connection refused
                I200214 11:21:35.377519 1 workload/cli/run.go:361  retrying after error while creating load: could not scatter ranges: Couldn't exec "ALTER TABLE warehouse SCATTER": dial tcp 10.128.0.39:26257: connect: connection refused
                I200214 11:21:35.524597 1 workload/cli/run.go:361  retrying after error while creating load: could not scatter ranges: Couldn't exec "ALTER TABLE warehouse SCATTER": dial tcp 10.128.0.39:26257: connect: connection refused
                I200214 11:21:35.636498 1 workload/cli/run.go:361  retrying after error while creating load: could not scatter ranges: Couldn't exec "ALTER TABLE warehouse SCATTER": dial tcp 10.128.0.39:26257: connect: connection refused
                I200214 11:21:35.902458 1 workload/cli/run.go:361  retrying after error while creating load: could not scatter ranges: Couldn't exec "ALTER TABLE warehouse SCATTER": dial tcp 10.128.0.39:26257: connect: connection refused

                stdout::
              - context canceled
            error running tpcc load generator
            main.runTPCCBench.func3.1
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tpcc.go:818
            main.(*monitor).Go.func1
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster.go:2344
            github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1
                /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:57
            runtime.goexit
                /usr/local/go/src/runtime/asm_amd64.s:1357
More

Artifacts: [/tpccbench/nodes=3/cpu=16](https://teamcity.cockroachdb.com/viewLog.html?buildId=1749385&tab=artifacts#/tpccbench/nodes=3/cpu=16) Related: - #44974 roachtest: tpccbench/nodes=3/cpu=16 failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-roachtest](https://api.github.com/repos/cockroachdb/cockroach/labels/O-roachtest) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-19.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-19.1) [release-blocker](https://api.github.com/repos/cockroachdb/cockroach/labels/release-blocker) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Atpccbench%2Fnodes%3D3%2Fcpu%3D16.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

irfansharif commented 4 years ago

restoring fixture: sql: expected 6 destination arguments in Scan, not 7

This particular failure was fixed in #45078.

cockroach-teamcity commented 4 years ago

(roachtest).tpccbench/nodes=3/cpu=16 failed on master@c473f40078994551cebcbe00fdbf1fa388957658:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20200309-1796240/tpccbench/nodes=3/cpu=16/run_1
    cluster.go:1410,context.go:135,cluster.go:1399,test_runner.go:778: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-1796240-1583738442-22-n4cpu16 --oneshot --ignore-empty-nodes: exit status 1 4: skipped
        3: dead
        2: 17592
        1: 17644
        Error:  3: dead
More

Artifacts: [/tpccbench/nodes=3/cpu=16](https://teamcity.cockroachdb.com/viewLog.html?buildId=1796240&tab=artifacts#/tpccbench/nodes=3/cpu=16) Related: - #44974 roachtest: tpccbench/nodes=3/cpu=16 failed [C-test-failure](https://api.github.com/repos/cockroachdb/cockroach/labels/C-test-failure) [O-roachtest](https://api.github.com/repos/cockroachdb/cockroach/labels/O-roachtest) [O-robot](https://api.github.com/repos/cockroachdb/cockroach/labels/O-robot) [branch-release-19.1](https://api.github.com/repos/cockroachdb/cockroach/labels/branch-release-19.1) [release-blocker](https://api.github.com/repos/cockroachdb/cockroach/labels/release-blocker) [See this test on roachdash](https://roachdash.crdb.dev/?filter=status%3Aopen+t%3A.%2Atpccbench%2Fnodes%3D3%2Fcpu%3D16.%2A&sort=title&restgroup=false&display=lastcommented+project) powered by [pkg/cmd/internal/issues](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/internal/issues)

nvanbenschoten commented 4 years ago

The last failure was 11 days ago. It was the following assertion failure:

F200309 10:38:58.828178 4596347 kv/kvserver/concurrency/concurrency_manager.go:295  [n3,s3,r5337/2:/Table/61/1/143{8/640…-9/680…}] caller violated contract
goroutine 4596347 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x744c101, 0xed5f813c2, 0x0, 0x773189)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0xb8
github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0x7448ea0, 0xc000000004, 0x69a9670, 0x2e, 0x127, 0xc0446f2200, 0x4b)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:211 +0xa0c
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x4b41220, 0xc026503170, 0x4, 0x2, 0x0, 0x0, 0xc037b60598, 0x1, 0x1)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x2c9
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x4b41220, 0xc026503170, 0x1, 0x4, 0x0, 0x0, 0xc037b60598, 0x1, 0x1)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:44 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatal(...)
    /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:164
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).OnLockUpdated(0xc0034abb80, 0x4b41220, 0xc026503170, 0xc01d929c30)
    /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:295 +0xe4

This was fixed 10 days ago by https://github.com/cockroachdb/cockroach/pull/45769.