roachtest: follower-reads/mixed-version/single-region failed

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

https://www.cockroachlabs.com

Other

30.1k stars 3.81k forks source link

roachtest: follower-reads/mixed-version/single-region failed #129546

Closed cockroach-teamcity closed 1 week ago

cockroach-teamcity commented 2 months ago

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ c57e04527fbe285402bcadb7f73ce559e85d0c27:

(follower_reads.go:873).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 23 intervals > 4 threshold. Bad intervals:
interval 07:38:10-07:38:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.002 n4 ratio: 0.000 
interval 07:38:20-07:38:30: n1 ratio: 0.008 n2 ratio: 0.001 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:38:30-07:38:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:38:40-07:38:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:38:50-07:39:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:39:00-07:39:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:39:10-07:39:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.001 
interval 07:39:20-07:39:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:39:30-07:39:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:39:40-07:39:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.001 n4 ratio: 0.000 
interval 07:39:50-07:40:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:40:00-07:40:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:40:10-07:40:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:40:20-07:40:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:40:30-07:40:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:40:40-07:40:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.001 n4 ratio: 0.000 
interval 07:40:50-07:41:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.999 n4 ratio: 0.000 
interval 07:41:00-07:41:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:41:10-07:41:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:41:20-07:41:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:41:30-07:41:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:41:40-07:41:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 07:41:50-07:42:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
(mixedversion.go:710).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

Same failure on other branches

- #129224 roachtest: follower-reads/mixed-version/single-region failed [C-test-failure O-roachtest O-robot P-1 T-kv X-duplicate branch-release-23.2.10-rc]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-41588

arulajmani commented 2 months ago

This one seems to be because of shared process multi-tenancy. From the logs:

Deployment mode:    shared-process

Seems like we're using https://github.com/cockroachdb/cockroach/issues/129167 as the main issue for this class of failures, so I'll mark this as a duplicate. We're also not treating these shared process multi-tenancy failures as release blockers, so I'll remove the label.

cockroach-teamcity commented 2 months ago

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

(follower_reads.go:873).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 9 intervals > 4 threshold. Bad intervals:
interval 09:25:10-09:25:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.005 n4 ratio: 0.000 
interval 09:25:20-09:25:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.001 n4 ratio: 0.000 
interval 09:25:30-09:25:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 09:25:40-09:25:50: n1 ratio: 0.002 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 09:25:50-09:26:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 09:26:00-09:26:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 09:26:10-09:26:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 09:26:20-09:26:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.000 
interval 09:26:30-09:26:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 1.000 n4 ratio: 0.234 
(mixedversion.go:710).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 1 month ago

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ 67e99ebec74c1f6a6dfbf1cc0bca2d255a55f867:

(follower_reads.go:873).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 14 intervals > 4 threshold. Bad intervals:
interval 07:12:00-07:12:10: n1 ratio: 0.825 n2 ratio: 0.201 n3 ratio: 1.774 n4 ratio: 0.000 
interval 07:12:10-07:12:20: n1 ratio: 0.805 n2 ratio: 0.000 n3 ratio: 1.874 n4 ratio: 0.000 
interval 07:12:20-07:12:30: n1 ratio: 0.811 n2 ratio: 0.000 n3 ratio: 1.971 n4 ratio: 0.000 
interval 07:12:30-07:12:40: n1 ratio: 0.807 n2 ratio: 0.000 n3 ratio: 1.976 n4 ratio: 0.000 
interval 07:12:40-07:12:50: n1 ratio: 0.830 n2 ratio: 0.000 n3 ratio: 1.866 n4 ratio: 0.000 
interval 07:12:50-07:13:00: n1 ratio: 0.833 n2 ratio: 0.000 n3 ratio: 1.876 n4 ratio: 0.000 
interval 07:13:00-07:13:10: n1 ratio: 0.886 n2 ratio: 0.000 n3 ratio: 1.835 n4 ratio: 0.000 
interval 07:13:10-07:13:20: n1 ratio: 0.726 n2 ratio: 0.000 n3 ratio: 1.875 n4 ratio: 0.000 
interval 07:13:20-07:13:30: n1 ratio: 0.534 n2 ratio: 0.000 n3 ratio: 2.413 n4 ratio: 0.000 
interval 07:13:30-07:13:40: n1 ratio: 0.553 n2 ratio: 0.000 n3 ratio: 2.195 n4 ratio: 0.000 
interval 07:14:00-07:14:10: n1 ratio: 0.449 n2 ratio: 0.000 n3 ratio: 2.485 n4 ratio: 0.000 
interval 07:14:10-07:14:20: n1 ratio: 0.895 n2 ratio: 0.000 n3 ratio: 1.999 n4 ratio: 0.000 
interval 07:14:20-07:14:30: n1 ratio: 0.606 n2 ratio: 0.000 n3 ratio: 2.414 n4 ratio: 0.000 
interval 07:14:30-07:14:40: n1 ratio: 0.629 n2 ratio: 0.000 n3 ratio: 2.088 n4 ratio: 0.000 
(mixedversion.go:710).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 1 month ago

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ 197c6ee5537ffb211ebd8dbcbe49edc6d5c710e1:

(follower_reads.go:873).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 23 intervals > 4 threshold. Bad intervals:
interval 07:48:20-07:48:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.698 n4 ratio: 0.011 
interval 07:48:30-07:48:40: n1 ratio: 0.000 n2 ratio: 0.001 n3 ratio: 2.747 n4 ratio: 0.000 
interval 07:48:40-07:48:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.772 n4 ratio: 0.000 
interval 07:48:50-07:49:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.795 n4 ratio: 0.000 
interval 07:49:00-07:49:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.772 n4 ratio: 0.000 
interval 07:49:10-07:49:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.761 n4 ratio: 0.000 
interval 07:49:20-07:49:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.667 n4 ratio: 0.000 
interval 07:49:30-07:49:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.664 n4 ratio: 0.000 
interval 07:49:40-07:49:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.826 n4 ratio: 0.000 
interval 07:49:50-07:50:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.853 n4 ratio: 0.000 
interval 07:50:00-07:50:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.835 n4 ratio: 0.000 
interval 07:50:10-07:50:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.813 n4 ratio: 0.000 
interval 07:50:20-07:50:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.916 n4 ratio: 0.000 
interval 07:50:30-07:50:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 3.042 n4 ratio: 0.000 
interval 07:50:40-07:50:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 3.061 n4 ratio: 0.000 
interval 07:50:50-07:51:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.869 n4 ratio: 0.000 
interval 07:51:00-07:51:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.764 n4 ratio: 0.000 
interval 07:51:10-07:51:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.856 n4 ratio: 0.000 
interval 07:51:20-07:51:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.794 n4 ratio: 0.000 
interval 07:51:30-07:51:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.737 n4 ratio: 0.000 
interval 07:51:40-07:51:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.986 n4 ratio: 0.000 
interval 07:51:50-07:52:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.642 n4 ratio: 0.000 
interval 07:52:00-07:52:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 2.627 n4 ratio: 0.000 
(mixedversion.go:720).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 1 month ago

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ 83589fb87caa92fb42e83994f1691978f37e4cbb:

(follower_reads.go:873).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 21 intervals > 4 threshold. Bad intervals:
interval 09:56:00-09:56:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 3.609 
interval 09:56:10-09:56:20: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.969 
interval 09:56:20-09:56:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.976 
interval 09:56:30-09:56:40: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.964 
interval 09:56:40-09:56:50: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.206 
interval 09:56:50-09:57:00: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.416 
interval 09:57:00-09:57:10: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.685 
interval 09:57:10-09:57:20: n1 ratio: 0.000 n2 ratio: 0.005 n3 ratio: 0.000 n4 ratio: 2.695 
interval 09:57:20-09:57:30: n1 ratio: 0.000 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.062 
interval 09:57:30-09:57:40: n1 ratio: 0.671 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.442 
interval 09:57:40-09:57:50: n1 ratio: 0.266 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.966 
interval 09:57:50-09:58:00: n1 ratio: 0.336 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.713 
interval 09:58:00-09:58:10: n1 ratio: 0.858 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.618 
interval 09:58:10-09:58:20: n1 ratio: 0.511 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.352 
interval 09:58:20-09:58:30: n1 ratio: 0.677 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 1.996 
interval 09:58:30-09:58:40: n1 ratio: 0.576 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.360 
interval 09:58:50-09:59:00: n1 ratio: 0.716 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.218 
interval 09:59:00-09:59:10: n1 ratio: 0.794 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.013 
interval 09:59:10-09:59:20: n1 ratio: 0.061 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.165 
interval 09:59:30-09:59:40: n1 ratio: 0.897 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.047 
interval 09:59:40-09:59:50: n1 ratio: 0.535 n2 ratio: 0.000 n3 ratio: 0.000 n4 ratio: 2.029 
(mixedversion.go:720).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 1 month ago

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ 8e6e4090457565a41bc3bd8ea954e437030d1c49:

(follower_reads.go:873).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 23 intervals > 4 threshold. Bad intervals:
interval 09:04:10-09:04:20: n1 ratio: 0.007 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:04:20-09:04:30: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:04:30-09:04:40: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:04:40-09:04:50: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:04:50-09:05:00: n1 ratio: 0.000 n2 ratio: 1.411 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:05:00-09:05:10: n1 ratio: 0.000 n2 ratio: 1.980 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:05:10-09:05:20: n1 ratio: 0.000 n2 ratio: 1.980 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:05:20-09:05:30: n1 ratio: 0.000 n2 ratio: 1.803 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:05:30-09:05:40: n1 ratio: 0.004 n2 ratio: 1.195 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:05:40-09:05:50: n1 ratio: 0.000 n2 ratio: 1.693 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:05:50-09:06:00: n1 ratio: 0.000 n2 ratio: 1.810 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:06:00-09:06:10: n1 ratio: 0.000 n2 ratio: 1.981 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:06:10-09:06:20: n1 ratio: 0.000 n2 ratio: 1.203 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:06:20-09:06:30: n1 ratio: 0.000 n2 ratio: 1.212 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:06:30-09:06:40: n1 ratio: 0.000 n2 ratio: 1.814 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:06:40-09:06:50: n1 ratio: 0.000 n2 ratio: 1.062 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:06:50-09:07:00: n1 ratio: 0.000 n2 ratio: 1.593 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:07:00-09:07:10: n1 ratio: 0.000 n2 ratio: 1.468 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:07:10-09:07:20: n1 ratio: 0.000 n2 ratio: 1.182 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:07:20-09:07:30: n1 ratio: 0.000 n2 ratio: 1.584 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:07:30-09:07:40: n1 ratio: 0.000 n2 ratio: 1.731 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:07:40-09:07:50: n1 ratio: 0.000 n2 ratio: 1.978 n3 ratio: 0.000 n4 ratio: 0.000 
interval 09:07:50-09:08:00: n1 ratio: 0.000 n2 ratio: 1.979 n3 ratio: 0.000 n4 ratio: 0.000 
(mixedversion.go:720).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 1 month ago

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ ec2573dc6aaeefc226440bb2c5a7c94a63989868:

(follower_reads.go:379).runFollowerReadsTest: fewer than 2 follower reads occurred: saw [63 0 6 1] before and [63 640 6 1] after
(mixedversion.go:737).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 1 month ago

roachtest.follower-reads/mixed-version/single-region failed with artifacts on master @ 4de315c9ca4ccf7c3bdbf53a5226e8c14c84a68e:

(follower_reads.go:870).verifyHighFollowerReadRatios: too many intervals with more than 2 nodes with low follower read ratios: 23 intervals > 4 threshold. Bad intervals:
interval 10:19:40-10:19:50: n1 ratio: 0.000 n2 ratio: 0.999 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:19:50-10:20:00: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:20:00-10:20:10: n1 ratio: 0.000 n2 ratio: 1.012 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:20:10-10:20:20: n1 ratio: 0.000 n2 ratio: 1.001 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:20:20-10:20:30: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:20:30-10:20:40: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:20:40-10:20:50: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.005 
interval 10:20:50-10:21:00: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:21:00-10:21:10: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:21:10-10:21:20: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:21:20-10:21:30: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:21:30-10:21:40: n1 ratio: 0.000 n2 ratio: 0.999 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:21:40-10:21:50: n1 ratio: 0.000 n2 ratio: 1.001 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:21:50-10:22:00: n1 ratio: 0.000 n2 ratio: 0.999 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:22:00-10:22:10: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:22:10-10:22:20: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:22:20-10:22:30: n1 ratio: 0.000 n2 ratio: 1.001 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:22:30-10:22:40: n1 ratio: 0.000 n2 ratio: 0.999 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:22:40-10:22:50: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:22:50-10:23:00: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:23:00-10:23:10: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:23:10-10:23:20: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
interval 10:23:20-10:23:30: n1 ratio: 0.000 n2 ratio: 1.000 n3 ratio: 0.000 n4 ratio: 0.000 
(mixedversion.go:732).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

blathers-crl[bot] commented 1 week ago

Based on the specified backports for linked PR #133092, I applied the following new label(s) to this issue: branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}