search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.22k stars 3.82k forks source link

pkg/ccl/benchccl/rttanalysisccl/rttanalysisccl_test: TestBenchmarkExpectation failed #136392

Open cockroach-teamcity opened 5 days ago

cockroach-teamcity commented 5 days ago

pkg/ccl/benchccl/rttanalysisccl/rttanalysisccl_test.TestBenchmarkExpectation failed with artifacts on release-24.3 @ 232c636b5477e06c1b25b04b3a3c710fdf7a8b2e:

=== RUN   TestBenchmarkExpectation
    validate_benchmark_data.go:186: [/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/29114/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/benchccl/rttanalysisccl/rttanalysisccl_test_/rttanalysisccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/benchccl/rttanalysisccl/rttanalysisccl_test_/rttanalysisccl_test --test.v=true --test.bench=^$]
I241129 07:40:35.931896 1 (gostd) rand.go:245  [-] 1  random seed: -7660433783757549874
=== RUN   TestBenchmarkExpectation
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/7418df5abfb13dbfaf6c0beee25f240a/logTestBenchmarkExpectation1715013001
    test_log_scope.go:76: use -show-logs to present logs inline
    validate_benchmark_data.go:120: success: expected AlterRegions/alter_empty_database_add_region to perform KV lookups in [13, 13], got 13
    validate_benchmark_data.go:120: success: expected AlterTableLocality/alter_from_global_to_regional_by_table to perform KV lookups in [16, 16], got 15
    validate_benchmark_data.go:117: fail: expected AlterRegions/alter_empty_database_drop_region to perform KV lookups in [18, 18], got 20
    validate_benchmark_data.go:120: success: expected AlterSurvivalGoals/alter_empty_database_from_region_to_zone to perform KV lookups in [14, 14], got 14
    validate_benchmark_data.go:120: success: expected AlterTableLocality/alter_from_regional_by_table_to_global to perform KV lookups in [16, 16], got 15
    validate_benchmark_data.go:120: success: expected AlterSurvivalGoals/alter_populated_database_from_region_to_zone to perform KV lookups in [24, 24], got 24
    validate_benchmark_data.go:120: success: expected AlterTableLocality/alter_from_regional_by_table_to_rbr to perform KV lookups in [25, 25], got 24
    validate_benchmark_data.go:120: success: expected AlterRegions/alter_populated_database_add_region to perform KV lookups in [13, 13], got 13
    validate_benchmark_data.go:120: success: expected AlterPrimaryRegion/alter_empty_database_set_initial_primary_region to perform KV lookups in [16, 16], got 15
    validate_benchmark_data.go:120: success: expected AlterPrimaryRegion/alter_empty_database_alter_primary_region to perform KV lookups in [14, 14], got 14
    validate_benchmark_data.go:120: success: expected AlterPrimaryRegion/alter_populated_database_alter_primary_region to perform KV lookups in [14, 14], got 14
    validate_benchmark_data.go:120: success: expected AlterPrimaryRegion/alter_populated_database_set_initial_primary_region to perform KV lookups in [17, 17], got 16
    validate_benchmark_data.go:120: success: expected AlterTableLocality/alter_from_global_to_rbr to perform KV lookups in [25, 25], got 24
    validate_benchmark_data.go:120: success: expected AlterTableLocality/alter_from_rbr_to_global to perform KV lookups in [13, 13], got 12
    validate_benchmark_data.go:120: success: expected AlterTableLocality/alter_from_rbr_to_regional_by_table to perform KV lookups in [13, 13], got 12
    validate_benchmark_data.go:120: success: expected AlterRegions/alter_populated_database_drop_region to perform KV lookups in [18, 18], got 18
    validate_benchmark_data.go:120: success: expected AlterSurvivalGoals/alter_empty_database_from_zone_to_region to perform KV lookups in [14, 14], got 14
    validate_benchmark_data.go:120: success: expected AlterSurvivalGoals/alter_populated_database_from_zone_to_region to perform KV lookups in [24, 24], got 24
    validate_benchmark_data.go:73: see the -rewrite flag to re-run the benchmarks and adjust the expectations
    validate_benchmark_data.go:74: usage: ./dev test --rewrite pkg/bench/rttanalysis -f=TestBenchmarkExpectation/ --test-args '-rewrite-iterations=N'
    validate_benchmark_data.go:107: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/7418df5abfb13dbfaf6c0beee25f240a/logTestBenchmarkExpectation1715013001
--- FAIL: TestBenchmarkExpectation (50.16s)
--- FAIL: TestBenchmarkExpectation (50.75s)
=== RUN   TestBenchmarkExpectation/AlterRegions
    --- FAIL: TestBenchmarkExpectation/AlterRegions (42.39s)
=== RUN   TestBenchmarkExpectation/AlterRegions/alter_empty_database_drop_region
    rtt_analysis_bench.go:210: AlterRegions/alter_empty_database_drop_region: got 20.5, expected 18
    rtt_analysis_bench.go:216: wrote jaeger trace to /artifacts/tmp/_tmp/7418df5abfb13dbfaf6c0beee25f240a/logTestBenchmarkExpectation1715013001/AlterRegions_alter_empty_database_drop_region.jaeger.json
        --- FAIL: TestBenchmarkExpectation/AlterRegions/alter_empty_database_drop_region (18.19s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-45012

fqazi commented 21 hours ago

We just need to change the range from 18 to 20 for this test: validate_benchmark_data.go:117: fail: expected AlterRegions/alter_empty_database_drop_region to perform KV lookups in [18, 18], got 20