search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.18k stars 3.82k forks source link

bench/rttanalysis: TestBenchmarkExpectation failed [Jobs/jobs_page_type_filtered_no_matches] #126750

Open cockroach-teamcity opened 4 months ago

cockroach-teamcity commented 4 months ago

bench/rttanalysis.TestBenchmarkExpectation failed with artifacts on release-24.1 @ 0f7c21d66ae913568060598dffe880b279283ba1:

    validate_benchmark_data.go:123: success: expected AlterTableSplit/alter_table_split_at_2_values to perform KV lookups in [11, 11], got 11
    validate_benchmark_data.go:123: success: expected ORMQueries/pg_class to perform KV lookups in [4, 4], got 4
    validate_benchmark_data.go:123: success: expected ORMQueries/activerecord_type_introspection_query to perform KV lookups in [3, 3], got 3
    validate_benchmark_data.go:123: success: expected ORMQueries/has_sequence_privilege to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:123: success: expected ORMQueries/has_column_privilege_using_column_name to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:123: success: expected ORMQueries/pg_is_other_temp_schema_multiple_times to perform KV lookups in [6, 6], got 6
    validate_benchmark_data.go:123: success: expected ORMQueries/pg_attribute to perform KV lookups in [4, 4], got 4
    validate_benchmark_data.go:123: success: expected ORMQueries/hasura_column_descriptions to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected ORMQueries/introspection_description_join to perform KV lookups in [4, 4], got 5
    validate_benchmark_data.go:123: success: expected ORMQueries/django_column_introspection_8_tables to perform KV lookups in [4, 4], got 4
    validate_benchmark_data.go:123: success: expected GenerateObjects/generate_50000_tables to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected ORMQueries/hasura_column_descriptions_modified to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected Grant/grant_all_on_1_table to perform KV lookups in [15, 15], got 15
    validate_benchmark_data.go:123: success: expected ShowGrants/grant_2_roles to perform KV lookups in [14, 14], got 14
    validate_benchmark_data.go:123: success: expected ORMQueries/has_table_privilege to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:123: success: expected DropTable/drop_2_tables to perform KV lookups in [17, 17], got 17
    validate_benchmark_data.go:123: success: expected ORMQueries/pg_is_other_temp_schema to perform KV lookups in [6, 6], got 6
    validate_benchmark_data.go:123: success: expected AlterTableConfigureZone/alter_table_configure_zone_7_replicas_ to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected ORMQueries/npgsql_types to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected AlterTableAddForeignKey/alter_table_add_1_foreign_key to perform KV lookups in [17, 17], got 17
    validate_benchmark_data.go:123: success: expected GenerateObjects/generate_10_tables to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected ORMQueries/pg_my_temp_schema to perform KV lookups in [3, 3], got 3
    validate_benchmark_data.go:123: success: expected DropDatabase/drop_database_1_table to perform KV lookups in [18, 18], got 18
    validate_benchmark_data.go:123: success: expected AlterTableAddForeignKey/alter_table_add_3_foreign_keys to perform KV lookups in [21, 21], got 21
    validate_benchmark_data.go:123: success: expected DropView/drop_3_views to perform KV lookups in [17, 17], got 17
    validate_benchmark_data.go:123: success: expected ShowGrants/grant_3_roles to perform KV lookups in [16, 16], got 16
    validate_benchmark_data.go:123: success: expected DropView/drop_1_view to perform KV lookups in [16, 16], got 16
    validate_benchmark_data.go:123: success: expected DropView/drop_2_views to perform KV lookups in [17, 17], got 17
    validate_benchmark_data.go:123: success: expected AlterRole/alter_role_with_2_options to perform KV lookups in [21, 21], got 21
    validate_benchmark_data.go:123: success: expected AlterRole/alter_role_with_3_options to perform KV lookups in [29, 29], got 29
    validate_benchmark_data.go:123: success: expected AlterRole/alter_role_with_1_option to perform KV lookups in [18, 18], got 18
    validate_benchmark_data.go:123: success: expected ORMQueries/django_table_introspection_8_tables to perform KV lookups in [6, 6], got 5
    validate_benchmark_data.go:123: success: expected GrantRole/grant_2_roles to perform KV lookups in [25, 25], got 25
    validate_benchmark_data.go:123: success: expected GrantRole/grant_1_role to perform KV lookups in [19, 19], got 19
    validate_benchmark_data.go:123: success: expected ORMQueries/column_descriptions_json_agg to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:123: success: expected ORMQueries/liquibase_migrations_on_multiple_dbs to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected ORMQueries/prisma_column_descriptions to perform KV lookups in [133, 133], got 133
    validate_benchmark_data.go:123: success: expected ORMQueries/asyncpg_types to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:76: see the -rewrite flag to re-run the benchmarks and adjust the expectations
    validate_benchmark_data.go:77: usage: ./dev test --rewrite pkg/bench/rttanalysis -f=TestBenchmarkExpectation/ --test-args '-rewrite-iterations=N'
    validate_benchmark_data.go:110: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/b03b0378e25aebc8fcd368039859e7a0/logTestBenchmarkExpectation582767021
--- FAIL: TestBenchmarkExpectation (269.42s)
--- FAIL: TestBenchmarkExpectation (270.39s)
=== RUN   TestBenchmarkExpectation/Jobs
    --- FAIL: TestBenchmarkExpectation/Jobs (205.47s)
=== RUN   TestBenchmarkExpectation/Jobs/jobs_page_type_filtered_no_matches
    rtt_analysis_bench.go:215: Jobs/jobs_page_type_filtered_no_matches: got 5.5, expected 6-9
    rtt_analysis_bench.go:221: wrote jaeger trace to /artifacts/tmp/_tmp/b03b0378e25aebc8fcd368039859e7a0/logTestBenchmarkExpectation582767021/Jobs_jobs_page_type_filtered_no_matches.jaeger.json
        --- FAIL: TestBenchmarkExpectation/Jobs/jobs_page_type_filtered_no_matches (111.97s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #126696 bench/rttanalysis: TestBenchmarkExpectation failed [C-test-failure O-robot T-sql-foundations branch-master release-blocker]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-40106

rafiss commented 4 months ago 
    validate_benchmark_data.go:120: fail: expected Jobs/jobs_page_type_filtered_no_matches to perform KV lookups in [6, 9], got 5
cockroach-teamcity commented 4 months ago

bench/rttanalysis.TestBenchmarkExpectation failed with artifacts on release-24.1 @ 37e0dbc034aabc9516694a89966c8df4171ae3a4:

    validate_benchmark_data.go:123: success: expected ShowGrants/grant_3_roles to perform KV lookups in [16, 16], got 16
    validate_benchmark_data.go:123: success: expected DropDatabase/drop_database_1_table to perform KV lookups in [18, 18], got 18
    validate_benchmark_data.go:123: success: expected Truncate/truncate_2_column_0_rows to perform KV lookups in [20, 20], got 19
    validate_benchmark_data.go:123: success: expected AlterTableUnsplit/alter_table_unsplit_at_3_values to perform KV lookups in [11, 11], got 11
    validate_benchmark_data.go:123: success: expected AlterTableConfigureZone/alter_table_configure_zone_5_replicas to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected DropDatabase/drop_database_2_tables to perform KV lookups in [18, 18], got 18
    validate_benchmark_data.go:123: success: expected ORMQueries/npgsql_fields to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected AlterTableAddForeignKey/alter_table_add_3_foreign_keys to perform KV lookups in [21, 21], got 21
    validate_benchmark_data.go:123: success: expected Truncate/truncate_1_column_0_rows to perform KV lookups in [20, 20], got 19
    validate_benchmark_data.go:123: success: expected ORMQueries/prisma_column_descriptions_updated to perform KV lookups in [3, 3], got 4
    validate_benchmark_data.go:123: success: expected AlterTableSplit/alter_table_split_at_1_value to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected Truncate/truncate_1_column_2_rows to perform KV lookups in [20, 20], got 19
    validate_benchmark_data.go:123: success: expected AlterTableConfigureZone/alter_table_configure_zone_7_replicas_ to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected ORMQueries/has_column_privilege_using_column_name to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:123: success: expected GenerateObjects/generate_100_tables_from_template to perform KV lookups in [11, 11], got 12
    validate_benchmark_data.go:123: success: expected ORMQueries/liquibase_migrations to perform KV lookups in [4, 4], got 4
    validate_benchmark_data.go:123: success: expected AlterTableDropConstraint/alter_table_drop_1_check_constraint to perform KV lookups in [15, 15], got 15
    validate_benchmark_data.go:123: success: expected AlterTableDropConstraint/alter_table_drop_2_check_constraints to perform KV lookups in [15, 15], got 15
    validate_benchmark_data.go:123: success: expected AlterTableConfigureZone/alter_table_configure_zone_ranges to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected AlterTableAddForeignKey/alter_table_add_2_foreign_keys to perform KV lookups in [19, 19], got 19
    validate_benchmark_data.go:123: success: expected DropRole/drop_2_roles to perform KV lookups in [43, 43], got 43
    validate_benchmark_data.go:123: success: expected AlterTableAddForeignKey/alter_table_add_1_foreign_key to perform KV lookups in [17, 17], got 17
    validate_benchmark_data.go:123: success: expected Jobs/show_job to perform KV lookups in [3, 3], got 3
    validate_benchmark_data.go:123: success: expected Jobs/cancel_job to perform KV lookups in [6, 6], got 6
    validate_benchmark_data.go:123: success: expected Jobs/pause_job to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected Jobs/resume_job to perform KV lookups in [6, 6], got 6
    validate_benchmark_data.go:116: no expectation for benchmark Jobs/non_admin_crdb_internal.system_jobs, got 9
    validate_benchmark_data.go:123: success: expected Jobs/show_jobs to perform KV lookups in [6, 9], got 8
    validate_benchmark_data.go:123: success: expected Jobs/non_admin_show_jobs to perform KV lookups in [6, 9], got 9
    validate_benchmark_data.go:123: success: expected Jobs/jobs_page_type_filtered to perform KV lookups in [6, 9], got 8
    validate_benchmark_data.go:123: success: expected Jobs/jobs_page_latest_50 to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected Jobs/jobs_page_type_filtered_no_matches to perform KV lookups in [6, 9], got 8
    validate_benchmark_data.go:123: success: expected Jobs/crdb_internal.system_jobs to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected ORMQueries/column_descriptions_json_agg to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:120: fail: expected Jobs/jobs_page_default to perform KV lookups in [8, 8], got 5
    validate_benchmark_data.go:123: success: expected ORMQueries/liquibase_migrations_on_multiple_dbs to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected ORMQueries/prisma_column_descriptions to perform KV lookups in [133, 133], got 133
    validate_benchmark_data.go:123: success: expected ORMQueries/asyncpg_types to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:76: see the -rewrite flag to re-run the benchmarks and adjust the expectations
    validate_benchmark_data.go:77: usage: ./dev test --rewrite pkg/bench/rttanalysis -f=TestBenchmarkExpectation/ --test-args '-rewrite-iterations=N'
    validate_benchmark_data.go:110: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/b03b0378e25aebc8fcd368039859e7a0/logTestBenchmarkExpectation480205776
--- FAIL: TestBenchmarkExpectation (256.59s)
--- FAIL: TestBenchmarkExpectation (257.13s)
=== RUN   TestBenchmarkExpectation/Jobs/jobs_page_default
    rtt_analysis_bench.go:215: Jobs/jobs_page_default: got 5.5, expected 8
    rtt_analysis_bench.go:221: wrote jaeger trace to /artifacts/tmp/_tmp/b03b0378e25aebc8fcd368039859e7a0/logTestBenchmarkExpectation480205776/Jobs_jobs_page_default.jaeger.json
        --- FAIL: TestBenchmarkExpectation/Jobs/jobs_page_default (188.37s)
=== RUN   TestBenchmarkExpectation/Jobs
    --- FAIL: TestBenchmarkExpectation/Jobs (188.37s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #126696 bench/rttanalysis: TestBenchmarkExpectation failed [Jobs/jobs_page_default] [A-jobs C-test-failure O-robot T-jobs branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

bench/rttanalysis.TestBenchmarkExpectation failed with artifacts on release-24.1 @ 5cf8d7b63341de8d8b234ebaf1fa6fe6e9258582:

    validate_benchmark_data.go:123: success: expected ORMQueries/django_comment_introspection_with_comments to perform KV lookups in [6, 6], got 5
    validate_benchmark_data.go:123: success: expected ORMQueries/pg_class to perform KV lookups in [4, 4], got 4
    validate_benchmark_data.go:123: success: expected ORMQueries/django_table_introspection_8_tables to perform KV lookups in [6, 6], got 5
    validate_benchmark_data.go:123: success: expected ORMQueries/introspection_description_join to perform KV lookups in [4, 4], got 5
    validate_benchmark_data.go:123: success: expected ORMQueries/pg_attribute to perform KV lookups in [4, 4], got 4
    validate_benchmark_data.go:123: success: expected ORMQueries/pg_namespace to perform KV lookups in [3, 3], got 3
    validate_benchmark_data.go:123: success: expected ORMQueries/hasura_column_descriptions to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected ORMQueries/hasura_column_descriptions_modified to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected Audit/select_from_an_audit_table to perform KV lookups in [5, 5], got 6
    validate_benchmark_data.go:123: success: expected ORMQueries/has_schema_privilege to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:123: success: expected ORMQueries/has_sequence_privilege to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:123: success: expected SystemDatabaseQueries/select_system.users_with_empty_database_Name to perform KV lookups in [1, 1], got 1
    validate_benchmark_data.go:123: success: expected SystemDatabaseQueries/select_system.users_with_schema_Name to perform KV lookups in [1, 1], got 1
    validate_benchmark_data.go:123: success: expected DropSequence/drop_3_sequences to perform KV lookups in [19, 19], got 19
    validate_benchmark_data.go:123: success: expected Jobs/jobs_page_latest_50 to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected DropSequence/drop_1_sequence to perform KV lookups in [15, 15], got 15
    validate_benchmark_data.go:123: success: expected ORMQueries/pg_my_temp_schema_multiple_times to perform KV lookups in [3, 3], got 3
    validate_benchmark_data.go:123: success: expected ORMQueries/pg_my_temp_schema to perform KV lookups in [3, 3], got 3
    validate_benchmark_data.go:123: success: expected ORMQueries/has_column_privilege_using_attnum to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:123: success: expected DropSequence/drop_2_sequences to perform KV lookups in [17, 17], got 17
    validate_benchmark_data.go:123: success: expected ORMQueries/has_column_privilege_using_column_name to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:123: success: expected ORMQueries/npgsql_types to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected ORMQueries/pg_is_other_temp_schema_multiple_times to perform KV lookups in [6, 6], got 6
    validate_benchmark_data.go:123: success: expected VirtualTableQueries/virtual_table_cache_with_schema_change to perform KV lookups in [15, 15], got 14
    validate_benchmark_data.go:123: success: expected ORMQueries/has_table_privilege to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:123: success: expected ORMQueries/npgsql_fields to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected SystemDatabaseQueries/select_system.users_without_schema_Name to perform KV lookups in [1, 1], got 1
    validate_benchmark_data.go:123: success: expected ORMQueries/pg_is_other_temp_schema to perform KV lookups in [6, 6], got 6
    validate_benchmark_data.go:123: success: expected GenerateObjects/generate_10x10_schemas_and_tables_in_existing_db to perform KV lookups in [16, 16], got 16
    validate_benchmark_data.go:123: success: expected AlterRole/alter_role_with_1_option to perform KV lookups in [18, 18], got 18
    validate_benchmark_data.go:123: success: expected UDFResolution/select_from_udf to perform KV lookups in [3, 3], got 3
    validate_benchmark_data.go:123: success: expected ORMQueries/hasura_column_descriptions_8_tables to perform KV lookups in [12, 12], got 12
    validate_benchmark_data.go:123: success: expected ORMQueries/prisma_column_descriptions_updated to perform KV lookups in [3, 3], got 4
    validate_benchmark_data.go:123: success: expected ORMQueries/information_schema._pg_index_position to perform KV lookups in [4, 4], got 4
    validate_benchmark_data.go:123: success: expected ORMQueries/liquibase_migrations to perform KV lookups in [4, 4], got 4
    validate_benchmark_data.go:123: success: expected ORMQueries/column_descriptions_json_agg to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:123: success: expected ORMQueries/liquibase_migrations_on_multiple_dbs to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected ORMQueries/prisma_column_descriptions to perform KV lookups in [133, 133], got 133
    validate_benchmark_data.go:123: success: expected ORMQueries/asyncpg_types to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:76: see the -rewrite flag to re-run the benchmarks and adjust the expectations
    validate_benchmark_data.go:77: usage: ./dev test --rewrite pkg/bench/rttanalysis -f=TestBenchmarkExpectation/ --test-args '-rewrite-iterations=N'
    validate_benchmark_data.go:110: -- test log scope end --
--- FAIL: TestBenchmarkExpectation (307.66s)
--- FAIL: TestBenchmarkExpectation (308.86s)
=== RUN   TestBenchmarkExpectation/Jobs
    --- FAIL: TestBenchmarkExpectation/Jobs (182.97s)
=== RUN   TestBenchmarkExpectation/Jobs/jobs_page_default
    rtt_analysis_bench.go:215: Jobs/jobs_page_default: got 5.5, expected 8
    rtt_analysis_bench.go:221: wrote jaeger trace to /artifacts/tmp/_tmp/b03b0378e25aebc8fcd368039859e7a0/logTestBenchmarkExpectation3147918048/Jobs_jobs_page_default.jaeger.json
        --- FAIL: TestBenchmarkExpectation/Jobs/jobs_page_default (108.18s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #127191 bench/rttanalysis: TestBenchmarkExpectation failed [C-test-failure O-robot T-sql-foundations branch-release-24.2 release-blocker] - #126696 bench/rttanalysis: TestBenchmarkExpectation failed [Jobs/jobs_page_default] [A-jobs C-test-failure O-robot T-jobs branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

bench/rttanalysis.TestBenchmarkExpectation failed with artifacts on release-24.1 @ 984bb6088542c5ef9b9ce8c44af6d980bd7db6f0:

    validate_benchmark_data.go:123: success: expected AlterTableAddForeignKey/alter_table_add_2_foreign_keys to perform KV lookups in [19, 19], got 19
    validate_benchmark_data.go:123: success: expected AlterTableAddForeignKey/alter_table_add_3_foreign_keys to perform KV lookups in [21, 21], got 21
    validate_benchmark_data.go:123: success: expected Jobs/jobs_page_type_filtered to perform KV lookups in [6, 9], got 8
    validate_benchmark_data.go:123: success: expected GenerateObjects/generate_1000_tables_-_this_test_should_use_the_same_number_of_RTTs_as_for_10_tables to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected ORMQueries/pg_my_temp_schema_multiple_times to perform KV lookups in [3, 3], got 3
    validate_benchmark_data.go:123: success: expected ShowGrants/grant_2_roles to perform KV lookups in [14, 14], got 14
    validate_benchmark_data.go:123: success: expected ShowGrants/grant_4_roles to perform KV lookups in [18, 18], got 18
    validate_benchmark_data.go:123: success: expected ORMQueries/hasura_column_descriptions_8_tables to perform KV lookups in [12, 12], got 12
    validate_benchmark_data.go:123: success: expected ShowGrants/grant_3_roles to perform KV lookups in [16, 16], got 16
    validate_benchmark_data.go:123: success: expected GenerateObjects/generate_50000_tables to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected GenerateObjects/generate_100_tables_from_template to perform KV lookups in [11, 11], got 12
    validate_benchmark_data.go:123: success: expected Truncate/truncate_1_column_0_rows to perform KV lookups in [20, 20], got 19
    validate_benchmark_data.go:123: success: expected ORMQueries/npgsql_types to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected ORMQueries/npgsql_fields to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected Truncate/truncate_1_column_1_row to perform KV lookups in [20, 20], got 19
    validate_benchmark_data.go:123: success: expected RevokeRole/revoke_2_roles to perform KV lookups in [21, 21], got 21
    validate_benchmark_data.go:123: success: expected RevokeRole/revoke_1_role to perform KV lookups in [17, 17], got 17
    validate_benchmark_data.go:123: success: expected Truncate/truncate_1_column_2_rows to perform KV lookups in [20, 20], got 19
    validate_benchmark_data.go:123: success: expected Truncate/truncate_2_column_0_rows to perform KV lookups in [20, 20], got 19
    validate_benchmark_data.go:123: success: expected Truncate/truncate_2_column_1_rows to perform KV lookups in [20, 20], got 19
    validate_benchmark_data.go:123: success: expected ORMQueries/pg_namespace to perform KV lookups in [3, 3], got 3
    validate_benchmark_data.go:123: success: expected Jobs/crdb_internal.system_jobs to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected DropTable/drop_1_table to perform KV lookups in [15, 15], got 15
    validate_benchmark_data.go:123: success: expected DropTable/drop_2_tables to perform KV lookups in [17, 17], got 17
    validate_benchmark_data.go:123: success: expected AlterTableDropConstraint/alter_table_drop_3_check_constraints to perform KV lookups in [15, 15], got 15
    validate_benchmark_data.go:123: success: expected DropTable/drop_3_tables to perform KV lookups in [19, 19], got 19
    validate_benchmark_data.go:123: success: expected DropRole/drop_1_role to perform KV lookups in [34, 34], got 34
    validate_benchmark_data.go:123: success: expected DropRole/drop_2_roles to perform KV lookups in [43, 43], got 43
    validate_benchmark_data.go:123: success: expected ORMQueries/liquibase_migrations to perform KV lookups in [4, 4], got 4
    validate_benchmark_data.go:123: success: expected ORMQueries/liquibase_migrations_on_multiple_dbs to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected GenerateObjects/generate_10x10_schemas_and_tables_in_existing_db to perform KV lookups in [16, 16], got 16
    validate_benchmark_data.go:123: success: expected ORMQueries/prisma_column_descriptions to perform KV lookups in [133, 133], got 133
    validate_benchmark_data.go:123: success: expected ORMQueries/asyncpg_types to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:76: see the -rewrite flag to re-run the benchmarks and adjust the expectations
    validate_benchmark_data.go:77: usage: ./dev test --rewrite pkg/bench/rttanalysis -f=TestBenchmarkExpectation/ --test-args '-rewrite-iterations=N'
    validate_benchmark_data.go:110: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/b03b0378e25aebc8fcd368039859e7a0/logTestBenchmarkExpectation2182594844
--- FAIL: TestBenchmarkExpectation (262.95s)
=== RUN   TestBenchmarkExpectation/Jobs
    --- FAIL: TestBenchmarkExpectation/Jobs (208.50s)
=== RUN   TestBenchmarkExpectation/Jobs/jobs_page_default
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    rtt_analysis_bench.go:215: Jobs/jobs_page_default: got 5.5, expected 8
    rtt_analysis_bench.go:221: wrote jaeger trace to /artifacts/tmp/_tmp/b03b0378e25aebc8fcd368039859e7a0/logTestBenchmarkExpectation2182594844/Jobs_jobs_page_default.jaeger.json
        --- FAIL: TestBenchmarkExpectation/Jobs/jobs_page_default (97.26s)
=== RUN   TestBenchmarkExpectation/Jobs/jobs_page_latest_50
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    rtt_analysis_bench.go:215: Jobs/jobs_page_latest_50: got 5.5, expected 8
    rtt_analysis_bench.go:221: wrote jaeger trace to /artifacts/tmp/_tmp/b03b0378e25aebc8fcd368039859e7a0/logTestBenchmarkExpectation2182594844/Jobs_jobs_page_latest_50.jaeger.json
        --- FAIL: TestBenchmarkExpectation/Jobs/jobs_page_latest_50 (184.69s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #127191 bench/rttanalysis: TestBenchmarkExpectation failed [C-test-failure O-robot P-1 T-sql-foundations branch-release-24.2] - #126696 bench/rttanalysis: TestBenchmarkExpectation failed [Jobs/jobs_page_default] [A-jobs C-test-failure O-robot T-jobs branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

bench/rttanalysis.TestBenchmarkExpectation failed with artifacts on release-24.1 @ a9e48ad3283bf466f659c4f30e541eb5215a2af3:

    validate_benchmark_data.go:123: success: expected AlterTableAddForeignKey/alter_table_add_2_foreign_keys to perform KV lookups in [19, 19], got 19
    validate_benchmark_data.go:123: success: expected AlterRole/alter_role_with_1_option to perform KV lookups in [18, 18], got 18
    validate_benchmark_data.go:123: success: expected ORMQueries/hasura_column_descriptions_modified to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected AlterTableSplit/alter_table_split_at_1_value to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected AlterTableAddForeignKey/alter_table_add_3_foreign_keys to perform KV lookups in [21, 21], got 21
    validate_benchmark_data.go:123: success: expected ORMQueries/information_schema._pg_index_position to perform KV lookups in [4, 4], got 4
    validate_benchmark_data.go:123: success: expected AlterTableSplit/alter_table_split_at_2_values to perform KV lookups in [11, 11], got 11
    validate_benchmark_data.go:123: success: expected Discard/DISCARD_ALL,_no_tables to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:123: success: expected AlterTableAddCheckConstraint/alter_table_add_1_check_constraint to perform KV lookups in [15, 15], got 15
    validate_benchmark_data.go:123: success: expected ORMQueries/hasura_column_descriptions_8_tables to perform KV lookups in [12, 12], got 12
    validate_benchmark_data.go:123: success: expected GenerateObjects/generate_1000_tables_-_this_test_should_use_the_same_number_of_RTTs_as_for_10_tables to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected AlterTableAddCheckConstraint/alter_table_add_3_check_constraints to perform KV lookups in [15, 15], got 15
    validate_benchmark_data.go:123: success: expected AlterTableAddCheckConstraint/alter_table_add_2_check_constraints to perform KV lookups in [15, 15], got 15
    validate_benchmark_data.go:123: success: expected GenerateObjects/generate_50000_tables to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected Grant/grant_all_on_2_tables to perform KV lookups in [19, 19], got 19
    validate_benchmark_data.go:123: success: expected Discard/DISCARD_ALL,_1_tables_in_1_db to perform KV lookups in [16, 16], got 16
    validate_benchmark_data.go:123: success: expected GenerateObjects/generate_100_tables_from_template to perform KV lookups in [11, 11], got 12
    validate_benchmark_data.go:123: success: expected CreateRole/create_role_with_1_option to perform KV lookups in [26, 26], got 26
    validate_benchmark_data.go:123: success: expected CreateRole/create_role_with_2_options to perform KV lookups in [29, 29], got 29
    validate_benchmark_data.go:123: success: expected CreateRole/create_role_with_no_options to perform KV lookups in [27, 27], got 27
    validate_benchmark_data.go:123: success: expected DropTable/drop_2_tables to perform KV lookups in [17, 17], got 17
    validate_benchmark_data.go:123: success: expected SystemDatabaseQueries/select_system.users_with_schema_Name to perform KV lookups in [1, 1], got 1
    validate_benchmark_data.go:123: success: expected SystemDatabaseQueries/select_system.users_without_schema_Name to perform KV lookups in [1, 1], got 1
    validate_benchmark_data.go:123: success: expected RevokeRole/revoke_1_role to perform KV lookups in [17, 17], got 17
    validate_benchmark_data.go:123: success: expected ORMQueries/npgsql_types to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected DropSequence/drop_2_sequences to perform KV lookups in [17, 17], got 17
    validate_benchmark_data.go:123: success: expected AlterTableConfigureZone/alter_table_configure_zone_7_replicas_ to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected ORMQueries/npgsql_fields to perform KV lookups in [5, 5], got 5
    validate_benchmark_data.go:123: success: expected AlterTableConfigureZone/alter_table_configure_zone_5_replicas to perform KV lookups in [8, 8], got 8
    validate_benchmark_data.go:123: success: expected DropSequence/drop_1_sequence to perform KV lookups in [15, 15], got 15
    validate_benchmark_data.go:123: success: expected ORMQueries/prisma_column_descriptions_updated to perform KV lookups in [3, 3], got 4
    validate_benchmark_data.go:123: success: expected ShowGrants/grant_2_roles to perform KV lookups in [14, 14], got 14
    validate_benchmark_data.go:123: success: expected ORMQueries/liquibase_migrations to perform KV lookups in [4, 4], got 4
    validate_benchmark_data.go:123: success: expected ShowGrants/grant_3_roles to perform KV lookups in [16, 16], got 16
    validate_benchmark_data.go:123: success: expected ORMQueries/liquibase_migrations_on_multiple_dbs to perform KV lookups in [8, 8], got 7
    validate_benchmark_data.go:123: success: expected ORMQueries/column_descriptions_json_agg to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:123: success: expected ORMQueries/prisma_column_descriptions to perform KV lookups in [133, 133], got 133
    validate_benchmark_data.go:123: success: expected ORMQueries/asyncpg_types to perform KV lookups in [0, 0], got 0
    validate_benchmark_data.go:76: see the -rewrite flag to re-run the benchmarks and adjust the expectations
    validate_benchmark_data.go:77: usage: ./dev test --rewrite pkg/bench/rttanalysis -f=TestBenchmarkExpectation/ --test-args '-rewrite-iterations=N'
    validate_benchmark_data.go:110: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/b03b0378e25aebc8fcd368039859e7a0/logTestBenchmarkExpectation2421514424
--- FAIL: TestBenchmarkExpectation (207.79s)
--- FAIL: TestBenchmarkExpectation (208.38s)
=== RUN   TestBenchmarkExpectation/Jobs
    --- FAIL: TestBenchmarkExpectation/Jobs (116.12s)
=== RUN   TestBenchmarkExpectation/Jobs/jobs_page_latest_50
    rtt_analysis_bench.go:215: Jobs/jobs_page_latest_50: got 5.5, expected 8
    rtt_analysis_bench.go:221: wrote jaeger trace to /artifacts/tmp/_tmp/b03b0378e25aebc8fcd368039859e7a0/logTestBenchmarkExpectation2421514424/Jobs_jobs_page_latest_50.jaeger.json
        --- FAIL: TestBenchmarkExpectation/Jobs/jobs_page_latest_50 (114.13s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

benbardin commented 1 month ago

Do we still consider this P2? Should we demote to P3 or close?