search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.02k stars 3.79k forks source link

pkg/sql/schemachanger/schemachanger_test: TestAlterTableDMLInjection failed [test only license enforcer race condition] #131007

Closed cockroach-teamcity closed 1 week ago

cockroach-teamcity commented 3 weeks ago

pkg/sql/schemachanger/schemachanger_test.TestAlterTableDMLInjection failed on release-24.2.3-rc @ 3b7f092a54444f023a393197b74572da2cd04d5a:

      pkg/sql/schemachanger/schemachanger_test_test/pkg/sql/schemachanger/dml_injection_test.go:426 +0x13fd
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Goroutine 148 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1742 +0x825
  pkg/sql/schemachanger/schemachanger_test_test.TestAlterTableDMLInjection()
      pkg/sql/schemachanger/schemachanger_test_test/pkg/sql/schemachanger/dml_injection_test.go:426 +0x13fd
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44
==================
=== RUN   TestAlterTableDMLInjection/add_column_stored
=== PAUSE TestAlterTableDMLInjection/add_column_stored
=== RUN   TestAlterTableDMLInjection/create_expression_index
=== PAUSE TestAlterTableDMLInjection/create_expression_index
=== CONT  TestAlterTableDMLInjection/create_expression_index
    test_server_shim.go:157: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.
=== RUN   TestAlterTableDMLInjection/drop_column_stored
=== PAUSE TestAlterTableDMLInjection/drop_column_stored
=== RUN   TestAlterTableDMLInjection/add_column_and_check_constraint
=== PAUSE TestAlterTableDMLInjection/add_column_and_check_constraint
=== RUN   TestAlterTableDMLInjection/create_partial_index
=== PAUSE TestAlterTableDMLInjection/create_partial_index
=== RUN   TestAlterTableDMLInjection/create_unique_index
=== PAUSE TestAlterTableDMLInjection/create_unique_index
=== RUN   TestAlterTableDMLInjection/drop_column_with_unique_index
=== PAUSE TestAlterTableDMLInjection/drop_column_with_unique_index
=== RUN   TestAlterTableDMLInjection/drop_constraint_unique_without_index
=== PAUSE TestAlterTableDMLInjection/drop_constraint_unique_without_index
=== RUN   TestAlterTableDMLInjection/drop_index
=== PAUSE TestAlterTableDMLInjection/drop_index
=== RUN   TestAlterTableDMLInjection/add_column_unique_not_null
=== PAUSE TestAlterTableDMLInjection/add_column_unique_not_null
=== RUN   TestAlterTableDMLInjection/add_constraint_check_not_valid
=== PAUSE TestAlterTableDMLInjection/add_constraint_check_not_valid
=== RUN   TestAlterTableDMLInjection/add_constraint_check_udf
=== PAUSE TestAlterTableDMLInjection/add_constraint_check_udf
=== RUN   TestAlterTableDMLInjection/add_constraint_foreign_key
=== PAUSE TestAlterTableDMLInjection/add_constraint_foreign_key
=== RUN   TestAlterTableDMLInjection/add_primary_key
=== PAUSE TestAlterTableDMLInjection/add_primary_key
=== RUN   TestAlterTableDMLInjection/drop_partial_index
=== PAUSE TestAlterTableDMLInjection/drop_partial_index
=== RUN   TestAlterTableDMLInjection/drop_unique_index
=== PAUSE TestAlterTableDMLInjection/drop_unique_index

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-42334

cockroach-teamcity commented 3 weeks ago

pkg/sql/schemachanger/schemachanger_test.TestAlterTableDMLInjection failed on release-24.2.3-rc @ 3b7f092a54444f023a393197b74572da2cd04d5a:

      github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1759 +0x242f
  github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:2070 +0x5aa4
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:790 +0x104
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).PreStart()
      github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:19 +0x72
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:415 +0x57e
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_cluster_shim.go:307 +0x10f
  pkg/sql/schemachanger/schemachanger_test_test.TestAlterTableDMLInjection.func2()
      pkg/sql/schemachanger/schemachanger_test_test/pkg/sql/schemachanger/dml_injection_test.go:443 +0x618
  pkg/sql/schemachanger/schemachanger_test_test.TestAlterTableDMLInjection.testCase.capture.func3()
      pkg/sql/schemachanger/schemachanger_test_test/pkg/sql/schemachanger/dml_injection_test.go:129 +0xa1
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Goroutine 107 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1742 +0x825
  pkg/sql/schemachanger/schemachanger_test_test.TestAlterTableDMLInjection()
      pkg/sql/schemachanger/schemachanger_test_test/pkg/sql/schemachanger/dml_injection_test.go:426 +0x13fd
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Goroutine 129 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1742 +0x825
  pkg/sql/schemachanger/schemachanger_test_test.TestAlterTableDMLInjection()
      pkg/sql/schemachanger/schemachanger_test_test/pkg/sql/schemachanger/dml_injection_test.go:426 +0x13fd
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44
==================
=== RUN   TestAlterTableDMLInjection/add_constraint_check_udf
=== PAUSE TestAlterTableDMLInjection/add_constraint_check_udf
=== RUN   TestAlterTableDMLInjection/add_constraint_foreign_key
=== PAUSE TestAlterTableDMLInjection/add_constraint_foreign_key
=== RUN   TestAlterTableDMLInjection/alter_primary_key_using_columns
=== PAUSE TestAlterTableDMLInjection/alter_primary_key_using_columns
=== RUN   TestAlterTableDMLInjection/create_index
=== PAUSE TestAlterTableDMLInjection/create_index
=== RUN   TestAlterTableDMLInjection/create_materialized_view_from_index
=== PAUSE TestAlterTableDMLInjection/create_materialized_view_from_index

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

fqazi commented 3 weeks ago

@spilchen This is linked to a data race in the enforce, if this is already fix then close it.

WARNING: DATA RACE
Write at 0x00c0014ecf00 by goroutine 130:
  github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).startLicenseEnforcer()
      github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1916 +0x14b
  github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart()
      github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1759 +0x242f
  github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:2070 +0x5aa4
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:790 +0x104
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).PreStart()
      github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:19 +0x72
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:415 +0x57e
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_cluster_shim.go:307 +0x10f
  pkg/sql/schemachanger/schemachanger_test_test.TestAlterTableDMLInjection.func2()
      pkg/sql/schemachanger/schemachanger_test_test/pkg/sql/schemachanger/dml_injection_test.go:443 +0x618
  pkg/sql/schemachanger/schemachanger_test_test.TestAlterTableDMLInjection.testCase.capture.func3()
      pkg/sql/schemachanger/schemachanger_test_test/pkg/sql/schemachanger/dml_injection_test.go:129 +0xa1
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44
Previous write at 0x00c0014ecf00 by goroutine 148:
  github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).startLicenseEnforcer()
      github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1916 +0x14b
  github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart()
      github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1759 +0x242f
  github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:2070 +0x5aa4
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:790 +0x104
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).PreStart()
      github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:19 +0x72
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:415 +0x57e
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.StartCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/serverutils/test_cluster_shim.go:307 +0x10f
  pkg/sql/schemachanger/schemachanger_test_test.TestAlterTableDMLInjection.func2()
      pkg/sql/schemachanger/schemachanger_test_test/pkg/sql/schemachanger/dml_injection_test.go:443 +0x618
  pkg/sql/schemachanger/schemachanger_test_test.TestAlterTableDMLInjection.testCase.capture.func3()
      pkg/sql/schemachanger/schemachanger_test_test/pkg/sql/schemachanger/dml_injection_test.go:129 +0xa1
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44
spilchen commented 3 weeks ago

This issue has been fixed, but the fix is not in the 24.2.3-rc branch. Since the release candidate has been chosen and is undergoing final testing, I'm unsure if it makes sense to apply the fix to this branch.

spilchen commented 2 weeks ago

I’m not going to address this in the RC branch, as it's very close to being published (ETA 9/30). The fix only impacts testing, and there’s no issue with the server itself since the license enforcer is currently disabled. The problem only arises due to parallel test servers running in the same process.

cockroach-teamcity commented 2 weeks ago

pkg/sql/schemachanger/schemachanger_test.TestAlterTableDMLInjection failed on release-24.2.3-rc @ 217b43e69da61e9712d73e3e19fd5076fbdb26df:

=== RUN   TestAlterTableDMLInjection/drop_expression_index
=== PAUSE TestAlterTableDMLInjection/drop_expression_index
=== RUN   TestAlterTableDMLInjection/validate_constraint
=== PAUSE TestAlterTableDMLInjection/validate_constraint
=== RUN   TestAlterTableDMLInjection/add_column_and_check_constraint
=== PAUSE TestAlterTableDMLInjection/add_column_and_check_constraint
=== RUN   TestAlterTableDMLInjection/add_constraint_check
=== PAUSE TestAlterTableDMLInjection/add_constraint_check
=== RUN   TestAlterTableDMLInjection/drop_column_stored_family
=== PAUSE TestAlterTableDMLInjection/drop_column_stored_family
=== RUN   TestAlterTableDMLInjection/drop_column_with_index_using_hash_cascade
=== PAUSE TestAlterTableDMLInjection/drop_column_with_index_using_hash_cascade
=== RUN   TestAlterTableDMLInjection/drop_index_cascade_to_materialized_view
=== PAUSE TestAlterTableDMLInjection/drop_index_cascade_to_materialized_view
=== RUN   TestAlterTableDMLInjection/add_primary_key
=== PAUSE TestAlterTableDMLInjection/add_primary_key
=== CONT  TestAlterTableDMLInjection/add_primary_key
    test_server_shim.go:157: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.
=== RUN   TestAlterTableDMLInjection/create_partial_index
=== PAUSE TestAlterTableDMLInjection/create_partial_index
=== RUN   TestAlterTableDMLInjection/drop_index
=== PAUSE TestAlterTableDMLInjection/drop_index
=== RUN   TestAlterTableDMLInjection/drop_unique_index
=== PAUSE TestAlterTableDMLInjection/drop_unique_index
=== CONT  TestAlterTableDMLInjection/drop_unique_index
    test_server_shim.go:157: automatically injected a shared process virtual cluster under test; see comment at top of test_server_shim.go for details.
=== RUN   TestAlterTableDMLInjection/add_column_default_sequence
=== PAUSE TestAlterTableDMLInjection/add_column_default_sequence
=== RUN   TestAlterTableDMLInjection/add_column_default_value
=== PAUSE TestAlterTableDMLInjection/add_column_default_value
=== RUN   TestAlterTableDMLInjection/alter_primary_key_using_columns_using_hash
=== PAUSE TestAlterTableDMLInjection/alter_primary_key_using_columns_using_hash
=== RUN   TestAlterTableDMLInjection/create_materialized_view_from_index
=== PAUSE TestAlterTableDMLInjection/create_materialized_view_from_index
=== RUN   TestAlterTableDMLInjection/drop_column
=== PAUSE TestAlterTableDMLInjection/drop_column
=== RUN   TestAlterTableDMLInjection/add_column_virtual_NOT_NULL
=== PAUSE TestAlterTableDMLInjection/add_column_virtual_NOT_NULL
=== RUN   TestAlterTableDMLInjection/add_constraint_check_udt
=== PAUSE TestAlterTableDMLInjection/add_constraint_check_udt
=== RUN   TestAlterTableDMLInjection/create_index
=== PAUSE TestAlterTableDMLInjection/create_index
=== RUN   TestAlterTableDMLInjection/create_index_using_hash
=== PAUSE TestAlterTableDMLInjection/create_index_using_hash
=== RUN   TestAlterTableDMLInjection/drop_column_with_index
=== PAUSE TestAlterTableDMLInjection/drop_column_with_index
=== RUN   TestAlterTableDMLInjection/drop_column_with_partial_index
=== PAUSE TestAlterTableDMLInjection/drop_column_with_partial_index
=== RUN   TestAlterTableDMLInjection/drop_column_with_unique_index
=== PAUSE TestAlterTableDMLInjection/drop_column_with_unique_index

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

rafiss commented 1 week ago

closing since 24.2.3 was released