search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.96k stars 3.79k forks source link

ccl/changefeedccl: TestSchemachangeDoesNotBreakSinklessFeed failed #125407

Closed cockroach-teamcity closed 2 months ago

cockroach-teamcity commented 3 months ago

ccl/changefeedccl.TestSchemachangeDoesNotBreakSinklessFeed failed on master @ e924a27896b9fbf495609d3965612366de6ebf78:

    test_log_scope.go:81: use -show-logs to present logs inline
    changefeed_test.go:8428: -- test log scope end --
test logs left over in: outputs.zip/logTestSchemachangeDoesNotBreakSinklessFeed1981966677
--- FAIL: TestSchemachangeDoesNotBreakSinklessFeed (15.77s)
=== RUN   TestSchemachangeDoesNotBreakSinklessFeed/sinkless
    helpers_test.go:864: making server as secondary tenant
    helpers_test.go:943: making sinkless feed factory
    helpers_test.go:1008: pgURL sinkless SinklessFeedUser
    helpers_test.go:1008: pgURL sinkless root
    changefeed_test.go:8418: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:199
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:274
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:8418
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1104
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1139
            Error:          Received unexpected error:
                            expected message
                            (1) assertion failure
                            Wraps: (2) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.readNextMessages
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:119
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBaseErr
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:210
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:202
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:267
                              | github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout
                              |     github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:29
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:263
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:200
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloads
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:274
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestSchemachangeDoesNotBreakSinklessFeed.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:8418
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamed.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1104
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamedWithSystem.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1139
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1689
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (3) expected message
                            Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
            Test:           TestSchemachangeDoesNotBreakSinklessFeed/sinkless
    --- FAIL: TestSchemachangeDoesNotBreakSinklessFeed/sinkless (15.76s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-39441

cockroach-teamcity commented 3 months ago

ccl/changefeedccl.TestSchemachangeDoesNotBreakSinklessFeed failed on master @ 5afd790501e946ef306abe2b592c5798c29c342f:

    test_log_scope.go:81: use -show-logs to present logs inline
    changefeed_test.go:8512: -- test log scope end --
test logs left over in: outputs.zip/logTestSchemachangeDoesNotBreakSinklessFeed2877182120
--- FAIL: TestSchemachangeDoesNotBreakSinklessFeed (15.50s)
=== RUN   TestSchemachangeDoesNotBreakSinklessFeed/sinkless
    helpers_test.go:864: making server as secondary tenant
    helpers_test.go:943: making sinkless feed factory
    helpers_test.go:1008: pgURL sinkless SinklessFeedUser
    helpers_test.go:1008: pgURL sinkless root
    changefeed_test.go:8502: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:199
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:274
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:8502
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1104
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1139
            Error:          Received unexpected error:
                            expected message
                            (1) assertion failure
                            Wraps: (2) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.readNextMessages
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:119
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBaseErr
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:210
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:202
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:267
                              | github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout
                              |     github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:29
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:263
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:200
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloads
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:274
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestSchemachangeDoesNotBreakSinklessFeed.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:8502
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamed.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1104
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamedWithSystem.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1139
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1689
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (3) expected message
                            Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
            Test:           TestSchemachangeDoesNotBreakSinklessFeed/sinkless
    --- FAIL: TestSchemachangeDoesNotBreakSinklessFeed/sinkless (15.49s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

wenyihu6 commented 3 months ago

The failure message here looks the same as https://github.com/cockroachdb/cockroach/issues/126096. But it shouldn't block the release. I will look into this more.

cockroach-teamcity commented 3 months ago

ccl/changefeedccl.TestSchemachangeDoesNotBreakSinklessFeed failed on master @ b4402507fdebe5682f77e348331a308d378c91e1:

    test_log_scope.go:81: use -show-logs to present logs inline
    changefeed_test.go:8516: -- test log scope end --
test logs left over in: outputs.zip/logTestSchemachangeDoesNotBreakSinklessFeed2666876705
--- FAIL: TestSchemachangeDoesNotBreakSinklessFeed (15.51s)
=== RUN   TestSchemachangeDoesNotBreakSinklessFeed/sinkless
    helpers_test.go:864: making server as secondary tenant
    helpers_test.go:943: making sinkless feed factory
    helpers_test.go:1008: pgURL sinkless SinklessFeedUser
    helpers_test.go:1008: pgURL sinkless root
    changefeed_test.go:8506: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:199
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:274
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:8506
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1104
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1139
            Error:          Received unexpected error:
                            expected message
                            (1) assertion failure
                            Wraps: (2) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.readNextMessages
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:119
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBaseErr
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:210
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:202
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:267
                              | github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout
                              |     github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:29
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:263
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:200
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloads
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:274
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestSchemachangeDoesNotBreakSinklessFeed.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:8506
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamed.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1104
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamedWithSystem.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1139
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1689
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (3) expected message
                            Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
            Test:           TestSchemachangeDoesNotBreakSinklessFeed/sinkless
    --- FAIL: TestSchemachangeDoesNotBreakSinklessFeed/sinkless (15.50s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

rharding6373 commented 2 months ago

I'm going to close this. As mentioned in https://github.com/cockroachdb/cockroach/issues/126890#issuecomment-2239826637 Andy added some logging to core changefeeds to help debug this issue if it fails again.