search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.1k stars 3.81k forks source link

ccl/changefeedccl: TestSchemachangeDoesNotBreakSinklessFeed failed #129777

Open cockroach-teamcity opened 2 months ago

cockroach-teamcity commented 2 months ago

ccl/changefeedccl.TestSchemachangeDoesNotBreakSinklessFeed failed on release-24.2 @ bd9b4c2bf6273e425bd4ff4f33ca11bda00d8069:

test logs left over in: outputs.zip/logTestSchemachangeDoesNotBreakSinklessFeed4030737070
--- FAIL: TestSchemachangeDoesNotBreakSinklessFeed (15.09s)
=== RUN   TestSchemachangeDoesNotBreakSinklessFeed/sinkless
    helpers_test.go:872: making server as secondary tenant
    helpers_test.go:951: making sinkless feed factory
    helpers_test.go:1016: pgURL sinkless SinklessFeedUser
    helpers_test.go:1016: pgURL sinkless root
    testfeed_test.go:273: sinkless feed creating changefeed: CREATE CHANGEFEED FOR TABLE mytable
    changefeed_test.go:8534: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:200
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:275
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:8534
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1123
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1158
            Error:          Received unexpected error:
                            expected message
                            (1) assertion failure
                            Wraps: (2) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.readNextMessages
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:120
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBaseErr
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:211
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:203
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:268
                              | github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout
                              |     github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:33
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:264
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:201
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloads
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:275
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestSchemachangeDoesNotBreakSinklessFeed.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:8534
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamed.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1123
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamedWithSystem.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1158
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1689
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (3) expected message
                            Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
            Test:           TestSchemachangeDoesNotBreakSinklessFeed/sinkless
    testfeed_test.go:280: closing sinkless feed
    --- FAIL: TestSchemachangeDoesNotBreakSinklessFeed/sinkless (15.08s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #129226 ccl/changefeedccl: TestSchemachangeDoesNotBreakSinklessFeed failed [A-cdc C-test-failure O-robot P-2 T-cdc branch-release-24.2.0-rc]

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-41705

andyyang890 commented 2 months ago

Same symptoms as #129045, but new logs added in #129335 show:

I240828 08:33:21.773690 24598036 ccl/changefeedccl/changefeed_stmt.go:356 ⋮ [T10,Vcluster-10,nsql1,client=127.0.0.1:44884,hostssl,user=‹sinklessfeeduser›] 403  core changefeed completed with no error

Will need to dig further to understand why this is happening.

cockroach-teamcity commented 2 months ago

ccl/changefeedccl.TestSchemachangeDoesNotBreakSinklessFeed failed on release-24.2 @ 14962dfb7274e11754658d4fa2be32a1dd31b94b:

test logs left over in: outputs.zip/logTestSchemachangeDoesNotBreakSinklessFeed1377494001
--- FAIL: TestSchemachangeDoesNotBreakSinklessFeed (16.11s)
=== RUN   TestSchemachangeDoesNotBreakSinklessFeed/sinkless
    helpers_test.go:872: making server as secondary tenant
    helpers_test.go:951: making sinkless feed factory
    helpers_test.go:1016: pgURL sinkless SinklessFeedUser
    helpers_test.go:1016: pgURL sinkless root
    testfeed_test.go:273: sinkless feed creating changefeed: CREATE CHANGEFEED FOR TABLE mytable
    changefeed_test.go:8534: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:200
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:275
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:8534
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1123
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1158
            Error:          Received unexpected error:
                            expected message
                            (1) assertion failure
                            Wraps: (2) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.readNextMessages
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:120
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBaseErr
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:211
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:203
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:268
                              | github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout
                              |     github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:33
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:264
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:201
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloads
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:275
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestSchemachangeDoesNotBreakSinklessFeed.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:8534
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamed.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1123
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamedWithSystem.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1158
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1689
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (3) expected message
                            Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
            Test:           TestSchemachangeDoesNotBreakSinklessFeed/sinkless
    testfeed_test.go:280: closing sinkless feed
    --- FAIL: TestSchemachangeDoesNotBreakSinklessFeed/sinkless (16.10s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #129226 ccl/changefeedccl: TestSchemachangeDoesNotBreakSinklessFeed failed [A-cdc C-test-failure O-robot P-2 T-cdc branch-release-24.2.0-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

ccl/changefeedccl.TestSchemachangeDoesNotBreakSinklessFeed failed on release-24.2 @ 15f96a5f628415513d7897928e7ccde08ebec5fb:

test logs left over in: outputs.zip/logTestSchemachangeDoesNotBreakSinklessFeed251712890
--- FAIL: TestSchemachangeDoesNotBreakSinklessFeed (15.22s)
=== RUN   TestSchemachangeDoesNotBreakSinklessFeed/sinkless
    helpers_test.go:872: making server as secondary tenant
    helpers_test.go:951: making sinkless feed factory
    helpers_test.go:1016: pgURL sinkless SinklessFeedUser
    helpers_test.go:1016: pgURL sinkless root
    testfeed_test.go:273: sinkless feed creating changefeed: CREATE CHANGEFEED FOR TABLE mytable
    changefeed_test.go:8589: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:200
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:275
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:8589
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1123
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1158
            Error:          Received unexpected error:
                            expected message
                            (1) assertion failure
                            Wraps: (2) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.readNextMessages
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:120
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBaseErr
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:211
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:203
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:268
                              | github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout
                              |     github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:33
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:264
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:201
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloads
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:275
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestSchemachangeDoesNotBreakSinklessFeed.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:8589
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamed.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1123
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamedWithSystem.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1158
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1689
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (3) expected message
                            Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
            Test:           TestSchemachangeDoesNotBreakSinklessFeed/sinkless
    testfeed_test.go:280: closing sinkless feed
    --- FAIL: TestSchemachangeDoesNotBreakSinklessFeed/sinkless (15.21s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #131186 ccl/changefeedccl: TestSchemachangeDoesNotBreakSinklessFeed failed [A-cdc C-test-failure O-robot P-2 T-cdc X-duplicate branch-release-24.2.3-rc] - #129226 ccl/changefeedccl: TestSchemachangeDoesNotBreakSinklessFeed failed [A-cdc C-test-failure O-robot P-2 T-cdc branch-release-24.2.0-rc]

This test on roachdash | Improve this report!