search

cockroachdb / cockroach

CockroachDB - the open source, cloud-native distributed SQL database.
https://www.cockroachlabs.com
Other
29.49k stars 3.7k forks source link

ccl/changefeedccl: TestNoBackfillAfterNonTargetColumnDrop failed #126096

Open cockroach-teamcity opened 1 week ago

cockroach-teamcity commented 1 week ago

ccl/changefeedccl.TestNoBackfillAfterNonTargetColumnDrop failed on master @ eefcdf2f7356849757646ba8c74364b9e2b8f1cf:

    test_log_scope.go:81: use -show-logs to present logs inline
    changefeed_test.go:1636: -- test log scope end --
test logs left over in: outputs.zip/logTestNoBackfillAfterNonTargetColumnDrop3287970811
--- FAIL: TestNoBackfillAfterNonTargetColumnDrop (15.63s)
=== RUN   TestNoBackfillAfterNonTargetColumnDrop/sinkless
    helpers_test.go:864: making server as secondary tenant
    helpers_test.go:943: making sinkless feed factory
    helpers_test.go:1008: pgURL sinkless SinklessFeedUser
    helpers_test.go:1008: pgURL sinkless root
    changefeed_test.go:1623: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:199
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:274
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:1623
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1104
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1139
            Error:          Received unexpected error:
                            expected message
                            (1) assertion failure
                            Wraps: (2) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.readNextMessages
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:119
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBaseErr
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:210
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:202
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:267
                              | github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout
                              |     github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:29
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.withTimeout
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:263
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloadsBase
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:200
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.assertPayloads
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:274
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestNoBackfillAfterNonTargetColumnDrop.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:1623
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamed.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1104
                              | github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamedWithSystem.func1
                              |     github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1139
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1689
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (3) expected message
                            Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
            Test:           TestNoBackfillAfterNonTargetColumnDrop/sinkless
    --- FAIL: TestNoBackfillAfterNonTargetColumnDrop/sinkless (15.63s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-39773

andyyang890 commented 4 days ago

I found this error in the logs:

E240624 11:06:53.127497 26037 ccl/changefeedccl/cdcevent/event.go:513 ⋮ [T10,Vcluster-10,nsql1,client=127.0.0.1:55006,hostssl,user=‹sinklessfeeduser›] 342  terminal error decoding KV: error decoding key ‹/Tenant/10/Table/106/1/0/0›@1719227212.791416215,0 (hex_kv: ‹0a06fe92f289888812160a08aa0ad5ad0a260161120a08978bf8d4a881fbed17›): watched table but unwatched family

We should look into this more, but doesn't seem like a release blocker since it seems really rare and doesn't reproduce under stress.