search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.07k stars 3.8k forks source link

ccl/changefeedccl: TestChangefeedSchemaChangeBackfillCheckpoint failed #132548

Open cockroach-teamcity opened 1 week ago

cockroach-teamcity commented 1 week ago

ccl/changefeedccl.TestChangefeedSchemaChangeBackfillCheckpoint failed on release-24.2 @ cf9b1574fd03c877cc831f4be41a5e12b1055a42:

=== RUN   TestChangefeedSchemaChangeBackfillCheckpoint
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestChangefeedSchemaChangeBackfillCheckpoint4096890465
    test_log_scope.go:76: use -show-logs to present logs inline
    changefeed_test.go:2250: random seed: 8069558234357739010
    changefeed_test.go:2521: -- test log scope end --
test logs left over in: outputs.zip/logTestChangefeedSchemaChangeBackfillCheckpoint4096890465
--- FAIL: TestChangefeedSchemaChangeBackfillCheckpoint (51.43s)
=== RUN   TestChangefeedSchemaChangeBackfillCheckpoint/pubsub
    helpers_test.go:872: making server as system tenant
    helpers_test.go:948: making pubsub feed factory
    helpers_test.go:65: using legacy schema changer
    changefeed_test.go:2316: highwater: 1728804604.847654689,0
    changefeed_test.go:2361: backfill timestamp: 1728804606.936638921,2147483647
    changefeed_test.go:2408: initial checkpoint: [/Table/106/1/{101/0-501/0}]
    changefeed_test.go:2467: condition failed to evaluate within 45s: from changefeed_test.go:2471: waiting for second checkpoint
    --- FAIL: TestChangefeedSchemaChangeBackfillCheckpoint/pubsub (51.41s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-43136

cockroach-teamcity commented 1 week ago

ccl/changefeedccl.TestChangefeedSchemaChangeBackfillCheckpoint failed on release-24.2 @ 52fafd7ae8a078124f983d0c60550e9f38c68462:

=== RUN   TestChangefeedSchemaChangeBackfillCheckpoint
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestChangefeedSchemaChangeBackfillCheckpoint3671982161
    test_log_scope.go:76: use -show-logs to present logs inline
    changefeed_test.go:2250: random seed: -8238013932062361849
    changefeed_test.go:2521: -- test log scope end --
test logs left over in: outputs.zip/logTestChangefeedSchemaChangeBackfillCheckpoint3671982161
--- FAIL: TestChangefeedSchemaChangeBackfillCheckpoint (52.42s)
=== RUN   TestChangefeedSchemaChangeBackfillCheckpoint/cloudstorage
    helpers_test.go:872: making server as system tenant
    helpers_test.go:948: making cloudstorage feed factory
    helpers_test.go:65: using legacy schema changer
    testfeed_test.go:301: creating external connection
    testfeed_test.go:304: ran create external connection
    changefeed_test.go:2316: highwater: 1728981421.362862902,0
    changefeed_test.go:2361: backfill timestamp: 1728981423.442265721,0
    changefeed_test.go:2408: initial checkpoint: [/Table/106/1/{101/0-501/0}]
    changefeed_test.go:2467: condition failed to evaluate within 45s: from changefeed_test.go:2471: waiting for second checkpoint
    --- FAIL: TestChangefeedSchemaChangeBackfillCheckpoint/cloudstorage (52.41s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

andyyang890 commented 1 day ago

I think I figured out the issue, we were sometimes not seeing the second checkpoint get bigger because we wrote a checkpoint again after just having processed the same spans from the initial checkpoint and then the backfill completed before we could write another checkpoint. The testing hooks prevented the changefeed from processing the completed backfill so the test eventually times out.

See https://github.com/cockroachdb/cockroach/issues/133492 for more details.

Removing the release blocker tag because it's mostly a test issue.