Open cockroach-teamcity opened 1 week ago
ccl/changefeedccl.TestChangefeedSchemaChangeBackfillCheckpoint failed on release-24.2 @ 52fafd7ae8a078124f983d0c60550e9f38c68462:
=== RUN TestChangefeedSchemaChangeBackfillCheckpoint
test_log_scope.go:165: test logs captured to: outputs.zip/logTestChangefeedSchemaChangeBackfillCheckpoint3671982161
test_log_scope.go:76: use -show-logs to present logs inline
changefeed_test.go:2250: random seed: -8238013932062361849
changefeed_test.go:2521: -- test log scope end --
test logs left over in: outputs.zip/logTestChangefeedSchemaChangeBackfillCheckpoint3671982161
--- FAIL: TestChangefeedSchemaChangeBackfillCheckpoint (52.42s)
=== RUN TestChangefeedSchemaChangeBackfillCheckpoint/cloudstorage
helpers_test.go:872: making server as system tenant
helpers_test.go:948: making cloudstorage feed factory
helpers_test.go:65: using legacy schema changer
testfeed_test.go:301: creating external connection
testfeed_test.go:304: ran create external connection
changefeed_test.go:2316: highwater: 1728981421.362862902,0
changefeed_test.go:2361: backfill timestamp: 1728981423.442265721,0
changefeed_test.go:2408: initial checkpoint: [/Table/106/1/{101/0-501/0}]
changefeed_test.go:2467: condition failed to evaluate within 45s: from changefeed_test.go:2471: waiting for second checkpoint
--- FAIL: TestChangefeedSchemaChangeBackfillCheckpoint/cloudstorage (52.41s)
Parameters:
attempt=1
run=18
shard=11
I think I figured out the issue, we were sometimes not seeing the second checkpoint get bigger because we wrote a checkpoint again after just having processed the same spans from the initial checkpoint and then the backfill completed before we could write another checkpoint. The testing hooks prevented the changefeed from processing the completed backfill so the test eventually times out.
See https://github.com/cockroachdb/cockroach/issues/133492 for more details.
Removing the release blocker tag because it's mostly a test issue.
ccl/changefeedccl.TestChangefeedSchemaChangeBackfillCheckpoint failed on release-24.2 @ cf9b1574fd03c877cc831f4be41a5e12b1055a42:
Parameters:
attempt=1
run=10
shard=11
Help
See also: How To Investigate a Go Test Failure (internal)
/cc @cockroachdb/cdcThis test on roachdash | Improve this report!
Jira issue: CRDB-43136