risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
7.06k stars 581 forks source link

feat(cdc): create external table reader with retry for backfill #19467

Open StrikeW opened 2 days ago

StrikeW commented 2 days ago

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Previously we try to create the table reader in src/stream/src/from_proto/stream_cdc_scan.rs, if upstream database is down, the executor cannot be built so that the cluster will enter recovery loop. This pr try to create the table reader after we forward the initial barrier, so that it won't block recovery if cannot connect to upstream db.

close #17807

Checklist

Documentation

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.