Hey, I'm wondering about Unbounded Source and data freshness, if I understand correctly, data will be pulled from BQ only when a partition is considered completed (Daily / Hourly). Could someone confirm 🙏
If that's the case, how easy would it be to have another configuration flag allowing us to periodically pull the current partition's content (say every X minutes | Y seconds).
Avoiding duplication through keeping track of the last synced row timestamp.
What would be the tradeoff in trying to do that ? Would it incur additional cost ?
Hey, I'm wondering about Unbounded Source and data freshness, if I understand correctly, data will be pulled from BQ only when a partition is considered completed (Daily / Hourly). Could someone confirm 🙏
If that's the case, how easy would it be to have another configuration flag allowing us to periodically pull the current partition's content (say every X minutes | Y seconds). Avoiding duplication through keeping track of the last synced row timestamp.
What would be the tradeoff in trying to do that ? Would it incur additional cost ?