search

supabase / pg_replicate

Build Postgres replication apps in Rust
Apache License 2.0
596 stars 22 forks source link

Add retries to bypass BigQuery's 10 MB payload limit #54

Open imor opened 1 week ago

imor commented 1 week ago

If the payload to the AppendRows call is greater than 10 MB, BigQuery will reject the call. Detect this and retry with the payload half the original size. Continue this logarithmic halving until the requests succeed. Also collect the retry data as metrics to tune the buffer sizes and retries.

Ziinc commented 2 days ago

Isn't it easier to cap the batches to like 8MB? That is the current logflare approach.