search

silverton-io / buz

Serverless multi-protocol + multi-destination event collection system.
https://buz.dev/
Apache License 2.0
186 stars 21 forks source link

[Sink Request] Update BigQuery sink to support Storage Write API #584

Open seanberto opened 6 months ago

seanberto commented 6 months ago

See: https://cloud.google.com/bigquery/docs/write-api.

As of this month, it looks like the Go package can wrap this API:

https://pkg.go.dev/cloud.google.com/go/bigquery/storage/managedwriter

The main benefit of using this API over the legacy tabledata.insertAll method is that the new method is much less expensive:

https://cloud.google.com/bigquery/pricing#data_ingestion_pricing

seanberto commented 5 months ago

@jakthom - Would love to help sponsor this feature, if you're open to it.