search

redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8.15k stars 842 forks source link

New output component for BigQuery for the new Write API #1167

Open Locustv2 opened 2 years ago

Locustv2 commented 2 years ago

As the current BigQuery component is still experimental, it would be good to replace it with a new component using the Write API from Big Query: BigQuery Write API explained: An overview of the Write API

The advantage of using this api is that it uses Protobuf data. Which means it's faster as it will be moving binary data around.

Here are some additional material: Learn how to stream JSON data into BigQuery using the new BigQuery Storage Write API

irfanhabib commented 2 years ago

The link appears to be dead! But I would be interested in this too. New link: https://cloud.google.com/bigquery/docs/write-api