search

googleapis / python-bigquery

Apache License 2.0
746 stars 306 forks source link

Provided functionality and support for BigQuery Storage Write API #2052

Open jlynchMicron opened 2 weeks ago

jlynchMicron commented 2 weeks ago

Is your feature request related to a problem? Please describe. No, just lacking support.

Describe the solution you'd like Please provide client functions and code examples of using the BigQuery Storage Write API, especially in regards to streaming writes. The only streaming write function I see is "client.insert_rows_json()" which uses the legacy streaming API. It would be nice to have a similarly easy to use 1-line function call to use the new BigQuery Storage Write API.

This documentation only shows examples for java and nodejs: https://cloud.google.com/bigquery/docs/write-api-streaming

Describe alternatives you've considered Just using legacy streaming API.

jlynchMicron commented 2 weeks ago

Additional support for Pandas DataFrame upload using streaming inserts instead of batch load inserts would be nice too, to avoid hitting quota limits on batch load jobs when trying to insert lots of small DataFrames from various clients in a real-time fashion.