alvarowolfx
commented
2 months ago @MatthewAry the tabledata.insertAll is called legacy due to it being the original way of sending data to BigQuery, but is not being deprecated or going to be removed. And that streaming buffer limitation exists on the service from the early days and that's why for some use cases, the BQ Storage API might be a better fit.
There is this whole set of BQ Storage APIs that are more optimized for high throughput for read and writing data, that if possible, is recommended to be used if your use case requires such improvements.
But it comes with the caveat that the BQ Storage Write API is not a 1:1 change from the tabledata.insertAll API, although we have been working on wrappers for it to make the experience more seamlessly. You can check some samples like this one:
https://github.com/googleapis/nodejs-bigquery-storage/blob/main/samples/append_rows_json_writer_default.js
Is your feature request related to a problem? Please describe. I recently ran into this problem, and I did not expect it. https://cloud.google.com/knowledge/kb/cannot-update-or-delete-over-bigquery-streaming-tables-000004334 I had spent a lot of time figuring out how to make use of this package in a Bun env to discover that I was using a legacy API that introduces certain caveats, like I can't delete data that I inserted using insertAll until the buffer clears (I think.). And that I should be using a different API and package instead.
Describe the solution you'd like I would like the Rest API docs and the Client SDK documentation to be updated to clearly indicate which methods or endpoints are a legacy API. Like
table.insert()-> https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/insertAll -> ... -> https://cloud.google.com/bigquery/docs/data-manipulation-language#limitations or something like that.Describe alternatives you've considered Talking to ChatGPT or another AI might be more helpful for figuring out why I had issues.