Open clairemcginty opened 5 years ago
Update: the client library bug affecting extract jobs has been fixed! https://github.com/googleapis/google-cloud-java/issues/3924
@ClaireMcGinty is this still worth looking?
Talked IRL, closing.
I would like us to reconsider re-opening this. I think there's still some subtle bugs in our current internal BigQuery client. Some of these bugs are related to not fallbacking to env setting properties.
@nevillelyh @ClaireMcGinty what was the reason to not go forward with this?
@regadas If I remember right, it was due to the complexity of integrating with Beam's BigQuery sources/sinks -- Beam returned types from google-api-services-bigquery
and a lot of the Google library functions that could convert those to google-cloud-bigquery
types were private.
This was awhile ago though, so maybe worth a second look?
@ClaireMcGinty interesting! I think it's worth looking into it again since we are already using the storage
impl to actually retrieve data.
Let's see if the other types are good to go as well. I'll book some time to look into this.
Thanks
Google documentation recommends using the client library
google-cloud-bigquery
rather than the API librarygoogle-api-services-bigquery
.Pros
google-cloud-bigquery
uses typed protobuf API request/response params rather than plain strings/ints, implicitly handles transport layer configurations, and potentially improves performance by making direct RPC calls rather than JSON over HTTP. We've seen intermittent failures with the scio-bigquery IT suite due to network timeouts, which might be solved by migrating.google-api-services-bigquery
library is in maintenance mode and aside from critical bug fixes, won't have any new features added.Cons
TableSchema
,TableReference
, andTableRow
- map toSchema
,TableId
, andFieldValueList
in the client library. So, if we end up migrating, we'd have to decide whether to change the externally facing Scio API or handle those conversions ourselves in private methods. I have a WIP branch for this migration I'll link to as soon as it's cleaned up.