Consider using google-cloud-bigquery library instead of google-api-services-bigquery

clairemcginty commented 6 years ago

Google documentation recommends using the client library google-cloud-bigquery rather than the API library google-api-services-bigquery.

Pros

google-cloud-bigquery uses typed protobuf API request/response params rather than plain strings/ints, implicitly handles transport layer configurations, and potentially improves performance by making direct RPC calls rather than JSON over HTTP. We've seen intermittent failures with the scio-bigquery IT suite due to network timeouts, which might be solved by migrating.
the google-api-services-bigquery library is in maintenance mode and aside from critical bug fixes, won't have any new features added.

Cons

Unfortunately, the data models are quite different, and the three classes from the API library that we publicly expose in Scio - TableSchema, TableReference, and TableRow - map to Schema, TableId, and FieldValueList in the client library. So, if we end up migrating, we'd have to decide whether to change the externally facing Scio API or handle those conversions ourselves in private methods. I have a WIP branch for this migration I'll link to as soon as it's cleaned up.
While I was developing that branch I found an issue with the client library that breaks cross-project extraction jobs: https://github.com/googleapis/google-cloud-java/issues/3924 , so in its current state, client library is not fully usable in Scio.

clairemcginty commented 6 years ago

Update: the client library bug affecting extract jobs has been fixed! https://github.com/googleapis/google-cloud-java/issues/3924

nevillelyh commented 4 years ago

@ClaireMcGinty is this still worth looking?

nevillelyh commented 4 years ago

Talked IRL, closing.

regadas commented 4 years ago

I would like us to reconsider re-opening this. I think there's still some subtle bugs in our current internal BigQuery client. Some of these bugs are related to not fallbacking to env setting properties.

regadas commented 4 years ago

@nevillelyh @ClaireMcGinty what was the reason to not go forward with this?

clairemcginty commented 4 years ago

@regadas If I remember right, it was due to the complexity of integrating with Beam's BigQuery sources/sinks -- Beam returned types from google-api-services-bigquery and a lot of the Google library functions that could convert those to google-cloud-bigquery types were private.

This was awhile ago though, so maybe worth a second look?

regadas commented 4 years ago

@ClaireMcGinty interesting! I think it's worth looking into it again since we are already using the storage impl to actually retrieve data.

Let's see if the other types are good to go as well. I'll book some time to look into this.

Thanks

spotify / scio

Consider using google-cloud-bigquery library instead of google-api-services-bigquery #1555