search

spotify / scio

A Scala API for Apache Beam and Google Cloud Dataflow.
https://spotify.github.io/scio
Apache License 2.0
2.55k stars 514 forks source link

Update to BigQuery read API v1 interface #5431

Open kmjung opened 1 month ago

kmjung commented 1 month ago

This change updates Scio's usage of the BigQuery storage read API to use the v1 interface rather than v1beta1, which will eventually be deprecated. The two interfaces are very similar, to the extent that this change is mostly string substitution.

I haven't yet been able to run end-to-end tests with this change. If there is no automated test coverage as part of the PR approval process, then I will figure out how to do so.

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.

Project coverage is 61.13%. Comparing base (b52036a) to head (31fcc0d). Report is 6 commits behind head on main.

Files Patch % Lines
...la/com/spotify/scio/bigquery/client/TableOps.scala 0.00% 4 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #5431 +/- ## ========================================== + Coverage 61.09% 61.13% +0.03% ========================================== Files 310 311 +1 Lines 11060 11073 +13 Branches 769 771 +2 ========================================== + Hits 6757 6769 +12 - Misses 4303 4304 +1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

RustedBones commented 1 month ago

Thanks for the heads up! We completely missed this upgrade. CI fails with some expected binary compatibility issue. We'll plan this upgrade in next 0.15 release

kmjung commented 1 month ago

Thanks for the heads up! We completely missed this upgrade.

To be clear, you didn't miss anything per se -- the official deprecation notice hasn't been sent out yet (although this is coming). However, Scio is one of the last read API use cases that I'm aware of which is in wide use and has yet to migrate to v1. If we can start the adoption process for the newer API surface now, the deprecation process will hopefully be less impactful for users.