search

googleapis / java-bigquery

Apache License 2.0
109 stars 119 forks source link

Need the new "unreachable" field to be supported by BigQuery::listDatasets API #3120

Open danielferraz-git opened 7 months ago

danielferraz-git commented 7 months ago

We are facing the same problem as issue googleapis/google-api-ruby-client#16987.

We need java API support for the new "unreachable" property in the BigQuery::listDatasets API response.

This is urgent, as this change will take place starting on Feb 15th, as announced by Google.

PhongChuong commented 7 months ago

FYI, we're currently discussing how to approach this internally. I'll post if there are any updates (ETA Feb [12, 13, 14] 2024).

sjmisterm commented 7 months ago

Thanks @PhongChuong .

Given we're getting an answer (and probably not a release) the earliest by Feb 12th, would it be possible for you to contact the BigQuery team and hold off this change while the community has time to adapt their software? The fact we're the only ones who submitted an issue shows a lot of lack of unawareness about the topic in general and certainly users won't upgrade timely and handle the new API results.

PhongChuong commented 7 months ago

Hi, Sorry for the delay. We're still in discussion on how to forward the unreachable information to the users in a manageable manner. I can pass the information to the BigQuery team to hold off this change. However, can you help me understand why that is needed? In particular, what is concern regarding enabling the feature.

sjmisterm commented 6 months ago

@PhongChuong , sorry for the delay as I was on vacation.

As stated by Google itself in the announcement sent to users:

"Today, in the uncommon case of one or more GCP regions being unreachable for a prolonged amount of time, datasets.list and jobs.list API calls will fail if the response should contain data from the unreachable region.

Starting February 15, 2024, these calls will succeed, with the additional field unreachable added to mark the locations in which data might have been skipped from dataset.list and jobs.list response in case of a prolonged unreachability of the entire region or multi-region.

Under normal circumstances, the field will be empty. However, if you rely on getting a full list of all your datasets and jobs from dataset.list and jobs.list API, you should start interpreting values of the new field if present."

Our use case - and most users, I guess - depends on this call being 100% reliable, i.e., either failing with an exception or returning all the datasets. Now an empty return in a region might mean "temporarily off" or no "datasets at all", which completely break the API.

As suggested by Google itself:

"Check the content of this field to see if there are any unreachable locations. If your API integration with BigQuery depends on getting a full list of datasets/jobs from all regions/multi-regions for which relevant data exists, make sure to adjust it depending on specific circumstances.

You may choose to fail the workflow, issue a warning, or continue without warning in case an unreachable field is populated."

We must fail the workflow given our use case - and I guess that's what most users want to do -, but currently there is no way to find out the empty response is due to an unreachable location and to do so.

PhongChuong commented 6 months ago

@sjmisterm, thanks for the context. We'll add the unreachable field to this library. I'll update with an ETA soon.

sjmisterm commented 6 months ago

Hi @PhongChuong ,

It's been a while since the last message. Is there an ETA already?

PhongChuong commented 5 months ago

Sorry for the delay. We're still trying to find staffing to work on this issue. I'll update as soon as we have a clear ETA.

sjmisterm commented 4 months ago

Hi @PhongChuong . It's been over 3 months since this change has been deployed and any partial results delivered by the API are wrongly assumed to be the whole data by all Java clients.

I understand you're not the one to blame, but this can potentially cause us a major problem at any time. Does Google really intend to fix this API issue or to revert this backend change?

sjmisterm commented 1 month ago

Hi @PhongChuong . Just pinging you again since it's been over three months from your last message and my previous one was ignored. Any news?

PhongChuong commented 1 month ago

Hi, @sjmisterm , Sorry. I should have been more on top of this issue and keep everyone posted. Unfortunately, my original proposal for Unreachable field fell through. The issue is that listDatasets and listJobs returns Page of dataset, job respectively which requires a longer term solution to how we handle additional fields in a paginated resource. As a result, this change is not exclusive to BigQuery.

I'll bring this issue up again to see if we can raise the priority.

Meanwhile, if you have a support contract with Google, please create this issue in the support console will help us prioritize the effort.