search

googleapis / python-bigquery-magics

https://googleapis.dev/python/bigquery-magics/latest/
Apache License 2.0
2 stars 1 forks source link

migrate to `query_and_wait` for lower-latency small queries #15

Open tswast opened 4 months ago

tswast commented 4 months ago

Similar to https://github.com/googleapis/python-bigquery-pandas/pull/722, let's use query_and_wait to issue queries.

Note: We may want to update query_and_wait to support progress bars on queries that last longer than the first call to jobs.query before doing this.

Thought: I wonder if migrating to use pandas-gbq to issue queries would make the most sense, given the desire to reduce redundancies?

tswast commented 4 months ago

Using read_gbq might also make #16 easier as well, since both code paths could just call the corresponding read_gbq method.

tswast commented 2 months ago

Progress bar is the hardest part. Maybe we add an option to query_and_wait that outputs a link to the job if the query doesn't finish in the first jobs.query API call?

https://github.com/googleapis/python-bigquery-dataframes/blob/014765c22410a0b4559896d163c440f46f7ce98f/bigframes/formatting_helpers.py#L233

for this job linking logic in BigQuery DataFrames.