add an "engine" parameter and context property to support bigframes DataFrame output

googleapis / python-bigquery-magics

https://googleapis.dev/python/bigquery-magics/latest/

Apache License 2.0

2 stars 1 forks source link

add an "engine" parameter and context property to support bigframes DataFrame output #16

Open tswast opened 4 months ago

tswast commented 4 months ago

Sub-tasks:

[ ] Add a bigquery_magics.context.engine option which defaults to "pandas" but can be set to "bigframes" for %%bigquery to return a BigQuery DataFrames object.
[ ] The default option can also be overridden by specifying --engine="bigframes" in the first line of the %%bigquery magics.

Googlers, see go/bf-magics design.

tswast commented 4 months ago

Benefits for why the user might want to support this:

Support queries that return > 10 GB of results.
Hopefully some day: more natural "async" support so as not to block the current thread while the query runs.

tswast commented 2 weeks ago

Another thought: We should use a code path that doesn't immediately run the query at read_gbq / read_gbq_query time (internal issue 347050555). Otherwise the magic would end up running at least two queries: the original query & one or two to visualize the results from __repr__.