Open rebexxxxxx opened 8 months ago
First, I reviewed the python code written by @jacobmpeters which is run at the end of the flattening pipeline repo:
if run_now:
client = bigquery.Client(project=project)
query_job = client.query(query_string) # Make an API request.
print("Running query.. Check GCP to confirm when complete.")
# query_job.result() # Wait for the job to complete.
specifically in the python file: _~/flatteningRequests/gcp_query_tools/update_gcp_scheduled_queryb.py
we want to do something very similar at the end of the bq2 query generation pipeline, i.e. we want to execute the sql queries we just built in _/bq2/generate_bq2queries.qmd so that the queries generate/update their corresponding views in BQ2. it looks like bigrquery has a function which will allow us to submit a query to BQ: bq_project_query() or bq_dataset_query() [https://bigrquery.r-dbi.org/reference/bq_query.html](). The only thing im a little confused about is if these functions will allow us to update a view instead of a table. i will look into this more.
We need to be able to update views in BQ2 using the generated queries (generated by our generate_bq2_queries.qmd file). We aren't sure how to create/update views via bigrquery, please do some research about how this task can be completed.