move-coop / parsons

A python library of connectors for the progressive community.
Other
255 stars 125 forks source link

GoogleBigQuery.query() fails if no select statement #1018

Closed cmdelrio closed 3 months ago

cmdelrio commented 3 months ago

Running a query like "drop table;" doesn't work

Detailed Description

Running this block of code would have work when using Redshift as db:

            db.query(
                f"""
                DROP TABLE IF EXISTS logs.{PREFIX}sync_log_{date}_old;
                CREATE TABLE logs.{PREFIX}sync_log_{date}_old AS
                    SELECT * FROM logs.{PREFIX}sync_log; 
                """
            )

But with BQ I get this traceback:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/cormac/Desktop/GitHub/parsons/parsons/google/google_bigquery.py", line 275, in query
    return self.query_with_connection(
  File "/Users/cormac/Desktop/GitHub/parsons/parsons/google/google_bigquery.py", line 321, in query_with_connection
    final_table = self._fetch_query_results(cursor=cursor)
  File "/Users/cormac/Desktop/GitHub/parsons/parsons/google/google_bigquery.py", line 1258, in _fetch_query_results
    batch = cursor.fetchmany(QUERY_BATCH_SIZE)
  File "/Users/cormac/Desktop/GitHub/canales/.venv/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/_helpers.py", line 494, in with_closed_check
    return method(self, *args, **kwargs)
  File "/Users/cormac/Desktop/GitHub/canales/.venv/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 360, in fetchmany
    self._try_fetch(size=size)
  File "/Users/cormac/Desktop/GitHub/canales/.venv/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 256, in _try_fetch
    rows_iterable = self._bqstorage_fetch(bqstorage_client)
  File "/Users/cormac/Desktop/GitHub/canales/.venv/lib/python3.9/site-packages/google/cloud/bigquery/dbapi/cursor.py", line 286, in _bqstorage_fetch
    table=table_reference.to_bqstorage(),
AttributeError: 'NoneType' object has no attribute 'to_bqstorage'

As a temporary workaround I just added a select statement to the end of the query:

            db.query(
                f"""
                DROP TABLE IF EXISTS logs.{PREFIX}sync_log_{date}_old;
                CREATE TABLE logs.{PREFIX}sync_log_{date}_old AS
                    SELECT * FROM logs.{PREFIX}sync_log; 
                select 'WOW SO NEAT'
                """
            )

Your Environment

Additional Context

Add any other context about the problem here.

Priority

low priority - easy workaround

IanRFerguson commented 3 months ago

did you try bq.query(query, return_results=False) @cmdelrio? I was having this problem in the fall so we updated the API

cmdelrio commented 3 months ago

Nope! That will probably do the trick. Thanks

austinweisgrau commented 3 months ago

Dupe with #999. This should be fixed on parsons 3.1.0 with 2617134627aec2f955702c7f14cddf14fe44f2ea (from #1000)