idabjorlig / google-bigquery-tools

Automatically exported from code.google.com/p/google-bigquery-tools
0 stars 0 forks source link

Query default max_rows of sys.maxint does not work on 64bit python #14

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago
What steps will reproduce the problem?
1. job = client.Query('SELECT mmsi FROM [strata_sq_pos.pos123] GROUP BY mmsi;')
2. dest_table = job['configuration']['query']['destinationTable']
3. fields, rows = client.ReadSchemaAndRows(dest_table)

What is the expected output? What do you see instead?

I would expect to get all the rows back.  I would expect the default parameter 
for max_rows to work.  Instead I get:

BigqueryServiceError: Invalid unsigned integer value: '9223372036854775807'.

max_rows=sys.maxint is not a good default on 64bit python.

What version of the product are you using? On what operating system?

bigquery 2.0.12 in a virtualenv installed with pip on Mac OSX 10.8.2 with 
python 2.7.3 from fink.

Please provide any additional information below.

Original issue reported on code.google.com by schwehr@google.com on 23 Feb 2013 at 8:31

GoogleCodeExporter commented 8 years ago
Still failing.

job = client.Query('SELECT mmsi FROM [io.pos123] GROUP BY mmsi;')
dest_table = job['configuration']['query']['destinationTable']
print job['statistics']
fields, rows = client.ReadSchemaAndRows(dest_table)
len(rows)

Gives:

Waiting on job_c3b56f50e79b46628d2fbdf933ee60f3 ... (0s) Current status: DONE   
---------------------------------------------------------------------------
BigqueryServiceError                      Traceback (most recent call last)
<ipython-input-34-69b4e3258639> in <module>()
      2 dest_table = job['configuration']['query']['destinationTable']
      3 print job['statistics']
----> 4 fields, rows = client.ReadSchemaAndRows(dest_table)
      5 len(rows)

/Users/schwehr/Desktop/io/io-ships/notebooks/ve/lib/python2.7/site-packages/bigq
uery_client.pyc in ReadSchemaAndRows(self, table_dict, max_rows)
    666     """
    667     return (self.GetTableSchema(table_dict).get('fields', []),
--> 668             self.ReadTableRows(table_dict, max_rows))
    669 
    670   @staticmethod

/Users/schwehr/Desktop/io/io-ships/notebooks/ve/lib/python2.7/site-packages/bigq
uery_client.pyc in ReadTableRows(self, table_dict, max_rows)
    638           maxResults=max_rows - len(rows),
    639           pageToken=page_token,
--> 640           **table_dict).execute()
    641       page_token = data.get('pageToken', None)
    642       max_rows = min(max_rows, int(data['totalRows']))

/Users/schwehr/Desktop/io/io-ships/notebooks/ve/lib/python2.7/site-packages/bigq
uery_client.pyc in execute(self, **kwds)
    278       self._model._log_response(e.resp, e.content)  # pylint:disable-msg=W0212
    279       if e.resp.get('content-type', '').startswith('application/json'):
--> 280         BigqueryClient.RaiseError(json.loads(e.content))
    281       else:
    282         raise BigqueryCommunicationError(

/Users/schwehr/Desktop/io/io-ships/notebooks/ve/lib/python2.7/site-packages/bigq
uery_client.pyc in RaiseError(result)
    716     """Raises an appropriate BigQuery error given the json error result."""
    717     error = result.get('error', {}).get('errors', [{}])[0]
--> 718     raise BigqueryError.Create(error, result, [])
    719 
    720   @staticmethod

BigqueryServiceError: Invalid unsigned integer value: '9223372036854775807'.

{u'query': {u'cacheHit': True, u'totalBytesProcessed': u'0'}, u'endTime': 
u'1367529734917', u'totalBytesProcessed': u'0', u'startTime': u'1367529734744'}

Original comment by schw...@gmail.com on 2 May 2013 at 9:23