sijocherian / google-bigquery

Automatically exported from code.google.com/p/google-bigquery
0 stars 0 forks source link

query cache with offset and limit #145

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago
It looks like bigquery query cache won't work for two query have different 
offset and limit.

for example:

SELECT items.campaign, user, count(*) c FROM table where (type = "track" OR type = "event") and user is not null and items.campaign is not null and items.campaign like "%:%:%" and items.campaign != "all:unknown" group each by items.campaign, user having c >= 1 limit 1000 offset 5273000

process 39270811501 bytes

SELECT items.campaign, user, count(*) c FROM table where (type = "track" OR type = "event") and user is not null and items.campaign is not null and items.campaign like "%:%:%" and items.campaign != "all:unknown" group each by items.campaign, user having c >= 1 limit 1000 offset 5274000


process another 39270811501 bytes

But I think these queries should be cacheable? 

Original issue reported on code.google.com by Luce...@gmail.com on 28 Aug 2014 at 7:15

GoogleCodeExporter commented 9 years ago
Unfortunately, the query cache does not operate at that layer of the query 
execution engine. The query cache simply reuses the output table for a query, 
and that table only contains the 1000 rows returned by the first query.

A better way to get the results you're looking for is to run the query without 
limit/offset (using "Allow Large Results" if needed), and then page through the 
output table using the tabledata.list API method. That approach will allow you 
to page through the output gradually (or jump to a specified row) efficiently.

An important advantage of this approach is that you will only have to pay for 
your query once, since tabledata.list is free. With the limit/offset approach, 
you will pay to scan the entire table for every query you run, which could get 
expensive if you're trying to fetch many chunks of a large table.

Original comment by jcon...@google.com on 28 Aug 2014 at 7:27