search

sijocherian / google-bigquery

Automatically exported from code.google.com/p/google-bigquery
0 stars 0 forks source link

save insert time stamp for each record #140

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
We found table decorator is a very helpful and must-have feature. But it only 
works when the data is in the cache. Really hope to have the time when data 
being streamed into Bigquery saved and be able to be seen and used in query.

Original issue reported on code.google.com by fionaz...@geotab.com on 19 Aug 2014 at 1:27

GoogleCodeExporter commented 9 years ago 
Hello, I'm glad table decorators are a useful feature for you.  I'd like to get 
some clarification about your request, though - are you asking that data added 
to a table via the streaming API be available for immediate querying via table 
decorators, or something else?

Original comment by thomasp...@google.com on 19 Aug 2014 at 11:58

GoogleCodeExporter commented 9 years ago 
well, in many cases, we need incremental queries (e.g. we have data summary
derived from tables in bigquery, as we are keeping streaming data in, we
need to update that data summary with new data), instead of querying on a
whole table, we want to query the latest part of data.
ᐧ

*Fiona Zhao*
Geotab Inc.
Big Data Analyst & Developer Intern

[Direct] +1 (647) 728-4484
[Toll-Free] +1 (877) 436-8221
[Visit] www.geotab.com
Twitter <http://twitter.com/geotab> | Facebook
<http://www.facebook.com/mygeotab> | YouTube
<http://www.youtube.com/mygeotab> | LinkedIn
<http://www.linkedin.com/company/102661?trk=tyah&trkInfo=tarId:1403199250031,tas
:geotab,idx:2-1-3>

Original comment by fionaz...@geotab.com on 20 Aug 2014 at 3:15

GoogleCodeExporter commented 9 years ago 
Let me loop in one of my colleagues to comment on latency and availability 
restrictions when streaming and table decorators are used together.

Original comment by thomasp...@google.com on 21 Aug 2014 at 12:31

GoogleCodeExporter commented 9 years ago 
We ran into the same issue. How are you dealing with the problem? Were you able 
to find a resolution?

Original comment by pentium10 on 18 Nov 2014 at 2:35

GoogleCodeExporter commented 9 years ago 
Query within half an hour  or so after streaming should be ok.
ᐧ

*Fiona Zhao*
Geotab Inc.
Big Data Analyst & Developer

[Direct] +1 (647) 728-4484
[Toll-Free] +1 (877) 436-8221
[Visit] www.geotab.com
Twitter <http://twitter.com/geotab> | Facebook
<http://www.facebook.com/mygeotab> | YouTube
<http://www.youtube.com/mygeotab> | LinkedIn
<http://www.linkedin.com/company/102661?trk=tyah&trkInfo=tarId:1403199250031,tas
:geotab,idx:2-1-3>

Original comment by fionaz...@geotab.com on 18 Nov 2014 at 2:43

GoogleCodeExporter commented 9 years ago 
and by the way how you managed to "update" the data summary with new data?

Original comment by pentium10 on 18 Nov 2014 at 2:45

GoogleCodeExporter commented 9 years ago 
query using table decorator, get query result and append it to the original
table, if that's what you asked.
ᐧ

*Fiona Zhao*
Geotab Inc.
Big Data Analyst & Developer

[Direct] +1 (647) 728-4484
[Toll-Free] +1 (877) 436-8221
[Visit] www.geotab.com
Twitter <http://twitter.com/geotab> | Facebook
<http://www.facebook.com/mygeotab> | YouTube
<http://www.youtube.com/mygeotab> | LinkedIn
<http://www.linkedin.com/company/102661?trk=tyah&trkInfo=tarId:1403199250031,tas
:geotab,idx:2-1-3>

Original comment by fionaz...@geotab.com on 18 Nov 2014 at 3:16

GoogleCodeExporter commented 9 years ago 
Issue 152 has been merged into this issue.

Original comment by jcon...@google.com on 23 Jun 2015 at 4:42