search

google-code-export / gdata-python-client

Automatically exported from code.google.com/p/gdata-python-client
1 stars 0 forks source link

gdata.analytics - lacks multithreading support #471

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
gdata.analytics - lacks multithreading support

Original issue reported on code.google.com by pikhovkin@gmail.com on 3 Dec 2010 at 12:31

GoogleCodeExporter commented 9 years ago 
Could you provide more detail?

Original comment by joe.gregorio@gmail.com on 3 Dec 2010 at 12:46

GoogleCodeExporter commented 9 years ago 
I create 4 threads, each thread listens to a queue and receives from her ga: 
tableId. Once the flow gets his ga: tableId, he gets all the pages to the site 
the previous day, then all the keywords for those pages, ie gets something like 
an object {'page_path': {'keyword1': {'bounces': 0, 'exits': 0 ...}, ...}, ...}
The problem is that some keywords for a particular path (most often it is the 
path "/") are in another thread. This means that the Key words which must be 
received in the first thread turn into an object that belongs to the second 
thread.

In the end everything is mixed, BUT the stream are isolated from each other.

My manual traceback:

Thread2 get data from queue: ['ga:13538458', 'moda.nesterova.ru', False]
Thread4 get data from queue: ['ga:33075170', 'stal-kovka.ru', False]
Thread1 get data from queue: ['ga:24424890', 'belorusof.ru', False]
Thread3 get data from queue: ['ga:32585455', 'undorstroy.ru', False]

Thread4: sent request = {'metrics': 'ga:pageviews', 'sort': '-ga:pageviews', 
'max-results': '5000', 'dimensions': 'ga:pagePath', 'filters': 'ga:visits>0', 
'start-date': '2010-12-02', 'start-index': '1', 'end-date': '2010-12-02', 
'ids': 'ga:32585455'}
Thread4: response (feed) = <ns0:feed ns1:etag="W/"DkQNSHcycSp7I2A9Wx9SE04."" 
ns1:kind="analytics#data" xmlns:ns0="http://www.w3.org/2005/Atom" 
xmlns:ns1="http://schemas.google.com/g/2005"><ns0:id>http://www.google.com/analy
tics/feeds/data?ids=ga:33075170&dimensions=ga:pagePath&metrics=ga:pageviews&filt
ers=ga:visits%3E0&start-date=2010-12-02&end-date=2010-12-02</ns0:id><ns2:dataSou
rce 
xmlns:ns2="http://schemas.google.com/analytics/2009"><ns2:tableName>stal-kovka.r
u</ns2:tableName><ns2:tableId>ga:33075170</ns2:tableId><ns2:property 
name="ga:profileId" value="33075170" /><ns2:property name="ga:webPropertyId" 
value="UA-9496597-38" /><ns2:property name="ga:accountName" value="s-t-u" 
/></ns2:dataSource><ns2:aggregates 
xmlns:ns2="http://schemas.google.com/analytics/2009"><ns2:metric 
confidenceInterval="0.0" name="ga:pageviews" type="integer" value="0" 
/></ns2:aggregates><ns0:updated>2010-12-02T15:59:59.999-08:00</ns0:updated><ns2:
containsSampledData 
xmlns:ns2="http://schemas.google.com/analytics/2009">false</ns2:containsSampledD
ata><ns0:generator version="1.0">Google 
Analytics</ns0:generator><ns2:startIndex 
xmlns:ns2="http://a9.com/-/spec/opensearchrss/1.0/">1</ns2:startIndex><ns0:title
>Google Analytics Data for Profile 
33075170</ns0:title><ns0:author><ns0:name>Google 
Analytics</ns0:name></ns0:author><ns0:link 
href="https://www.google.com/analytics/feeds/data?start-index=1&max-results=5000
&sort=-ga%3Apageviews&end-date=2010-12-02&start-date=2010-12-02&metrics=ga%3Apag
eviews&ids=ga%3A33075170&dimensions=ga%3ApagePath&filters=ga%3Avisits%3E0" 
rel="self" type="application/atom+xml" /><ns2:endDate 
xmlns:ns2="http://schemas.google.com/analytics/2009">2010-12-02</ns2:endDate><ns
2:totalResults 
xmlns:ns2="http://a9.com/-/spec/opensearchrss/1.0/">0</ns2:totalResults><ns2:sta
rtDate 
xmlns:ns2="http://schemas.google.com/analytics/2009">2010-12-02</ns2:startDate><
ns2:itemsPerPage 
xmlns:ns2="http://a9.com/-/spec/opensearchrss/1.0/">5000</ns2:itemsPerPage></ns0
:feed>
Thread4: result object = {}
Thread4 get data from queue: ['ga:17030176', 'visionvip.biz', False]

Thread1: sent request = {'metrics': 'ga:pageviews', 'sort': '-ga:pageviews', 
'max-results': '5000', 'dimensions': 'ga:pagePath', 'filters': 'ga:visits>0', 
'start-date': '2010-12-02', 'start-index': '1', 'end-date': '2010-12-02', 
'ids': 'ga:17030176'}
Thread1: response (feed) = <ns0:feed ns1:etag="W/"DkQNSHcycSp7I2A9Wx9SE04."" 
ns1:kind="analytics#data" xmlns:ns0="http://www.w3.org/2005/Atom" 
xmlns:ns1="http://schemas.google.com/g/2005"><ns0:id>http://www.google.com/analy
tics/feeds/data?ids=ga:24424890&dimensions=ga:pagePath&metrics=ga:pageviews&filt
ers=ga:visits%3E0&start-date=2010-12-02&end-date=2010-12-02</ns0:id><ns2:dataSou
rce 
xmlns:ns2="http://schemas.google.com/analytics/2009"><ns2:tableName>www.beloruso
f.ru</ns2:tableName><ns2:tableId>ga:24424890</ns2:tableId><ns2:property 
name="ga:profileId" value="24424890" /><ns2:property name="ga:webPropertyId" 
value="UA-9443824-27" /><ns2:property name="ga:accountName" value="a-b-c" 
/></ns2:dataSource><ns2:aggregates 
xmlns:ns2="http://schemas.google.com/analytics/2009"><ns2:metric 
confidenceInterval="0.0" name="ga:pageviews" type="integer" value="48" 
/></ns2:aggregates><ns0:updated>2010-12-02T15:59:59.999-08:00</ns0:updated><ns2:
containsSampledData 
xmlns:ns2="http://schemas.google.com/analytics/2009">false</ns2:containsSampledD
ata><ns0:entry ns1:etag="W/"AkcEQX47eSp7I2A9Wx9SEkk."" 
ns1:kind="analytics#datarow"><ns0:id>http://www.google.com/analytics/feeds/data?
ids=ga:24424890&ga:pagePath=/index.php?page%3Dtseny&filters=ga:visits%3E0&start-
date=2010-12-02&end-date=2010-12-02</ns0:id><ns0:updated>2010-12-01T16:00:00.001
-08:00</ns0:updated><ns2:dimension name="ga:pagePath" 
value="/index.php?page=tseny" 
xmlns:ns2="http://schemas.google.com/analytics/2009" 
/><ns0:title>ga:pagePath=/index.php?page=tseny</ns0:title><ns2:metric 
confidenceInterval="0.0" name="ga:pageviews" type="integer" value="16" 
xmlns:ns2="http://schemas.google.com/analytics/2009" /><ns0:link 
href="http://www.google.com/analytics" rel="alternate" type="text/html" 
/></ns0:entry><ns0:entry ns1:etag="W/"AkcEQX47eSp7I2A9Wx9SEkk."" 
ns1:kind="analytics#datarow"><ns0:id>http://www.google.com/analytics/feeds/data?
ids=ga:24424890&ga:pagePath=/index.php?page%3Dtehnologiya-ustrojstva-styazhki-po
lov&filters=ga:visits%3E0&start-date=2010-12-02&end-date=2010-12-02</ns0:id><ns0
:updated>2010-12-01T16:00:00.001-08:00</ns0:updated><ns2:dimension 
name="ga:pagePath" 
value="/index.php?page=tehnologiya-ustrojstva-styazhki-polov" 
xmlns:ns2="http://schemas.google.com/analytics/2009" 
/><ns0:title>ga:pagePath=/index.php?page=tehnologiya-ustrojstva-styazhki-polov</
ns0:title><ns2:metric confidenceInterval="0.0" name="ga:pageviews" 
type="integer" value="15" xmlns:ns2="http://schemas.google.com/analytics/2009" 
/><ns0:link href="http://www.google.com/analytics" rel="alternate" 
type="text/html" /></ns0:entry><ns0:entry 
ns1:etag="W/"AkcEQX47eSp7I2A9Wx9SEkk."" 
ns1:kind="analytics#datarow"><ns0:id>http://www.google.com/analytics/feeds/data?
ids=ga:24424890&ga:pagePath=/&filters=ga:visits%3E0&start-date=2010-12-02&end-da
te=2010-12-02</ns0:id><ns0:updated>2010-12-01T16:00:00.001-08:00</ns0:updated><n
s2:dimension name="ga:pagePath" value="/" 
xmlns:ns2="http://schemas.google.com/analytics/2009" 
/><ns0:title>ga:pagePath=/</ns0:title><ns2:metric confidenceInterval="0.0" 
name="ga:pageviews" type="integer" value="8" 
xmlns:ns2="http://schemas.google.com/analytics/2009" /><ns0:link 
href="http://www.google.com/analytics" rel="alternate" type="text/html" 
/></ns0:entry><ns0:entry ns1:etag="W/"AkcEQX47eSp7I2A9Wx9SEkk."" 
ns1:kind="analytics#datarow"><ns0:id>http://www.google.com/analytics/feeds/data?
ids=ga:24424890&ga:pagePath=/index.php?page%3Dkontakty&filters=ga:visits%3E0&sta
rt-date=2010-12-02&end-date=2010-12-02</ns0:id><ns0:updated>2010-12-01T16:00:00.
001-08:00</ns0:updated><ns2:dimension name="ga:pagePath" 
value="/index.php?page=kontakty" 
xmlns:ns2="http://schemas.google.com/analytics/2009" 
/><ns0:title>ga:pagePath=/index.php?page=kontakty</ns0:title><ns2:metric 
confidenceInterval="0.0" name="ga:pageviews" type="integer" value="4" 
xmlns:ns2="http://schemas.google.com/analytics/2009" /><ns0:link 
href="http://www.google.com/analytics" rel="alternate" type="text/html" 
/></ns0:entry><ns0:entry ns1:etag="W/"AkcEQX47eSp7I2A9Wx9SEkk."" 
ns1:kind="analytics#datarow"><ns0:id>http://www.google.com/analytics/feeds/data?
ids=ga:24424890&ga:pagePath=/index.php?page%3Dustrojstvo-styazhki-pola&filters=g
a:visits%3E0&start-date=2010-12-02&end-date=2010-12-02</ns0:id><ns0:updated>2010
-12-01T16:00:00.001-08:00</ns0:updated><ns2:dimension name="ga:pagePath" 
value="/index.php?page=ustrojstvo-styazhki-pola" 
xmlns:ns2="http://schemas.google.com/analytics/2009" 
/><ns0:title>ga:pagePath=/index.php?page=ustrojstvo-styazhki-pola</ns0:title><ns
2:metric confidenceInterval="0.0" name="ga:pageviews" type="integer" value="3" 
xmlns:ns2="http://schemas.google.com/analytics/2009" /><ns0:link 
href="http://www.google.com/analytics" rel="alternate" type="text/html" 
/></ns0:entry><ns0:entry ns1:etag="W/"AkcEQX47eSp7I2A9Wx9SEkk."" 
ns1:kind="analytics#datarow"><ns0:id>http://www.google.com/analytics/feeds/data?
ids=ga:24424890&ga:pagePath=/index.php?page%3Dstyazhka-polov&filters=ga:visits%3
E0&start-date=2010-12-02&end-date=2010-12-02</ns0:id><ns0:updated>2010-12-01T16:
00:00.001-08:00</ns0:updated><ns2:dimension name="ga:pagePath" 
value="/index.php?page=styazhka-polov" 
xmlns:ns2="http://schemas.google.com/analytics/2009" 
/><ns0:title>ga:pagePath=/index.php?page=styazhka-polov</ns0:title><ns2:metric 
confidenceInterval="0.0" name="ga:pageviews" type="integer" value="2" 
xmlns:ns2="http://schemas.google.com/analytics/2009" /><ns0:link 
href="http://www.google.com/analytics" rel="alternate" type="text/html" 
/></ns0:entry><ns0:generator version="1.0">Google 
Analytics</ns0:generator><ns2:startIndex 
xmlns:ns2="http://a9.com/-/spec/opensearchrss/1.0/">1</ns2:startIndex><ns0:title
>Google Analytics Data for Profile 
24424890</ns0:title><ns0:author><ns0:name>Google 
Analytics</ns0:name></ns0:author><ns0:link 
href="https://www.google.com/analytics/feeds/data?start-index=1&max-results=5000
&sort=-ga%3Apageviews&end-date=2010-12-02&start-date=2010-12-02&metrics=ga%3Apag
eviews&ids=ga%3A24424890&dimensions=ga%3ApagePath&filters=ga%3Avisits%3E0" 
rel="self" type="application/atom+xml" /><ns2:endDate 
xmlns:ns2="http://schemas.google.com/analytics/2009">2010-12-02</ns2:endDate><ns
2:totalResults 
xmlns:ns2="http://a9.com/-/spec/opensearchrss/1.0/">6</ns2:totalResults><ns2:sta
rtDate 
xmlns:ns2="http://schemas.google.com/analytics/2009">2010-12-02</ns2:startDate><
ns2:itemsPerPage 
xmlns:ns2="http://a9.com/-/spec/opensearchrss/1.0/">5000</ns2:itemsPerPage></ns0
:feed>
Thread2: sent request = {'metrics': 'ga:pageviews', 'sort': '-ga:pageviews', 
'max-results': '5000', 'dimensions': 'ga:pagePath', 'filters': 'ga:visits>0', 
'start-date': '2010-12-02', 'start-index': '1', 'end-date': '2010-12-02', 
'ids': 'ga:17030176'}
Thread2: response (feed) = <ns0:feed ns1:etag="W/"DkQNSHcycSp7I2A9Wx9SE04."" 
ns1:kind="analytics#data" xmlns:ns0="http://www.w3.org/2005/Atom" 
xmlns:ns1="http://schemas.google.com/g/2005"><ns0:id>http://www.google.com/analy
tics/feeds/data?ids=ga:13538458&dimensions=ga:pagePath&metrics=ga:pageviews&filt
ers=ga:visits%3E0&start-date=2010-12-02&end-date=2010-12-02</ns0:id><ns2:dataSou
rce 
xmlns:ns2="http://schemas.google.com/analytics/2009"><ns2:tableName>moda.nestero
va.ru</ns2:tableName><ns2:tableId>ga:13538458</ns2:tableId><ns2:property 
name="ga:profileId" value="13538458" /><ns2:property name="ga:webPropertyId" 
value="UA-4618787-59" /><ns2:property name="ga:accountName" value="_ALTEGroup 
2" /></ns2:dataSource><ns2:aggregates 
xmlns:ns2="http://schemas.google.com/analytics/2009"><ns2:metric 
confidenceInterval="0.0" name="ga:pageviews" type="integer" value="0" 
/></ns2:aggregates><ns0:updated>2010-12-02T15:59:59.999-08:00</ns0:updated><ns2:
containsSampledData 
xmlns:ns2="http://schemas.google.com/analytics/2009">false</ns2:containsSampledD
ata><ns0:generator version="1.0">Google 
Analytics</ns0:generator><ns2:startIndex 
xmlns:ns2="http://a9.com/-/spec/opensearchrss/1.0/">1</ns2:startIndex><ns0:title
>Google Analytics Data for Profile 
13538458</ns0:title><ns0:author><ns0:name>Google 
Analytics</ns0:name></ns0:author><ns0:link 
href="https://www.google.com/analytics/feeds/data?start-index=1&max-results=5000
&sort=-ga%3Apageviews&end-date=2010-12-02&start-date=2010-12-02&metrics=ga%3Apag
eviews&ids=ga%3A13538458&dimensions=ga%3ApagePath&filters=ga%3Avisits%3E0" 
rel="self" type="application/atom+xml" /><ns2:endDate 
xmlns:ns2="http://schemas.google.com/analytics/2009">2010-12-02</ns2:endDate><ns
2:totalResults 
xmlns:ns2="http://a9.com/-/spec/opensearchrss/1.0/">0</ns2:totalResults><ns2:sta
rtDate 
xmlns:ns2="http://schemas.google.com/analytics/2009">2010-12-02</ns2:startDate><
ns2:itemsPerPage 
xmlns:ns2="http://a9.com/-/spec/opensearchrss/1.0/">5000</ns2:itemsPerPage></ns0
:feed>
Thread2: result object = {}
Thread2 get data from queue: ['ga:11029900', 'st-interior.ru', False]
...
I think that's enough

Original comment by pikhovkin@gmail.com on 3 Dec 2010 at 3:38

GoogleCodeExporter commented 9 years ago 
I can send the source code of the script

Original comment by pikhovkin@gmail.com on 3 Dec 2010 at 3:46

GoogleCodeExporter commented 9 years ago 
The library is not written to be thread safe, and would require substantial 
work to be made thread safe.

Original comment by joe.gregorio@gmail.com on 24 Jan 2011 at 6:00