ngulam-ai / Sherlock

0 stars 0 forks source link

[URG] Data incorrectly stored in Bigquery #51

Closed ngulamai closed 6 years ago

ngulamai commented 6 years ago

Hi Alexander, Daniel We have reviewed the way that each hit is stored in Bigquery, and found that there is something wrong with the serverlet that incorrectly stores the information inside bigquery

Please take the example from below, extracted from yesterday streaming base As you can see, despite the fact that we are calling the endpoint with datapoints such as: cd4=android cd5=MTN%20SA

Those parameters are not stored in the corresponding custom dimension. As you can see below, we are also seinding in dl the entire call, so you can see that indeed the endpoint was correctly called, but then incorrectly stored. Can you please fix it? Thanks

{"jid":"808123708","bf":"Android WebView","gjid":"2028265638","cd96":"2018","cd94":"Monday","cd95":"08","cd92":"21:59:57.894","cd93":"27","cd91":"Africa/Johannesburg","bv":"4.0","uip":"41.113.66.166","gtm":"G86P3S77DX","_gid":"966668281.1535399997","pr1nm":"ZA - 100735 - Guessing Game","je":"0","dm":"general Mobile Phone","cd29":"363","cd28":"Chrome","cd25":"65","cd30":"Johannesburg","sd":"32-bit","uid":"South Africa;Gauteng;Johannesburg;null;null;Android;6.0;Mozilla/5.0 (Linux; Android 6.0; Azumi_KIREI_A45_D Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/55.0.2883.91 Mobile Safari/537.36;41.113.66.166;null;null;null","pr1ca":"Gaming","sr":"320x570","a":"83857328","cd18":"234053812.1535399997_1535399996778","cm":"Pop","cd12":"Avazu MDSP","istc":true,"dc":"Mobile Phone","t":"event","v":"1","z":"208011576","date":"20180827","de":"UTF-8","dl":"https://n38adshostnet.com/ads?key=1b617172b554ee6125add931473411c8&ec=AdClick&ea=Click&cd4=android&cd5=MTN%20SA&cd7=null&cd8=null&cd9=1535399992&cd10=null&cd13=adventurefeeds&cd14=null&cd15=site&cd16=106539_124860_6615-ca13293e17d17548967412fc00fa08cf&cd17=com.mediatek.factorymode&cd20=Android%206.0&cd21=null&cd22=com.mediatek.factorymode&dh=com.mediatek.factorymode&cd24=0x0&cd29=319275&ci=319275&cd34=ZA&cd36=generic&cd38=null&cd40=null&cd44=-26.230896%2C28.058304&cd49=41.113.66.166&uip=41.113.66.166&cd53=-26.230896&cd55=28.058304&cd59=6.0&cd60=null&cd65=null&cd68=null&cd74=null&cd76=Mozilla%2F5.0%20%28Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36&ua=Mozilla%2F5.0%20%28Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36&cd77=null&cd80=11945135399471972&cd90=LCsgNaseyPa12BW&cd98=cpc&cc=1296967&cm=Pop&cs=106539_124860_6615-ca13293e17d17548967412fc00fa08cf&uid=ZA;MTN%20SA;generic;android;6.0;Mozilla%2F5.0%20%28Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36;41.113.66.166;null&cd12=Avazu%20MDSP&cn=319275","cd48":"41.113.66.166","cd45":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","ua":"Mozilla/5.0 (Linux; Android 6.0; Azumi_KIREI_A45_D Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/55.0.2883.91 Mobile Safari/537.36","cd46":"en-US","tid":"UA-109728168-4","dr":"http://track.opticks.io/g/9709085df5160d6","cd52":"-26.20227","cd50":"en","pr1id":"363","ul":"en-us","of":"Android","ea":"Click","ec":"AdClick","pr1br":"Kimia","cd32":"c4b88080-aa33-11e8-aa44-d48564c62f44","cd33":"South Africa","vp":"","ni":"0","cid":"234053812.1535399997","cd69":"Gauteng","cd67":"http://track.opticks.io/g/9709085df5160d6","cd66":"track.opticks.io","cd75":"Mozilla/5.0 (Linux; Android 6.0; Azumi_KIREI_A45_D Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/55.0.2883.91 Mobile Safari/537.36","cd73":"1535399995530","hour":"21","hitId":"9529976a-661f-49fd-8716-b1c512cb20a9","cd58":"6.0","ov":"6.0","cd56":"Android","cd54":"28.04363","minute":"59","cd2":"Kimia","cd63":"https","cd1":"South Africa;Gauteng;Johannesburg;null;null;Android;6.0;Mozilla/5.0 (Linux; Android 6.0; Azumi_KIREI_A45_D Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/55.0.2883.91 Mobile Safari/537.36;41.113.66.166;null;null;null","pa":"click","cd61":"622","_r":"1","cd3":"ZA - 100735 - Guessing Game","cd62":"Kimia - ZA - 100735 - Guessing Game","_s":"1","cd6":"MjEjMzYzIzY1IzYyMnw5ODR8WkF8M3w1fHx8dmhlMjZia2Jrb3lpfGM0Yjg4MDgwLWFhMzMtMTFlOC1hYTQ0LWQ0ODU2NGM2MmY0NHx8Y29tLm1lZGlhdGVrLmZhY3Rvcnltb2Rl","_u":"aGBAAEAL~","_v":"j68","time":"1535407197894","__isb":false}

akolchin-MM commented 6 years ago

Hello,

We already have discussed it (you can find it in our Upwork conversations).

Yes, I can see these custom dimensions as parameters in the dl parameter but as well don't see them as the parameters in your call to the /collec servlet itself. This can mean only that you don't send them as parameters to it.

I don't know what to add to it.

I know that problem are not in the /collect servlet and further pipeline, probably somewhere in GTM or even before it. I can try to help with it but I need more details and access to this GTM first of all. Then I need to be able to reproduce and debug the situation in which these events occur.

ngulamai commented 6 years ago

Hi Alexander If you use the above link with the given parameters (https://n38adshostnet.com/ads?key=1b617172b554ee6125add931473411c8&ec=AdClick&ea=Click&cd4=android&cd5=MTN%20SA&cd7=null&cd8=null&cd9=1535399992&cd10=null&cd13=adventurefeeds&cd14=null&cd15=site&cd16=106539_124860_6615-ca13293e17d17548967412fc00fa08cf&cd17=com.mediatek.factorymode&cd20=Android%206.0&cd21=null&cd22=com.mediatek.factorymode&dh=com.mediatek.factorymode&cd24=0x0&cd29=319275&ci=319275&cd34=ZA&cd36=generic&cd38=null&cd40=null&cd44=-26.230896%2C28.058304&cd49=41.113.66.166&uip=41.113.66.166&cd53=-26.230896&cd55=28.058304&cd59=6.0&cd60=null&cd65=null&cd68=null&cd74=null&cd76=Mozilla%2F5.0%20%28Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36&ua=Mozilla%2F5.0%20%28Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36&cd77=null&cd80=11945135399471972&cd90=LCsgNaseyPa12BW&cd98=cpc&cc=1296967&cm=Pop&cs=106539_124860_6615-ca13293e17d17548967412fc00fa08cf&uid=ZA;MTN%20SA;generic;android;6.0;Mozilla%2F5.0%20%28Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36;41.113.66.166;null&cd12=Avazu%20MDSP&cn=319275)

and you debud the call on a browser, you will see that the collect endpoint is called with this payload:

v=1&_v=j68&a=1161255330&t=event&ni=0&_s=1&dl=https%3A%2F%2Fn38adshostnet.com%2Fads%3Fkey%3D1b617172b554ee6125add931473411c8%26ec%3DAdClick%26ea%3DClick%26cd4%3Dandroid%26cd5%3DMTN%2520SA%26cd7%3Dnull%26cd8%3Dnull%26cd9%3D1535399992%26cd10%3Dnull%26cd13%3Dadventurefeeds%26cd14%3Dnull%26cd15%3Dsite%26cd16%3D106539_124860_6615-ca13293e17d17548967412fc00fa08cf%26cd17%3Dcom.mediatek.factorymode%26cd20%3DAndroid%25206.0%26cd21%3Dnull%26cd22%3Dcom.mediatek.factorymode%26dh%3Dcom.mediatek.factorymode%26cd24%3D0x0%26cd29%3D319275%26ci%3D319275%26cd34%3DZA%26cd36%3Dgeneric%26cd38%3Dnull%26cd40%3Dnull%26cd44%3D-26.230896%252C28.058304%26cd49%3D41.113.66.166%26uip%3D41.113.66.166%26cd53%3D-26.230896%26cd55%3D28.058304%26cd59%3D6.0%26cd60%3Dnull%26cd65%3Dnull%26cd68%3Dnull%26cd74%3Dnull%26cd76%3DMozilla%252F5.0%2520%2528Linux%253B%2520Android%25206.0%253B%2520Azumi_KIREI_A45_D%2520Build%252FMRA58K%253B%2520wv%2529%2520AppleWebKit%252F537.36%2520%2528KHTML%252C%2520like%2520Gecko%2529%2520Version%252F4.0%2520Chrome%252F55.0.2883.91%2520Mobile%2520Safari%252F537.36%26ua%3DMozilla%252F5.0%2520%2528Linux%253B%2520Android%25206.0%253B%2520Azumi_KIREI_A45_D%2520Build%252FMRA58K%253B%2520wv%2529%2520AppleWebKit%252F537.36%2520%2528KHTML%252C%2520like%2520Gecko%2529%2520Version%252F4.0%2520Chrome%252F55.0.2883.91%2520Mobile%2520Safari%252F537.36%26cd77%3Dnull%26cd80%3D11945135399471972%26cd90%3DLCsgNaseyPa12BW%26cd98%3Dcpc%26cc%3D1296967%26cm%3DPop%26cs%3D106539_124860_6615-ca13293e17d17548967412fc00fa08cf%26uid%3DZA%3BMTN%2520SA%3Bgeneric%3Bandroid%3B6.0%3BMozilla%252F5.0%2520%2528Linux%253B%2520Android%25206.0%253B%2520Azumi_KIREI_A45_D%2520Build%252FMRA58K%253B%2520wv%2529%2520AppleWebKit%252F537.36%2520%2528KHTML%252C%2520like%2520Gecko%2529%2520Version%252F4.0%2520Chrome%252F55.0.2883.91%2520Mobile%2520Safari%252F537.36%3B41.113.66.166%3Bnull%26cd12%3DAvazu%2520MDSP%26cn%3D319275&ul=en-us&de=UTF-8&sd=24-bit&sr=1440x900&vp=645x734&je=0&cn=%24%24CUSTOM_PARAM(cn)%24%24&cs=%24%24CUSTOM_PARAM(cs)%24%24&cm=Pop&cc=%24%24CUSTOM_PARAM(cc)%24%24&ec=AdClick&ea=Click&_u=aGBAAEAL~&jid=2114188560&gjid=1267487959&cid=1453746889.1535541921&uid=Spain%3BPais%20Vasco%3BBilbao%3Bunknown%3Bunknown%3BAndroid%3B6.0%3BMozilla%2F5.0%20(Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36%3B212.81.201.234%3B%24%24CUSTOM_PARAM(cd11)%24%24%3B%24%24CUSTOM_PARAM(cd7)%24%24%3B%24%24CUSTOM_PARAM(cd10)%24%24&tid=UA-109728168-4&_gid=1317808649.1535541921&_r=1&gtm=G86P3S77DX&cd1=Spain%3BPais%20Vasco%3BBilbao%3Bunknown%3Bunknown%3BAndroid%3B6.0%3BMozilla%2F5.0%20(Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36%3B212.81.201.234%3B%24%24CUSTOM_PARAM(cd11)%24%24%3B%24%24CUSTOM_PARAM(cd7)%24%24%3B%24%24CUSTOM_PARAM(cd10)%24%24&cd2=Kimia&cd3=ZA%20-%20100735%20-%20Guessing%20Game&cd4=%24%24CUSTOM_PARAM(cd4)%24%24&cd5=%24%24CUSTOM_PARAM(cd5)%24%24&cd6=MjEjMzYzIzY1IzYyMnw5ODR8RVN8M3w1fHx8MjN3ZXQ3djZveGl2fDM2YmFmMDUwLWFiN2UtMTFlOC1iMmJhLWY4YmMxMjUzODFiOHx8&cd7=%24%24CUSTOM_PARAM(cd7)%24%24&cd8=%24%24CUSTOM_PARAM(cd8)%24%24&cd9=%24%24CUSTOM_PARAM(cd9)%24%24&cd10=%24%24CUSTOM_PARAM(cd10)%24%24&cd11=n%2Fa&cd12=Avazu%20MDSP&cd13=%24%24CUSTOM_PARAM(cd13)%24%24&cd14=%24%24CUSTOM_PARAM(cd14)%24%24&cd15=%24%24CUSTOM_PARAM(cd15)%24%24&cd16=%24%24CUSTOM_PARAM(cd16)%24%24&cd17=%24%24CUSTOM_PARAM(cd17)%24%24&cd18=1453746889.1535541921_1535541921326&cd20=%24%24CUSTOM_PARAM(cd20)%24%24&cd21=%24%24CUSTOM_PARAM(cd21)%24%24&cd22=%24%24CUSTOM_PARAM(cd22)%24%24&cd23=%24%24CUSTOM_PARAM(cd23)%24%24&cd24=%24%24CUSTOM_PARAM(cd24)%24%24&cd25=65&cd26=%24%24CUSTOM_PARAM(cd26)%24%24&cd28=Chrome&cd29=363&cd30=Bilbao&cd31=%24%24CUSTOM_PARAM(cd31)%24%24&cd32=36baf050-ab7e-11e8-b2ba-f8bc125381b8&cd33=Spain&cd34=%24%24CUSTOM_PARAM(cd34)%24%24&cd35=unknown&cd36=%24%24CUSTOM_PARAM(cd36)%24%24&cd37=%24%24CUSTOM_PARAM(cd37)%24%24&cd38=%24%24CUSTOM_PARAM(cd38)%24%24&cd39=%24%24CUSTOM_PARAM(cd39)%24%24&cd40=%24%24CUSTOM_PARAM(cd40)%24%24&cd41=%24%24CUSTOM_PARAM(cd41)%24%24&cd42=%24%24CUSTOM_PARAM(cd42)%24%24&cd43=%24%24CUSTOM_PARAM(cd43)%24%24&cd44=%24%24CUSTOM_PARAM(cd44)%24%24&cd45=text%2Fhtml%2Capplication%2Fxhtml%2Bxml%2Capplication%2Fxml%3Bq%3D0.9%2Cimage%2Fwebp%2Cimage%2Fapng%2C%2F%3Bq%3D0.8&cd46=en-US%2Cen%3Bq%3D0.9&cd47=%24%24CUSTOM_PARAM(cd47)%24%24&cd48=212.81.201.234&cd49=%24%24CUSTOM_PARAM(cd49)%24%24&cd50=en&cd51=%24%24CUSTOM_PARAM(cd51)%24%24&cd52=43.26271&cd53=%24%24CUSTOM_PARAM(cd53)%24%24&cd54=-2.92528&cd55=%24%24CUSTOM_PARAM(cd55)%24%24&cd56=Android&cd57=unknown&cd58=6.0&cd59=%24%24CUSTOM_PARAM(cd59)%24%24&cd60=%24%24CUSTOM_PARAM(cd60)%24%24&cd61=622&cd62=Kimia%20-%20ZA%20-%20100735%20-%20Guessing%20Game&cd63=https&cd64=%24%24CUSTOM_PARAM(cd64)%24%24&cd65=%24%24CUSTOM_PARAM(cd65)%24%24&cd66=&cd67=&cd68=%24%24CUSTOM_PARAM(cd68)%24%24&cd69=Pais%20Vasco&cd70=%24%24CUSTOM_PARAM(cd70)%24%24&cd71=%24%24CUSTOM_PARAM(cd71)%24%24&cd72=%24%24CUSTOM_PARAM(cd72)%24%24&cd73=1535541920726&cd74=%24%24CUSTOM_PARAM(cd74)%24%24&cd75=Mozilla%2F5.0%20(Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36&cd76=%24%24CUSTOM_PARAM(cd76)%24%24&cd77=%24%24CUSTOM_PARAM(cd77)%24%24&cd78=%24%24CUSTOM_PARAM(cd78)%24%24&cd80=%24%24CUSTOM_PARAM(cd80)%24%24&cd81=%24%24CUSTOM_PARAM(cd81)%24%24&cd82=%24%24CUSTOM_PARAM(cd82)%24%24&cd83=%24%24CUSTOM_PARAM(cd83)%24%24&cd84=%24%24CUSTOM_PARAM(cd84)%24%24&cd85=%24%24CUSTOM_PARAM(cd85)%24%24&cd86=%24%24CUSTOM_PARAM(cd86)%24%24&cd87=%24%24CUSTOM_PARAM(cd87)%24%24&cd88=%24%24CUSTOM_PARAM(cd88)%24%24&cd89=%24%24CUSTOM_PARAM(cd89)%24%24&cd90=%24%24CUSTOM_PARAM(cd90)%24%24&pa=click&pr1nm=ZA%20-%20100735%20-%20Guessing%20Game&pr1id=363&pr1ca=Gaming&pr1br=Kimia&z=945938446

To us this means that we seems to adding at least two times the same custom dimensions to the call

v=1

_v=j68 a=1161255330 t=event ni=0 _s=1 dl=https://n38adshostnet.com/ads?key=1b617172b554ee6125add931473411c8 ec=AdClick ea=Click cd4=android cd5=MTN%20SA cd7=null cd8=null cd9=1535399992 cd10=null cd13=adventurefeeds cd14=null cd15=site cd16=106539_124860_6615-ca13293e17d17548967412fc00fa08cf cd17=com.mediatek.factorymode cd20=Android%206.0 cd21=null cd22=com.mediatek.factorymode dh=com.mediatek.factorymode cd24=0x0 cd29=319275 ci=319275 cd34=ZA cd36=generic cd38=null cd40=null cd44=-26.230896%2C28.058304 cd49=41.113.66.166 uip=41.113.66.166 cd53=-26.230896 cd55=28.058304 cd59=6.0 cd60=null cd65=null cd68=null cd74=null cd76=Mozilla%2F5.0%20%28Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36 ua=Mozilla%2F5.0%20%28Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36 cd77=null cd80=11945135399471972 cd90=LCsgNaseyPa12BW cd98=cpc cc=1296967 cm=Pop cs=106539_124860_6615-ca13293e17d17548967412fc00fa08cf uid=ZA;MTN%20SA;generic;android;6.0;Mozilla%2F5.0%20%28Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36;41.113.66.166;null cd12=Avazu%20MDSP cn=319275 ul=en-us de=UTF-8 sd=24-bit sr=1440x900 vp=645x734 je=0 cn=$$CUSTOM_PARAM(cn)$$ cs=$$CUSTOM_PARAM(cs)$$ cm=Pop cc=$$CUSTOM_PARAM(cc)$$ ec=AdClick ea=Click _u=aGBAAEAL~ jid=2114188560 gjid=1267487959 cid=1453746889.1535541921 uid=Spain;Pais Vasco;Bilbao;unknown;unknown;Android;6.0;Mozilla/5.0 (Linux; Android 6.0; Azumi_KIREI_A45_D Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/55.0.2883.91 Mobile Safari/537.36;212.81.201.234;$$CUSTOM_PARAM(cd11)$$;$$CUSTOM_PARAM(cd7)$$;$$CUSTOM_PARAM(cd10)$$ tid=UA-109728168-4 _gid=1317808649.1535541921 _r=1 gtm=G86P3S77DX cd1=Spain;Pais Vasco;Bilbao;unknown;unknown;Android;6.0;Mozilla/5.0 (Linux; Android 6.0; Azumi_KIREI_A45_D Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/55.0.2883.91 Mobile Safari/537.36;212.81.201.234;$$CUSTOM_PARAM(cd11)$$;$$CUSTOM_PARAM(cd7)$$;$$CUSTOM_PARAM(cd10)$$ cd2=Kimia cd3=ZA - 100735 - Guessing Game cd4=$$CUSTOM_PARAM(cd4)$$ cd5=$$CUSTOM_PARAM(cd5)$$ cd6=MjEjMzYzIzY1IzYyMnw5ODR8RVN8M3w1fHx8MjN3ZXQ3djZveGl2fDM2YmFmMDUwLWFiN2UtMTFlOC1iMmJhLWY4YmMxMjUzODFiOHx8 cd7=$$CUSTOM_PARAM(cd7)$$ cd8=$$CUSTOM_PARAM(cd8)$$ cd9=$$CUSTOM_PARAM(cd9)$$ cd10=$$CUSTOM_PARAM(cd10)$$ cd11=n/a cd12=Avazu MDSP cd13=$$CUSTOM_PARAM(cd13)$$ cd14=$$CUSTOM_PARAM(cd14)$$ cd15=$$CUSTOM_PARAM(cd15)$$ cd16=$$CUSTOM_PARAM(cd16)$$ cd17=$$CUSTOM_PARAM(cd17)$$ cd18=1453746889.1535541921_1535541921326 cd20=$$CUSTOM_PARAM(cd20)$$ cd21=$$CUSTOM_PARAM(cd21)$$ cd22=$$CUSTOM_PARAM(cd22)$$ cd23=$$CUSTOM_PARAM(cd23)$$ cd24=$$CUSTOM_PARAM(cd24)$$ cd25=65 cd26=$$CUSTOM_PARAM(cd26)$$ cd28=Chrome cd29=363 cd30=Bilbao cd31=$$CUSTOM_PARAM(cd31)$$ cd32=36baf050-ab7e-11e8-b2ba-f8bc125381b8 cd33=Spain cd34=$$CUSTOM_PARAM(cd34)$$ cd35=unknown cd36=$$CUSTOM_PARAM(cd36)$$ cd37=$$CUSTOM_PARAM(cd37)$$ cd38=$$CUSTOM_PARAM(cd38)$$ cd39=$$CUSTOM_PARAM(cd39)$$ cd40=$$CUSTOM_PARAM(cd40)$$ cd41=$$CUSTOM_PARAM(cd41)$$ cd42=$$CUSTOM_PARAM(cd42)$$ cd43=$$CUSTOM_PARAM(cd43)$$ cd44=$$CUSTOM_PARAM(cd44)$$ cd45=text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8 cd46=en-US,en;q=0.9 cd47=$$CUSTOM_PARAM(cd47)$$ cd48=212.81.201.234 cd49=$$CUSTOM_PARAM(cd49)$$ cd50=en cd51=$$CUSTOM_PARAM(cd51)$$ cd52=43.26271 cd53=$$CUSTOM_PARAM(cd53)$$ cd54=-2.92528 cd55=$$CUSTOM_PARAM(cd55)$$ cd56=Android cd57=unknown cd58=6.0 cd59=$$CUSTOM_PARAM(cd59)$$ cd60=$$CUSTOM_PARAM(cd60)$$ cd61=622 cd62=Kimia - ZA - 100735 - Guessing Game cd63=https cd64=$$CUSTOM_PARAM(cd64)$$ cd65=$$CUSTOM_PARAM(cd65)$$ cd66= cd67= cd68=$$CUSTOM_PARAM(cd68)$$ cd69=Pais Vasco cd70=$$CUSTOM_PARAM(cd70)$$ cd71=$$CUSTOM_PARAM(cd71)$$ cd72=$$CUSTOM_PARAM(cd72)$$ cd73=1535541920726 cd74=$$CUSTOM_PARAM(cd74)$$ cd75=Mozilla/5.0 (Linux; Android 6.0; Azumi_KIREI_A45_D Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/55.0.2883.91 Mobile Safari/537.36 cd76=$$CUSTOM_PARAM(cd76)$$ cd77=$$CUSTOM_PARAM(cd77)$$ cd78=$$CUSTOM_PARAM(cd78)$$ cd80=$$CUSTOM_PARAM(cd80)$$ cd81=$$CUSTOM_PARAM(cd81)$$ cd82=$$CUSTOM_PARAM(cd82)$$ cd83=$$CUSTOM_PARAM(cd83)$$ cd84=$$CUSTOM_PARAM(cd84)$$ cd85=$$CUSTOM_PARAM(cd85)$$ cd86=$$CUSTOM_PARAM(cd86)$$ cd87=$$CUSTOM_PARAM(cd87)$$ cd88=$$CUSTOM_PARAM(cd88)$$ cd89=$$CUSTOM_PARAM(cd89)$$ cd90=$$CUSTOM_PARAM(cd90)$$ pa=click pr1nm=ZA - 100735 - Guessing Game pr1id=363 pr1ca=Gaming pr1br=Kimia z=945938446

If you may remember, you created a custom task in Tag manager that collects all data and sends it to the end point.

Could it be that the task is incorrectly sending the datapoints to the endpoint?

ngulamai commented 6 years ago

Please find here the container (json format) GTM-P3S77DX_workspace9.json.txt

akolchin-MM commented 6 years ago

In the provided payload I see provided cd4 parameter ONLY as a part of another parameter: dl. And also as meaningless cd4=%24%24CUSTOM_PARAM(cd4)%24%24 which we should clean out according to your requirements.

So I don't understand how did you get provided parsing of this payload. Actually, it will looks like this (an it is how it actually comes to the /collect servlet)

_v=j68
a=1161255330
t=event
ni=0
_s=1
dl=https%3A%2F%2Fn38adshostnet.com%2Fads%3Fkey%3D1b617172b554ee6125add931473411c8%26ec%3DAdClick%26ea%3DClick%26cd4%3Dandroid%26cd5%3DMTN%2520SA%26cd7%3Dnull%26cd8%3Dnull%26cd9%3D1535399992%26cd10%3Dnull%26cd13%3Dadventurefeeds%26cd14%3Dnull%26cd15%3Dsite%26cd16%3D106539_124860_6615-ca13293e17d17548967412fc00fa08cf%26cd17%3Dcom.mediatek.factorymode%26cd20%3DAndroid%25206.0%26cd21%3Dnull%26cd22%3Dcom.mediatek.factorymode%26dh%3Dcom.mediatek.factorymode%26cd24%3D0x0%26cd29%3D319275%26ci%3D319275%26cd34%3DZA%26cd36%3Dgeneric%26cd38%3Dnull%26cd40%3Dnull%26cd44%3D-26.230896%252C28.058304%26cd49%3D41.113.66.166%26uip%3D41.113.66.166%26cd53%3D-26.230896%26cd55%3D28.058304%26cd59%3D6.0%26cd60%3Dnull%26cd65%3Dnull%26cd68%3Dnull%26cd74%3Dnull%26cd76%3DMozilla%252F5.0%2520%2528Linux%253B%2520Android%25206.0%253B%2520Azumi_KIREI_A45_D%2520Build%252FMRA58K%253B%2520wv%2529%2520AppleWebKit%252F537.36%2520%2528KHTML%252C%2520like%2520Gecko%2529%2520Version%252F4.0%2520Chrome%252F55.0.2883.91%2520Mobile%2520Safari%252F537.36%26ua%3DMozilla%252F5.0%2520%2528Linux%253B%2520Android%25206.0%253B%2520Azumi_KIREI_A45_D%2520Build%252FMRA58K%253B%2520wv%2529%2520AppleWebKit%252F537.36%2520%2528KHTML%252C%2520like%2520Gecko%2529%2520Version%252F4.0%2520Chrome%252F55.0.2883.91%2520Mobile%2520Safari%252F537.36%26cd77%3Dnull%26cd80%3D11945135399471972%26cd90%3DLCsgNaseyPa12BW%26cd98%3Dcpc%26cc%3D1296967%26cm%3DPop%26cs%3D106539_124860_6615-ca13293e17d17548967412fc00fa08cf%26uid%3DZA%3BMTN%2520SA%3Bgeneric%3Bandroid%3B6.0%3BMozilla%252F5.0%2520%2528Linux%253B%2520Android%25206.0%253B%2520Azumi_KIREI_A45_D%2520Build%252FMRA58K%253B%2520wv%2529%2520AppleWebKit%252F537.36%2520%2528KHTML%252C%2520like%2520Gecko%2529%2520Version%252F4.0%2520Chrome%252F55.0.2883.91%2520Mobile%2520Safari%252F537.36%3B41.113.66.166%3Bnull%26cd12%3DAvazu%2520MDSP%26cn%3D319275
ul=en-us
de=UTF-8
sd=24-bit
sr=1440x900
vp=645x734
je=0
cn=%24%24CUSTOM_PARAM(cn)%24%24
cs=%24%24CUSTOM_PARAM(cs)%24%24
cm=Pop
cc=%24%24CUSTOM_PARAM(cc)%24%24
ec=AdClick
ea=Click
_u=aGBAAEAL~
jid=2114188560
gjid=1267487959
cid=1453746889.1535541921
uid=Spain%3BPais%20Vasco%3BBilbao%3Bunknown%3Bunknown%3BAndroid%3B6.0%3BMozilla%2F5.0%20(Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36%3B212.81.201.234%3B%24%24CUSTOM_PARAM(cd11)%24%24%3B%24%24CUSTOM_PARAM(cd7)%24%24%3B%24%24CUSTOM_PARAM(cd10)%24%24
tid=UA-109728168-4
_gid=1317808649.1535541921
_r=1
gtm=G86P3S77DX
cd1=Spain%3BPais%20Vasco%3BBilbao%3Bunknown%3Bunknown%3BAndroid%3B6.0%3BMozilla%2F5.0%20(Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36%3B212.81.201.234%3B%24%24CUSTOM_PARAM(cd11)%24%24%3B%24%24CUSTOM_PARAM(cd7)%24%24%3B%24%24CUSTOM_PARAM(cd10)%24%24
cd2=Kimia
cd3=ZA%20-%20100735%20-%20Guessing%20Game
cd4=%24%24CUSTOM_PARAM(cd4)%24%24
cd5=%24%24CUSTOM_PARAM(cd5)%24%24
cd6=MjEjMzYzIzY1IzYyMnw5ODR8RVN8M3w1fHx8MjN3ZXQ3djZveGl2fDM2YmFmMDUwLWFiN2UtMTFlOC1iMmJhLWY4YmMxMjUzODFiOHx8
cd7=%24%24CUSTOM_PARAM(cd7)%24%24
cd8=%24%24CUSTOM_PARAM(cd8)%24%24
cd9=%24%24CUSTOM_PARAM(cd9)%24%24
cd10=%24%24CUSTOM_PARAM(cd10)%24%24
cd11=n%2Fa
cd12=Avazu%20MDSP
cd13=%24%24CUSTOM_PARAM(cd13)%24%24
cd14=%24%24CUSTOM_PARAM(cd14)%24%24
cd15=%24%24CUSTOM_PARAM(cd15)%24%24
cd16=%24%24CUSTOM_PARAM(cd16)%24%24
cd17=%24%24CUSTOM_PARAM(cd17)%24%24
cd18=1453746889.1535541921_1535541921326
cd20=%24%24CUSTOM_PARAM(cd20)%24%24
cd21=%24%24CUSTOM_PARAM(cd21)%24%24
cd22=%24%24CUSTOM_PARAM(cd22)%24%24
cd23=%24%24CUSTOM_PARAM(cd23)%24%24
cd24=%24%24CUSTOM_PARAM(cd24)%24%24
cd25=65
cd26=%24%24CUSTOM_PARAM(cd26)%24%24
cd28=Chrome
cd29=363
cd30=Bilbao
cd31=%24%24CUSTOM_PARAM(cd31)%24%24
cd32=36baf050-ab7e-11e8-b2ba-f8bc125381b8
cd33=Spain
cd34=%24%24CUSTOM_PARAM(cd34)%24%24
cd35=unknown
cd36=%24%24CUSTOM_PARAM(cd36)%24%24
cd37=%24%24CUSTOM_PARAM(cd37)%24%24
cd38=%24%24CUSTOM_PARAM(cd38)%24%24
cd39=%24%24CUSTOM_PARAM(cd39)%24%24
cd40=%24%24CUSTOM_PARAM(cd40)%24%24
cd41=%24%24CUSTOM_PARAM(cd41)%24%24
cd42=%24%24CUSTOM_PARAM(cd42)%24%24
cd43=%24%24CUSTOM_PARAM(cd43)%24%24
cd44=%24%24CUSTOM_PARAM(cd44)%24%24
cd45=text%2Fhtml%2Capplication%2Fxhtml%2Bxml%2Capplication%2Fxml%3Bq%3D0.9%2Cimage%2Fwebp%2Cimage%2Fapng%2C*%2F*%3Bq%3D0.8
cd46=en-US%2Cen%3Bq%3D0.9
cd47=%24%24CUSTOM_PARAM(cd47)%24%24
cd48=212.81.201.234
cd49=%24%24CUSTOM_PARAM(cd49)%24%24
cd50=en
cd51=%24%24CUSTOM_PARAM(cd51)%24%24
cd52=43.26271
cd53=%24%24CUSTOM_PARAM(cd53)%24%24
cd54=-2.92528
cd55=%24%24CUSTOM_PARAM(cd55)%24%24
cd56=Android
cd57=unknown
cd58=6.0
cd59=%24%24CUSTOM_PARAM(cd59)%24%24
cd60=%24%24CUSTOM_PARAM(cd60)%24%24
cd61=622
cd62=Kimia%20-%20ZA%20-%20100735%20-%20Guessing%20Game
cd63=https
cd64=%24%24CUSTOM_PARAM(cd64)%24%24
cd65=%24%24CUSTOM_PARAM(cd65)%24%24
cd66=
cd67=
cd68=%24%24CUSTOM_PARAM(cd68)%24%24
cd69=Pais%20Vasco
cd70=%24%24CUSTOM_PARAM(cd70)%24%24
cd71=%24%24CUSTOM_PARAM(cd71)%24%24
cd72=%24%24CUSTOM_PARAM(cd72)%24%24
cd73=1535541920726
cd74=%24%24CUSTOM_PARAM(cd74)%24%24
cd75=Mozilla%2F5.0%20(Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36
cd76=%24%24CUSTOM_PARAM(cd76)%24%24
cd77=%24%24CUSTOM_PARAM(cd77)%24%24
cd78=%24%24CUSTOM_PARAM(cd78)%24%24
cd80=%24%24CUSTOM_PARAM(cd80)%24%24
cd81=%24%24CUSTOM_PARAM(cd81)%24%24
cd82=%24%24CUSTOM_PARAM(cd82)%24%24
cd83=%24%24CUSTOM_PARAM(cd83)%24%24
cd84=%24%24CUSTOM_PARAM(cd84)%24%24
cd85=%24%24CUSTOM_PARAM(cd85)%24%24
cd86=%24%24CUSTOM_PARAM(cd86)%24%24
cd87=%24%24CUSTOM_PARAM(cd87)%24%24
cd88=%24%24CUSTOM_PARAM(cd88)%24%24
cd89=%24%24CUSTOM_PARAM(cd89)%24%24
cd90=%24%24CUSTOM_PARAM(cd90)%24%24
pa=click
pr1nm=ZA%20-%20100735%20-%20Guessing%20Game
pr1id=363
pr1ca=Gaming
pr1br=Kimia
z=945938446

By the way, you should not provide parameters twice in the request - but it is not the cause of this problem for now.

If you may remember, you created a custom task in Tag manager that collects all data and sends it to the end point. Could it be that the task is incorrectly sending the datapoints to the endpoint?

Yes, I remember. But usually, there are dozens other tags and triggers in the GTM container which can impact result. I will try to import provided json and investigate it if it could be possible.

But for now I see that you just compose payload incorrectly and I don't understand why you expect that it will be processed as you wish.

ngulamai commented 6 years ago

In the provided container, you only have google analytics and the task to send the variables in the datalayer to the endpoint. If the payload is incorrectly sent, it should be because this custom task is incorrectly building it. Please let us know your thoughts once you review the entire chain. If helps, please see below the code that is triggering the GTM container. The flow is as follows

1) Consumer clicks on banner (exaample: https://n38adshostnet.com/ads?key=1b617172b554ee6125add931473411c8&ec=AdClick&ea=Click&cd4=android&cd5=MTN%20SA&cd7=null&cd8=null&cd9=1535399992&cd10=null&cd13=adventurefeeds&cd14=null&cd15=site&cd16=106539_124860_6615-ca13293e17d17548967412fc00fa08cf&cd17=com.mediatek.factorymode&cd20=Android%206.0&cd21=null&cd22=com.mediatek.factorymode&dh=com.mediatek.factorymode&cd24=0x0&cd29=319275&ci=319275&cd34=ZA&cd36=generic&cd38=null&cd40=null&cd44=-26.230896%2C28.058304&cd49=41.113.66.166&uip=41.113.66.166&cd53=-26.230896&cd55=28.058304&cd59=6.0&cd60=null&cd65=null&cd68=null&cd74=null&cd76=Mozilla%2F5.0%20%28Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36&ua=Mozilla%2F5.0%20%28Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36&cd77=null&cd80=11945135399471972&cd90=LCsgNaseyPa12BW&cd98=cpc&cc=1296967&cm=Pop&cs=106539_124860_6615-ca13293e17d17548967412fc00fa08cf&uid=ZA;MTN%20SA;generic;android;6.0;Mozilla%2F5.0%20%28Linux%3B%20Android%206.0%3B%20Azumi_KIREI_A45_D%20Build%2FMRA58K%3B%20wv%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Version%2F4.0%20Chrome%2F55.0.2883.91%20Mobile%20Safari%2F537.36;41.113.66.166;null&cd12=Avazu%20MDSP&cn=319275)

2) the following code is executed



3) The data is sent to the collect endpoint while in parallel the consumer is sent to the final destination.

akolchin-MM commented 6 years ago

Honestly, I am trying to be helpful.

If the payload is incorrectly sent, it should be because this custom task is incorrectly building it.

To make it clear, the purpose of customTask itself is only to copy the Analytics tracking request to our /collect servlet also. And as far as I can see, it is doing it exactly. And my responsibility in relation to GTM settings was only this task code. All other things in this container were configured by somebody else from your teams and I can not be sure of his certain intentions.

But I still try to help with it. At least I will explain what I can see.

In the provided script I see this code: "OS_Name_Publisher":"$$CUSTOM_PARAM(cd4)$$", I can suppose that it tries to assign provided in the URL cd4 parameter to the OS_Name_Publisher data layer variable. Which then used in GTM Analytics tag as the source of value for the custom dimension with index 4 (see screenshot).
image

Unfortunately, these parts of the provided script are not workable and I even can't understand what was the meant of such code and thus how to fix it. I only can suppose that this string $$CUSTOM_PARAM(cd4)$$ should be substituted on the server side (n38adshostnet.com -?) by the value of cd4 provided in initial URL but by some reason, it is not happening.

That is the reason why actually we see this text literally $$CUSTOM_PARAM(cd4)$$ as the value of cd4 sent to our /collect. And we are removing out such type of values according to your own requirements.

And by the way, this is also true for all other custom dimensions and some other parameters which are filled in the same way in the provided script. image

I don't know who was the author of this script and how it should work according to his opinion. Maybe it is better to ask him. Unfortunately, for now, I don't know how I can also help with this.

I suppose that the best way is to fix it on the server side so It will work the way it was intended to work (by whom?).

But if it is not possible now for some reasons please let me know, I will try to manage it somehow, but I don't have good ideas for it yet.

ngulamai commented 6 years ago

Hi Alexander, thank you for your feedback. we shared with the tracking team so they can try to fix it.