search

dhowe / AdNauseam

AdNauseam: Fight back against advertising surveillance
GNU General Public License v3.0
4.59k stars 191 forks source link

Incoming cookies (non-ChannelId) incorrectly allowed following Ad visits #590

Closed cqx931 closed 7 years ago

cqx931 commented 8 years ago

Note: see issue #602 for issues relating to Channel ID cookies. In terms of this ticket, we are still waiting for a recreate-able case of a non-Channel ID cookie.

Please import the following ad to reproduce:

{

  "https://www.google.com.hk/?gfe_rd=cr&ei=U1ofWOWTEq7Y8ge9g4EQ#q=credit+card": {

    "google.com.hk::www.hangseng.com/enjoycard::享全年折扣優惠及賺2x enJoy Dollars ,網上申請可享高達$300 enJoy Dollars::恒生enJoy卡永久免年費 - hangseng.com‎": {
      "id": 7,
      "attempts": 0,
      "visitedTs": 0,
      "attemptedTs": 0,
      "contentData": {
        "title": "恒生enJoy卡永久免年費 - hangseng.com‎",
        "text": "享全年折扣優惠及賺2x enJoy Dollars ,網上申請可享高達$300 enJoy Dollars",
        "site": "www.hangseng.com/enjoycard"
      },
      "contentType": "text",
      "title": "恒生enJoy卡永久免年費 - hangseng.com‎",
      "resolvedTargetUrl": null,
      "foundTs": 1478449752801,
      "targetUrl": "https://www.google.com.hk/aclk?sa=L&ai=DChcSEwjS0J3CxpTQAhWJALwKHWoSCQ4YABAG&sig=AOD64_1Wf4eiYqkQAYkZOixfOd_MAk7bnA&q=&ved=0ahUKEwiQ1JrCxpTQAhVIyrwKHZUZCUgQ0QwILQ&adurl=",
      "pageTitle": "Google",
      "pageUrl": "https://www.google.com.hk/?gfe_rd=cr&ei=U1ofWOWTEq7Y8ge9g4EQ#q=credit+card",
      "errors": null,
      "current": true,
      "pageDomain": "google.com.hk",
      "version": "2.3.83",
      "targetDomain": "google.com.hk"
    }
  }
}

Though I can see this cookie blocked message from blockIncomingCookies() in core.js, The cookies are still saved.

screen shot 2016-11-07 at 1 20 19 am screen shot 2016-11-07 at 1 08 18 am

dhowe commented 8 years ago

This appears to be a 'channel id' token set by (in this case) google. I've also seen them set by doubleclick, which would make sense. I'm not sure how we want to handle these, but it is not a standard cookie.

cqx931 commented 8 years ago
  1. To me, this Token Binding sounds completely for the security sake and there is no particular reason for us to block them...
  2. But we still have this problem with cookies as I can observe many cookies saved but just used this as an example.
dhowe commented 8 years ago

The reason for us to block them is that they may be used for identification purposed. But as you can see here (all the incoming headers for the ad request above), I don't it being set anywhere:

[TRYING] Ad#7(text) https://www.google.com.hk/aclk?sa=L&ai=DChcSEwjS0J3CxpTQAhWJALwKHWoSCQ4YABA…kQAYkZOixfOd_MAk7bnA&q=&ved=0ahUKEwiQ1JrCxpTQAhVIyrwKHZUZCUgQ0QwILQ&adurl=
[UAGENT] (Default) Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36 https://www.google.com.hk/aclk?sa=L&ai=DChcSEwjS0J3CxpTQAhWJALwKHWoSCQ4YABA…kQAYkZOixfOd_MAk7bnA&q=&ved=0ahUKEwiQ1JrCxpTQAhVIyrwKHZUZCUgQ0QwILQ&adurl=

[HEADERS] (Incoming) https://www.google.com.hk/aclk?sa=L&ai=DChcSEwjS0J3CxpTQAhWJALwKHWoSCQ4YABA…kQAYkZOixfOd_MAk7bnA&q=&ved=0ahUKEwiQ1JrCxpTQAhVIyrwKHZUZCUgQ0QwILQ&adurl=
13) alt-svc quic=":443"; ma=2592000; v="36,35,34"
12) set-cookie NID=90=JFAM6qjJj9obC8-cubUMH8ASHS1zsrG4szI1uuXfwQz02Ara42LkRBGZPErRvBpp-5aAGpy6TKmswFyJFj5Q44c1A1WGjXS5f49DL4fDKEYqA_s1SrOh7Lky9YgbLpuW; expires=Tue, 09-May-2017 13:35:49 GMT; path=/; domain=.google.com.hk; HttpOnly
[COOKIE] (Block) NID=90=JFAM6qjJj9obC8-cubUMH8ASHS1zsrG4szI1uuXfwQz02Ara42LkRBGZPErRvBpp-5aAGpy6TKmswFyJFj5Q44c1A1WGjXS5f49DL4fDKEYqA_s1SrOh7Lky9YgbLpuW; expires=Tue, 09-May-2017 13:35:49 GMT; path=/; domain=.google.com.hk; HttpOnly
11) x-xss-protection 1; mode=block
10) content-length 0
9) server adclick_server
8) x-content-type-options nosniff
7) p3p CP="This is not a P3P policy! See https://www.google.com/support/accounts/answer/151657?hl=en for more info."
6) content-type text/html; charset=UTF-8
5) location http://bs.serving-sys.com/Serving/?cn=search&semdid=825-329-7781_600831076_…_c$$&urtu=http://www.hangseng.com/cms/emkt/pmo/grp06/p13/chi/index.html%3F
4) cache-control no-cache, must-revalidate
3) expires Fri, 01 Jan 1990 00:00:00 GMT
2) pragma no-cache
1) date Mon, 07 Nov 2016 13:35:49 GMT
0) status 302
[UAGENT] (Default) Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36 http://bs.serving-sys.com/Serving/?cn=search&semdid=825-329-7781_600831076_…_c$$&urtu=http://www.hangseng.com/cms/emkt/pmo/grp06/p13/chi/index.html%3F

[HEADERS] (Incoming-redirect) http://bs.serving-sys.com/Serving/?cn=search&semdid=825-329-7781_600831076_…_c$$&urtu=http://www.hangseng.com/cms/emkt/pmo/grp06/p13/chi/index.html%3F
14) content-length 236
13) date Mon, 07 Nov 2016 13:35:48 GMT
12) p3p CP="NOI DEVa OUR BUS UNI"
11) x-powered-by ASP.NET
10) set-cookie u2=213e27bc-338a-42b5-a20f-8839b3a247dd4bo010; expires=Sun, 05-Feb-2017 08:35:00 GMT; domain=.serving-sys.com; path=/
[COOKIE] (Block) u2=213e27bc-338a-42b5-a20f-8839b3a247dd4bo010; expires=Sun, 05-Feb-2017 08:35:00 GMT; domain=.serving-sys.com; path=/
9) set-cookie eyeblaster=; expires=Mon, 01-Jan-2000 00:00:00 GMT; domain=.serving-sys.com; path=/
[COOKIE] (Block) eyeblaster=; expires=Mon, 01-Jan-2000 00:00:00 GMT; domain=.serving-sys.com; path=/
8) set-cookie S1=000001000mGm3.....dLL300H825-329-7781_600831076_27884780461_Jhm11kOh000000; expires=Sun, 05-Feb-2017 08:35:00 GMT; domain=.serving-sys.com; path=/
[COOKIE] (Block) S1=000001000mGm3.....dLL300H825-329-7781_600831076_27884780461_Jhm11kOh000000; expires=Sun, 05-Feb-2017 08:35:00 GMT; domain=.serving-sys.com; path=/
7) set-cookie r1=1478507749_1
[COOKIE] (Block) r1=1478507749_1
6) set-cookie searchsession_1%24825-329-7781_600831076_27884780461_Jhm11kOh=0.3603395
[COOKIE] (Block) searchsession_1%24825-329-7781_600831076_27884780461_Jhm11kOh=0.3603395
5) server Microsoft-IIS/7.5
4) location http://www.hangseng.com/cms/emkt/pmo/grp06/p13/chi/index.html?&mkwid=sJhm11kOh_103312515661_credit%20card_e_c
3) expires Sun, 05-Jun-2005 22:00:00 GMT
2) content-type text/html; charset=UTF-8
1) pragma no-cache
0) cache-control no-cache, no-store
[UAGENT] (Default) Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36 http://www.hangseng.com/cms/emkt/pmo/grp06/p13/chi/index.html?&mkwid=sJhm11kOh_103312515661_credit%20card_e_c

[HEADERS] (Incoming-redirect) http://www.hangseng.com/cms/emkt/pmo/grp06/p13/chi/index.html?&mkwid=sJhm11kOh_103312515661_credit%20card_e_c
5) date Mon, 07 Nov 2016 13:35:48 GMT
4) x-powered-by ASP.NET
3) etag "0d019e3f419d21:d28f"
2) last-modified Thu, 29 Sep 2016 01:57:52 GMT
1) content-type text/html
0) content-length 28396
[VISIT] Ad#7(text) 恒生enJoy卡獨享全年折扣優惠，賺2x enJoy Dollars!

But in any case, can you show me a similar example (include a single ad JSON again) that is not a 'CHANNEL ID' cookie? Perhaps the problem is more widespread than I thought...

dhowe commented 8 years ago

@cqx931 please also take a look at #600, using the same ad file in Firefox

cqx931 commented 8 years ago

Finally find this cookie from ad visit that is not Channel ID:

{
  "https://www.yahoo.com/news/?ref=gs": {
    "yahoo.com::https://www.yahoo.com/sy/uu/api/res/1.2/5JMihz_qyvIqxtfHmIh8rQ--/Zmk9c3RyaW07aD0xMjM7cHlvZmY9MDtxPTgwO3c9MjIwO3NtPTE7YXBwaWQ9eXRhY2h5b24-/https://s.yimg.com/av/moneyball/ads/1476884064237-5806.jpg.cf.webp": {
      "id": 2,
      "attempts": 0,
      "visitedTs": 0,
      "attemptedTs": 0,
      "contentData": {
        "src": "https://www.yahoo.com/sy/uu/api/res/1.2/5JMihz_qyvIqxtfHmIh8rQ--/Zmk9c3RyaW07aD0xMjM7cHlvZmY9MDtxPTgwO3c9MjIwO3NtPTE7YXBwaWQ9eXRhY2h5b24-/https://s.yimg.com/av/moneyball/ads/1476884064237-5806.jpg.cf.webp",
        "width": 220,
        "height": 123
      },
      "contentType": "img",
      "title": "Pending",
      "resolvedTargetUrl": null,
      "foundTs": 1478533281375,
      "targetUrl": "https://beap.gemini.yahoo.com/mbclk?bv=1.0.0&es=l7h.ICYGIS_kGw5ERQB68GsxVHkzxaIO3x_kMybL.ccQ7vsd.XYRmXPp6QC7CryJ1f2sPgppHLwzPigFqyGVQfla51o3O.nDIqqHHjFqYgcWhOMqbe2bldl3Yk9xgzp67DMtCigxkdVFKdV6MuQScidI.IdANviqqXyC1xSAi8Kvg.QRLOsCIcdnppKMWnozaF7M2nyj9ocIQuRQJOTcet06XqtXUgCx8MP0I4tMkF6tD.D5BTmuuATL3i.55ZNoYM3E2af3zR0OSs47XJFXvz66qS7fGlsv7Mf2SBrsUNDc9Om32zt31gsGKQ.L_dzFXhryASeoUbSO4nQBhOFZy7PajCPsOpS82Hu6zRLbHnoAC8f.8O3j6LfVji608icBp5lMxoh_CyuheeRHafxC8bkw1FHJUyT2UOwCe00isXQdDuj2gJTYmocEVtRpCSOf7zzh7bZ0SrHOinW.eUzLAZxhki532U1mSchbqvRATU7vRiWSzBX7JGQcnsqXJJa8VvhRD8yvKVG4UD628XULlu5PkH6xjA--%26lp=http%3A%2F%2Fwww.richdadregistration-international.com%2Fhongkong%2Findex.cgi%3Fmid%3D6315224",
      "pageTitle": "Yahoo News - Latest News & Headlines",
      "pageUrl": "https://www.yahoo.com/news/?ref=gs",
      "errors": null,
      "current": true,
      "pageDomain": "yahoo.com",
      "version": "2.3.83",
      "targetDomain": "yahoo.com"
    }
  }
}
screen shot 2016-11-07 at 11 50 40 pm screen shot 2016-11-07 at 11 50 57 pm
dhowe commented 8 years ago

I can't recreate this. But I've just checked in some better logging code. If you flip the dbug switch at core.js, line 1286, you should see all the incoming headers/cookies... You might also want to set clearVisitData (at top of core.js) to 1, so that you can keep re-using the same imported ad data. Let me know if you see anything interesting in your log (mine is below).

[TRYING] Ad#2(img) https://beap.gemini.yahoo.com/mbclk?bv=1.0.0&es=l7h.ICYGIS_kGw5ERQB68GsxVHk…chdadregistration-international.com%2Fhongkong%2Findex.cgi%3Fmid%3D6315224
[UAGENT] (Default) Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36 https://beap.gemini.yahoo.com/mbclk?bv=1.0.0&es=l7h.ICYGIS_kGw5ERQB68GsxVHk…chdadregistration-international.com%2Fhongkong%2Findex.cgi%3Fmid%3D6315224
[HEADERS] (Incoming) https://beap.gemini.yahoo.com/mbclk?bv=1.0.0&es=l7h.ICYGIS_kGw5ERQB68GsxVHk…chdadregistration-international.com%2Fhongkong%2Findex.cgi%3Fmid%3D6315224
6) server ATS
5) connection close
4) age 0
3) date Mon, 07 Nov 2016 16:07:05 GMT
2) content-length 0
1) x-trace-id 9d8a8bdc-a500-11e6-bc20-008cfac0b920-7f41f8b38700
0) location http://www.richdadregistration-international.com/hongkong/index.cgi?mid=6315224
[UAGENT] (Default) Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36 http://www.richdadregistration-international.com/hongkong/index.cgi?mid=6315224
[HEADERS] (Incoming-redirect) http://www.richdadregistration-international.com/hongkong/index.cgi?mid=6315224
13) viewmode desktop
12) x-cache-status BYPASS
11) vsid false
10) viewmode desktop
9) x-cache-status BYPASS
8) x-powered-by ASP.NET
7) location http://www.richdadregistration-international.com/hongkong/9.0000/index.cgi?mid=6315224&otsrid=91c96c38-ecdb-46ae-b34c-730f97871194
6) expires -1
5) pragma no-cache
4) cache-control no-cache
3) connection keep-alive
2) content-length 334
1) date Mon, 07 Nov 2016 16:07:06 GMT
0) server nginx
[UAGENT] (Default) Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.85 Safari/537.36 http://www.richdadregistration-international.com/hongkong/9.0000/index.cgi?mid=6315224&otsrid=91c96c38-ecdb-46ae-b34c-730f97871194
[HEADERS] (Incoming-redirect) http://www.richdadregistration-international.com/hongkong/9.0000/index.cgi?mid=6315224&otsrid=91c96c38-ecdb-46ae-b34c-730f97871194
17) viewmode desktop
16) x-cache-status BYPASS
15) content-encoding gzip
14) vsid false
13) viewmode desktop
12) x-cache-status BYPASS
11) x-powered-by ASP.NET
10) x-aspnet-version 4.0.30319
9) x-aspnetmvc-version 3.0
8) expires -1
7) pragma no-cache
6) cache-control no-cache
5) vary Accept-Encoding
4) connection keep-alive
3) transfer-encoding chunked
2) content-type text/html; charset=utf-8
1) date Mon, 07 Nov 2016 16:07:06 GMT
0) server nginx
[VISIT] Ad#2(img) Rich Dad Education FREE Workshop

screen shot 2016-11-08 at 12 13 22 am

dhowe commented 8 years ago

Though I think the two issues are separate, and thus should be separate tickets (Channel ID cookies, and regular cookies not being blocked).

See issue #602 for Channel ID cookies

cqx931 commented 7 years ago

see #621