search

gorhill / httpswitchboard

Point & click to forbid/allow any class of requests made by your browser. Use it to block scripts, iframes, ads, facebook, etc.
GNU General Public License v3.0
1.33k stars 84 forks source link

not all of the Behind-the-scene requests are being intercepted #182

Closed requiredregistration closed 10 years ago

requiredregistration commented 10 years ago

latest chrome stable

chrome still establishes connections and they are not shown on the matrix either.

gorhill commented 10 years ago

Need more details. How to reproduce on my side?

requiredregistration commented 10 years ago

clean install of chrome, run, click 'Skip for now' in the 'Chrome' tab, go to 'Settings', set 'Open a specific page or set of pages.' to about:blank, install HTTPSB, go to the rules manager:

chromium-behind-the-scene whitelist blacklist * *

commit all, exit and run again, you'll see connections being established.

gorhill commented 10 years ago

How did you check for "connections being established"? And to what server? When it comes to details, more is better in bug reporting.

requiredregistration commented 10 years ago

*.1e100.net URLs. they belong to google.

http://technet.microsoft.com/en-us/sysinternals/bb897437

my-password-is-password commented 10 years ago

I was able to reproduce this.

  1. Open 2 tabs
  2. Tab 1 go to chrome://net-internals/#events
  3. Tab 2 go to chrome://settings/

googtranslate

It a requests to https://translate.googleapis.com/translate_a/l?client=chrome&cb=sl&hl=en&key=AIzaSyBOti4mM-6x9WDnZIjIeyEU21OpBXqWBgw&alpha=1

Probably a Chrome feature they don't want extensions blocking? There's a Languages section in settings with an option Offer to translate pages that aren't in a language I read. Probably needs the file to keep up to date. Idk

gorhill commented 10 years ago

For sure there are requests which can't be seen by extensions, like when visiting the chrome store for example (this would be a security risk). Whatever can not be intercepted and reported by HTTPSB, I need to document, so that a user does not get a false idea that all is filtered/reported.

I did launched ntop yesterday night, after literally blocking all in HTTPSB, and quitting all apps which could generate net traffic.

I've noticed a connection to client2.google.net about every two hours (presumingly from the browser). This is not reported in the matrix, which means it's bypassing chrome.webRequest.onBeforeRequest. This is from Chromium on Linux, so this is not using the closed-source Chrome.

If you use Chrome though instead of Chromium, this is not unexpected that it will do things even hidden to extensions. There is nothing extensions can do about this.

However regarding Chromium, I need to investigate further what is this connection, my understanding was that with all settings which could result in net traffic being turned off, there should be no connection to Google server whatsoever.

gorhill commented 10 years ago

I was able to reproduce this

This is at launch, right? As reported in the other issue you opened, extensions are not immediately up and functioning at launch time, Chromium decides when to launch extensions, and this might be after some requests have been done by the browser. There is nothing an extension can do about this. We should focus on net requests which are done after HTTPSB has been launched, since there is nothing HTTPSB can do for requests made before it executes.

gorhill commented 10 years ago

Also, certainly requests related to browser or extensions update are not relayed to extensions. Investigating this is time consuming, which mean I am not working on stuff which allows me to release versions. So I will ask you guys you investigate fully what you think should be reported in HTTPSB while it is not. It appears to me at this point what is reported is normal browser behavior given you are using Chrome on Windows. So please:

Bottom line, I can't fix browser issue, I can only address HTTPSB issues.

And I will have to start to be more hardcore on bug which are described with only "chrome still establishes connections and they are not shown on the matrix either". From now on, issues like these will be close with "not enough details", given that they put the whole burden on the developer's shoulders to figure what is the detailed problem. Please keep in mind my time is as valuable to me as yours is to you.

gorhill commented 10 years ago

https://code.google.com/p/chromium/codesearch#chromium/src/chrome/browser/extensions/api/web_request/web_request_permissions.cc&q=IsSensitiveURL&sq=package:chromium&l=24:

// Returns true if the URL is sensitive and requests to this URL must not be
// modified/canceled by extensions, e.g. because it is targeted to the webstore
// to check for updates, extension blacklisting, etc.
requiredregistration commented 10 years ago

gorhill, it's not that my description of the issue isn't complete. the description of the issue is clear. connections are still being made after the installation and the initialization of HTTPSB, and they are not shown on the matrix. it's that i don't know what you know and don't know. now, i do care about your time, and we can progress a lot faster if we communicate in real-time. i'm waiting for you: https://webchat.freenode.net/, in the #httpsb channel.

requiredregistration commented 10 years ago

this is from chrome://net-internals/#events of a chromium startup (the established connections outside of chromium and chrome are to *.1e100.net URLs)

ID | Source Type | Description
7   URL_REQUEST https://translate.googleapis.com/translate_a/l?client=chrome&cb=sl&hl=en&key=AIzaSyBOti4mM-6x9WDnZIjIeyEU21OpBXqWBgw&alpha=1
8   HTTP_STREAM_JOB 
13  CONNECT_JOB 
22  SOCKET  
23  CERT_VERIFIER_JOB   
33  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/background.html
34  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/lib/punycode.min.js
35  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/lib/publicsuffixlist.min.js
36  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/types.js
37  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/strpacker.js
38  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/uritools.js
39  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/usersettings.js
40  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/async.js
41  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/lists.js
42  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/background.js
43  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/httpsb.js
44  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/reqstats.js
45  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/cookies.js
46  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/profiler.js
47  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/storage.js
48  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/tab.js
49  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/traffic.js
50  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/contextmenu.js
51  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/contentscripthandlers.js
52  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/js/start.js
53  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/assets/thirdparties/mxr.mozilla.org/effective_tld_names.dat
54  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/assets/httpsb/presets.txt
55  HOST_RESOLVER_IMPL_REQUEST  translate.googleapis.com:443
56  IPV6_REACHABILITY_CHECK 
57  UDP_SOCKET  [2001:4860:4860::8888]:53
58  SPDY_SESSION    translate.googleapis.com:443 (DIRECT)
59  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/assets/httpsb/blacklist.txt
60  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/assets/thirdparties/hosts-file.net/ad-servers
61  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/assets/thirdparties/mirror1.malwaredomains.com/files/immortal_domains.txt
62  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/assets/thirdparties/mirror1.malwaredomains.com/files/justdomains
63  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/assets/thirdparties/pgl.yoyo.org/as/serverlist
64  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/assets/thirdparties/someonewhocares.org/hosts/hosts
65  URL_REQUEST chrome-extension://iicfjgknabcnpannjkmgncccmfomoddb/assets/thirdparties/www.malwaredomainlist.com/hostslist/hosts.txt
67  URL_REQUEST 
69  URL_REQUEST http://pjntqhfqfb/
70  HTTP_STREAM_JOB http://pjntqhfqfb/
71  HOST_RESOLVER_IMPL_REQUEST  pjntqhfqfb:80
72  IPV6_REACHABILITY_CHECK 
73  UDP_SOCKET  [2001:4860:4860::8888]:53
74  CONNECT_JOB pm/pjntqhfqfb:80
75  HOST_RESOLVER_IMPL_REQUEST  pjntqhfqfb:80
76  IPV6_REACHABILITY_CHECK 
77  UDP_SOCKET  [2001:4860:4860::8888]:53
78  HOST_RESOLVER_IMPL_JOB  pjntqhfqfb
80  URL_REQUEST http://futoaenwcf/
81  HTTP_STREAM_JOB http://futoaenwcf/
82  HOST_RESOLVER_IMPL_REQUEST  futoaenwcf:80
83  IPV6_REACHABILITY_CHECK 
84  UDP_SOCKET  [2001:4860:4860::8888]:53
85  CONNECT_JOB pm/futoaenwcf:80
86  HOST_RESOLVER_IMPL_REQUEST  futoaenwcf:80
87  IPV6_REACHABILITY_CHECK 
88  UDP_SOCKET  [2001:4860:4860::8888]:53
89  HOST_RESOLVER_IMPL_JOB  futoaenwcf
91  URL_REQUEST http://mdqvixgxwi/
92  HTTP_STREAM_JOB http://mdqvixgxwi/
93  HOST_RESOLVER_IMPL_REQUEST  mdqvixgxwi:80
94  IPV6_REACHABILITY_CHECK 
95  UDP_SOCKET  [2001:4860:4860::8888]:53
96  CONNECT_JOB pm/mdqvixgxwi:80
97  HOST_RESOLVER_IMPL_REQUEST  mdqvixgxwi:80
98  IPV6_REACHABILITY_CHECK 
99  UDP_SOCKET  [2001:4860:4860::8888]:53
100 HOST_RESOLVER_IMPL_JOB  mdqvixgxwi
102 URL_REQUEST https://ssl.gstatic.com/safebrowsing/csd/client_model_v5.pb
103 HTTP_STREAM_JOB https://ssl.gstatic.com/
104 HOST_RESOLVER_IMPL_REQUEST  ssl.gstatic.com:443
105 IPV6_REACHABILITY_CHECK 
106 UDP_SOCKET  [2001:4860:4860::8888]:53
107 CONNECT_JOB ssl/ssl.gstatic.com:443
108 CONNECT_JOB ssl/ssl.gstatic.com:443
109 HOST_RESOLVER_IMPL_REQUEST  ssl.gstatic.com:443
110 IPV6_REACHABILITY_CHECK 
111 UDP_SOCKET  [2001:4860:4860::8888]:53
112 HOST_RESOLVER_IMPL_JOB  ssl.gstatic.com
113 HOST_RESOLVER_IMPL_REQUEST  ssl.gstatic.com:443
114 IPV6_REACHABILITY_CHECK 
115 UDP_SOCKET  [2001:4860:4860::8888]:53
116 SOCKET  ssl/ssl.gstatic.com:443
117 CERT_VERIFIER_JOB   
118 HOST_RESOLVER_IMPL_REQUEST  ssl.gstatic.com:443
119 IPV6_REACHABILITY_CHECK 
120 UDP_SOCKET  [2001:4860:4860::8888]:53
121 SPDY_SESSION    ssl.gstatic.com:443 (DIRECT)
my-password-is-password commented 10 years ago

"This is at launch, right?"

At launch chromium requests the same 2 files that show up in requiredregistration's net-internals logs.

These files might have something to do with these chromium settings shown below. Disabling these seem to stop the requests at launch.

chromiumsetting

But in the gif I posted earlier, the request was done after. HTTPSB was already loaded ( shown in window 2 in the gif. ) Going to chrome://settings/ after HTTPSB has launched still makes request to

that doesn't show up in the matrix.

gorhill commented 10 years ago

To confirm whether the request goes through webRequest.OnBeforeRequest() is to remove the comment prefix (//) from this line, then restart the browser. This is the entry point for OnBeforeRequest(), which means if something really goes through OnBeforeRequest(), this will be shown at the console.

If a request is not reported at the console, then it's a request the Chrome browser chooses to not make available to extensions, if it shows in the console but not in the matrix, then HTTPSB fails to report as it should.

gorhill commented 10 years ago

Ok, using Chrome 32 in a VM, chrome://net-internals and HTTPSB's console, I confirm the following requests do not go through the webRequest API:

50  SPDY_SESSION    translate.googleapis.com:443 (undefined)
53  SPDY_SESSION    clients4.google.com:443 (undefined)
68  SPDY_SESSION    accounts.google.com:443 (undefined)
110 SOCKET  
154 SPDY_SESSION    fonts.googleapis.com:443 (undefined)
175 SOCKET  
182 SPDY_SESSION    accounts.youtube.com:443 (undefined)
198 SOCKET  
203 SPDY_SESSION    clients1.google.com:443 (undefined)
234 SOCKET  
238 SPDY_SESSION    www.google.com:443 (undefined)
333 SOCKET  
337 SPDY_SESSION    www.gstatic.com:443 (undefined)

This, with all options disabled, including "Offer to translate pages...".

requiredregistration commented 10 years ago

gorhill: Please keep in mind my time is as valuable to me as yours is to you.

many hours passed. you didn't come to the channel.

gorhill commented 10 years ago

Re. 1e100.net (Google), this one is not reported in the chrome://net-internals itself, let alone extensions.

I think at this point we have a lot of details. So for all requests which do not go through extensions, the only thing I can do is document these so a user will know what to expect if using Chromium or a derived browser.

gorhill commented 10 years ago

So here is what I have this morning, after 17 hours of having Chrome/Windows 7 idling with no tabs opened (showing only URL_REQUEST, removed local requests):

55  SPDY_SESSION    clients4.google.com:443 (undefined)
62  SPDY_SESSION    accounts.google.com:443 (undefined)
141 SPDY_SESSION    fonts.googleapis.com:443 (undefined)
144 SPDY_SESSION    ssl.gstatic.com:443 (undefined)
162 SPDY_SESSION    accounts.youtube.com:443 (undefined)
193 SPDY_SESSION    themes.googleusercontent.com:443 (undefined)
210 SPDY_SESSION    www.google.com:443 (undefined)
239 SPDY_SESSION    translate.googleapis.com:443 (undefined)
277 URL_REQUEST http://pmeuysheos/
286 URL_REQUEST http://bvjvlcwlwt/
295 URL_REQUEST http://aionkiupax/
423 URL_REQUEST http://clients2.google.com/service/update2/crx?os=win&arch=x86&nacl_arch=x86-64&prod=chrome&prodchannel=stable&prodversion=32.0.1700.102&wow64=1&x=id%3Dmimojjlkmoijpicakmndhoigimigcmbb%26v%3D0.0.0.0%26fp%3D%26uc&x=id%3Dnhfgdggnnopgbfdlpeoalgcjdgfafocg%26v%3D1.0.5.0%26fp%3D%26uc&x=id%3Doimompecagnajdejgnnjijobebaeigek%26v%3D1.4.1.377%26fp%3D1.3709e9fd05003ef453670ba7d4dcd6846787a15a10db982e53a94f319f368710%26uc&x=id%3Dhnimpnehoodheedghdeeijklkeaacbdc%26v%3D0.1.0.12332%26fp%3D1.242f72950352cc4578ff306500829fdc91517f0262478b8da3da042429cabd6c%26uc&x=id%3Dhfnkpimlhhgieaddgfemjhofmfblmnib%26v%3D1431%26fp%3D%26uc&x=id%3Dnpdjjkjlcidkjlamlmmdelcjbcpdjocm%26v%3D0.0.0.0%26fp%3D%26uc
424 HTTP_STREAM_JOB http://clients2.google.com/
441 URL_REQUEST https://clients4.google.com/chrome-variations/seed?osname=win
442 HTTP_STREAM_JOB https://clients4.google.com/
456 SPDY_SESSION    clients4.google.com:443 (DIRECT)
458 URL_REQUEST https://clients2.google.com/service/update2/crx?os=win&arch=x86&nacl_arch=x86-64&prod=chromecrx&prodchannel=stable&prodversion=32.0.1700.102&x=id%3Dapdfllckaahabafndbhieahigkjlhalf%26v%3D6.3%26uc&x=id%3Dblpcfgokakmgnkcojhhkbfbldkacnbeo%26v%3D4.2.6%26uc&x=id%3Dcoobgpohoikkiipiblmjeljniedjpjpf%26v%3D0.0.0.20%26uc&x=id%3Dnmmhkkegccagdldgiimedpiccmgmieda%26v%3D0.0.6.0%26uc&x=id%3Dpjkljhegncpnkpknbcohdijeoejaedia%26v%3D7%26uc
459 HTTP_STREAM_JOB https://clients2.google.com/
473 SPDY_SESSION    clients2.google.com:443 (DIRECT)
475 URL_REQUEST https://clients3.google.com/crsignal/client?dist=stable&osname=win&branding=32.0.1700.102&osver=6.1+SP1&hl=en-GB
476 HTTP_STREAM_JOB https://clients3.google.com/
490 SPDY_SESSION    clients3.google.com:443 (DIRECT)
492 URL_REQUEST http://clients2.google.com/service/update2/crx?os=win&arch=x86&nacl_arch=x86-64&prod=chrome&prodchannel=stable&prodversion=32.0.1700.102&wow64=1&x=id%3Dmimojjlkmoijpicakmndhoigimigcmbb%26v%3D0.0.0.0%26fp%3D%26uc&x=id%3Dnhfgdggnnopgbfdlpeoalgcjdgfafocg%26v%3D1.0.5.0%26fp%3D%26uc&x=id%3Doimompecagnajdejgnnjijobebaeigek%26v%3D1.4.1.377%26fp%3D1.3709e9fd05003ef453670ba7d4dcd6846787a15a10db982e53a94f319f368710%26uc&x=id%3Dhnimpnehoodheedghdeeijklkeaacbdc%26v%3D0.1.0.12332%26fp%3D1.242f72950352cc4578ff306500829fdc91517f0262478b8da3da042429cabd6c%26uc&x=id%3Dhfnkpimlhhgieaddgfemjhofmfblmnib%26v%3D1431%26fp%3D%26uc&x=id%3Dnpdjjkjlcidkjlamlmmdelcjbcpdjocm%26v%3D0.0.0.0%26fp%3D%26uc
493 HTTP_STREAM_JOB http://clients2.google.com/
502 URL_REQUEST https://clients4.google.com/chrome-variations/seed?osname=win
503 HTTP_STREAM_JOB https://clients4.google.com/
517 SPDY_SESSION    clients4.google.com:443 (DIRECT)
519 URL_REQUEST https://clients2.google.com/service/update2/crx?os=win&arch=x86&nacl_arch=x86-64&prod=chromecrx&prodchannel=stable&prodversion=32.0.1700.102&x=id%3Dapdfllckaahabafndbhieahigkjlhalf%26v%3D6.3%26uc&x=id%3Dblpcfgokakmgnkcojhhkbfbldkacnbeo%26v%3D4.2.6%26uc&x=id%3Dcoobgpohoikkiipiblmjeljniedjpjpf%26v%3D0.0.0.20%26uc&x=id%3Dnmmhkkegccagdldgiimedpiccmgmieda%26v%3D0.0.6.0%26uc&x=id%3Dpjkljhegncpnkpknbcohdijeoejaedia%26v%3D7%26uc
520 HTTP_STREAM_JOB https://clients2.google.com/
534 SPDY_SESSION    clients2.google.com:443 (DIRECT)
536 URL_REQUEST https://clients3.google.com/crsignal/client?dist=stable&osname=win&branding=32.0.1700.102&osver=6.1+SP1&hl=en-GB
537 HTTP_STREAM_JOB https://clients3.google.com/
551 SPDY_SESSION    clients3.google.com:443 (DIRECT)
553 URL_REQUEST http://clients2.google.com/service/update2/crx?os=win&arch=x86&nacl_arch=x86-64&prod=chrome&prodchannel=stable&prodversion=32.0.1700.102&wow64=1&x=id%3Dmimojjlkmoijpicakmndhoigimigcmbb%26v%3D0.0.0.0%26fp%3D%26uc&x=id%3Dnhfgdggnnopgbfdlpeoalgcjdgfafocg%26v%3D1.0.5.0%26fp%3D%26uc&x=id%3Doimompecagnajdejgnnjijobebaeigek%26v%3D1.4.1.377%26fp%3D1.3709e9fd05003ef453670ba7d4dcd6846787a15a10db982e53a94f319f368710%26uc&x=id%3Dhnimpnehoodheedghdeeijklkeaacbdc%26v%3D0.1.0.12332%26fp%3D1.242f72950352cc4578ff306500829fdc91517f0262478b8da3da042429cabd6c%26uc&x=id%3Dhfnkpimlhhgieaddgfemjhofmfblmnib%26v%3D1431%26fp%3D%26uc&x=id%3Dnpdjjkjlcidkjlamlmmdelcjbcpdjocm%26v%3D0.0.0.0%26fp%3D%26uc
554 HTTP_STREAM_JOB http://clients2.google.com/
563 URL_REQUEST https://clients4.google.com/chrome-variations/seed?osname=win
564 HTTP_STREAM_JOB https://clients4.google.com/
578 SPDY_SESSION    clients4.google.com:443 (DIRECT)
580 URL_REQUEST https://clients2.google.com/service/update2/crx?os=win&arch=x86&nacl_arch=x86-64&prod=chromecrx&prodchannel=stable&prodversion=32.0.1700.102&x=id%3Dapdfllckaahabafndbhieahigkjlhalf%26v%3D6.3%26uc%26ping%3Dr%253D1&x=id%3Dblpcfgokakmgnkcojhhkbfbldkacnbeo%26v%3D4.2.6%26uc%26ping%3Dr%253D1&x=id%3Dcoobgpohoikkiipiblmjeljniedjpjpf%26v%3D0.0.0.20%26uc%26ping%3Dr%253D1&x=id%3Dnmmhkkegccagdldgiimedpiccmgmieda%26v%3D0.0.6.0%26uc%26ping%3Dr%253D1&x=id%3Dpjkljhegncpnkpknbcohdijeoejaedia%26v%3D7%26uc%26ping%3Dr%253D1
581 HTTP_STREAM_JOB https://clients2.google.com/
595 SPDY_SESSION    clients2.google.com:443 (DIRECT)

And what I got at the console:

HTTP Switchboard> Beginning to intercept net requests at 2014-02-04T20:40:51.151Z traffic.js:554
onBeforeRequestHandler()> "https://www.google.com/searchdomaincheck?format=url&type=chrome"

So there a request, https://www.google.com/searchdomaincheck?format=url&type=chrome which is reported in the request log, and in the behind-the-scene matrix.

The others were not reported by the webRequest API, and there is not much I can do for these aside documenting them. And then there are the 1e100.net requests not showing here (I suspect they are related to certificate stuff?).

requiredregistration commented 10 years ago

maybe these requests are tunneled through the *.1e100.net connections. maybe SPDY connections are handled differently?

gorhill commented 10 years ago

maybe SPDY connections are handled differently?

EDIT: Err, it seems I confuse "protocol" with "scheme". Never mind. In any case, what is quoted below is still relevant.

Yes, this is what this comment in Chromium source code suggest:

// Returns true if the scheme is one we want to allow extensions to have access
// to. Extensions still need specific permissions for a given URL, which is
// covered by CanExtensionAccessURL.
// Returns true if the scheme is one we want to allow extensions to have access
// to. Extensions still need specific permissions for a given URL, which is
// covered by CanExtensionAccessURL.
bool HasWebRequestScheme(const GURL& url) {
  return (url.SchemeIs(chrome::kAboutScheme) ||
          url.SchemeIs(content::kFileScheme) ||
          url.SchemeIs(content::kFileSystemScheme) ||
          url.SchemeIs(content::kFtpScheme) ||
          url.SchemeIs(content::kHttpScheme) ||
          url.SchemeIs(content::kHttpsScheme) ||
          url.SchemeIs(extensions::kExtensionScheme));
}

Ref.: https://code.google.com/p/chromium/codesearch#chromium/src/chrome/browser/extensions/api/web_request/web_request_permissions.cc&sq=package:chromium&l=68&rcl=1391507053

So apparently "SPDY" is one of these schemes for which requests are not reported to extensions. Add to this the requests to clients?.google.com which are not reported to extensions for security reasons, as per source comment:

// Returns true if the URL is sensitive and requests to this URL must not be
// modified/canceled by extensions, e.g. because it is targeted to the webstore
// to check for updates, extension blacklisting, etc.
bool IsSensitiveURL(const GURL& url) {
  // TODO(battre) Merge this, CanExtensionAccessURL and
  // PermissionsData::CanExecuteScriptOnPage into one function.
  bool sensitive_chrome_url = false;
  const std::string host = url.host();
  const char kGoogleCom[] = ".google.com";
  const char kClient[] = "clients";
  if (EndsWith(host, kGoogleCom, true)) {
    // Check for "clients[0-9]*.google.com" hosts.
    // This protects requests to several internal services such as sync,
    // extension update pings, captive portal detection, fraudulent certificate
    // reporting, autofill and others.
    if (StartsWithASCII(host, kClient, true)) {
...

https://code.google.com/p/chromium/codesearch#chromium/src/chrome/browser/extensions/api/web_request/web_request_permissions.cc&q=IsSensitiveURL&sq=package:chromium&l=24

Now I do find disturbing these connections to fonts.googleapis.com, account.youtube.com, etc., as far as I read so far, this is not disclosed in their Google Chrome Privacy Whitepaper (I will have to go read again to be sure).

Possible mitigation: using one of the browser switches to filter by hostnames -- somebody did that somewhere for google-analytics.com, I don't know if that would work for these internal requests. (Side effect would be to break auto-update).

gorhill commented 10 years ago

Here is what the whitepaper says re. translate.googleapis.com:

Language detection is done entirely using a client-side library, and does not involve any Google servers. For translation, the contents of a web page are only sent to Google if you explicitly decide to translate it by clicking “Translate” on the bar, or if you’ve previously chosen “Always translate” for a given language via the translate bar Options menu.

This statement doesn't match the request seen to translate.googleapis.com at start up.

Seems this is related to Chrome's "new tab" tab. I don't get the requests below if I force only "about:blank" tab when launching Chrome:

62  SPDY_SESSION    accounts.google.com:443 (undefined)
141 SPDY_SESSION    fonts.googleapis.com:443 (undefined)
144 SPDY_SESSION    ssl.gstatic.com:443 (undefined)
162 SPDY_SESSION    accounts.youtube.com:443 (undefined)
193 SPDY_SESSION    themes.googleusercontent.com:443 (undefined)
210 SPDY_SESSION    www.google.com:443 (undefined)
239 SPDY_SESSION    translate.googleapis.com:443 (undefined)
gorhill commented 10 years ago

So, I think the outcome here is that there are requests which are not passed to extensions for examination/filtering, and what you report seems to fall into that category.

gorhill commented 10 years ago

I still want to run similar test with Chromium, as the expectation of privacy is higher with Chromium, but I did already find that Chromium, just like Chrome, ping clients?.google.com servers every few hours, probably for extension updates purpose (although I had no extension from the store installed...)

requiredregistration commented 10 years ago

it still connects and checks these even with a about:blank startup, with everything in the settings unchecked. it connects on its own without any user interaction.

gorhill commented 10 years ago

it still connects even with a about:blank startup

You mean to 1e100.net, right?

Edit: Ok, I see that when Chrome connects to clients2.google.com, TCPView reports a connection to [gibberish].1e100.net.

requiredregistration commented 10 years ago

yes.

gorhill commented 10 years ago

yes

So this means HTTPSB can't report these, as per quoted commented code somewhere above (clients?.google.com requests are not seen by extensions).

What is really needed is a wiki page where we can report findings like where behind-the-scene behind-the-scene requests (which cannot be reported by HTTPSB) are made by various Chromium-based browser.

gorhill commented 10 years ago

Here, anybody welcomed to add to this (no special permissions required AFAICT): Privacy matters: Hidden remote connections

gorhill commented 10 years ago

Alright, all requests I've seen so far were requests the browser does not expose to extensions, thus there is nothing this extension can do for these requests.

my-password-is-password commented 10 years ago

I went to chrome://chrome-urls/ and went through each chrome-url, (restarting chromium each time a request was made), and found these url made request to translate.googleapis.com

chrome://chrome/
chrome://extensions/
chrome://help/
chrome://settings/
chrome://translate-internals/

I downloaded the file from

and its just a json file with all the languages used to pouplate the dropdown menu in chrome://settings/addLanguage.

chromelanguage

sl({"sl":{"auto":"Detect language","af":"Afrikaans","sq":"Albanian","ar":"Arabic","hy":"Armenian","az":"Azerbaijani","eu":"Basque","be":"Belarusian","bn":"Bengali","bs":"Bosnian","bg":"Bulgarian","ca":"Catalan","ceb":"Cebuano","zh-CN":"Chinese","hr":"Croatian","cs":"Czech","da":"Danish","nl":"Dutch","en":"English","eo":"Esperanto","et":"Estonian","tl":"Filipino","fi":"Finnish","fr":"French","gl":"Galician","ka":"Georgian","de":"German","el":"Greek","gu":"Gujarati","ht":"Haitian Creole","ha":"Hausa","iw":"Hebrew","hi":"Hindi","hmn":"Hmong","hu":"Hungarian","is":"Icelandic","ig":"Igbo","id":"Indonesian","ga":"Irish","it":"Italian","ja":"Japanese","jw":"Javanese","kn":"Kannada","km":"Khmer","ko":"Korean","lo":"Lao","la":"Latin","lv":"Latvian","lt":"Lithuanian","mk":"Macedonian","ms":"Malay","mt":"Maltese","mi":"Maori","mr":"Marathi","mn":"Mongolian","ne":"Nepali","no":"Norwegian","fa":"Persian","pl":"Polish","pt":"Portuguese","pa":"Punjabi","ro":"Romanian","ru":"Russian","sr":"Serbian","sk":"Slovak","sl":"Slovenian","so":"Somali","es":"Spanish","sw":"Swahili","sv":"Swedish","ta":"Tamil","te":"Telugu","th":"Thai","tr":"Turkish","uk":"Ukrainian","ur":"Urdu","vi":"Vietnamese","cy":"Welsh","yi":"Yiddish","yo":"Yoruba","zu":"Zulu"},"tl":{"af":"Afrikaans","sq":"Albanian","ar":"Arabic","hy":"Armenian","az":"Azerbaijani","eu":"Basque","be":"Belarusian","bn":"Bengali","bs":"Bosnian","bg":"Bulgarian","ca":"Catalan","ceb":"Cebuano","zh-CN":"Chinese (Simplified)","zh-TW":"Chinese (Traditional)","hr":"Croatian","cs":"Czech","da":"Danish","nl":"Dutch","en":"English","eo":"Esperanto","et":"Estonian","tl":"Filipino","fi":"Finnish","fr":"French","gl":"Galician","ka":"Georgian","de":"German","el":"Greek","gu":"Gujarati","ht":"Haitian Creole","ha":"Hausa","iw":"Hebrew","hi":"Hindi","hmn":"Hmong","hu":"Hungarian","is":"Icelandic","ig":"Igbo","id":"Indonesian","ga":"Irish","it":"Italian","ja":"Japanese","jw":"Javanese","kn":"Kannada","km":"Khmer","ko":"Korean","lo":"Lao","la":"Latin","lv":"Latvian","lt":"Lithuanian","mk":"Macedonian","ms":"Malay","mt":"Maltese","mi":"Maori","mr":"Marathi","mn":"Mongolian","ne":"Nepali","no":"Norwegian","fa":"Persian","pl":"Polish","pt":"Portuguese","pa":"Punjabi","ro":"Romanian","ru":"Russian","sr":"Serbian","sk":"Slovak","sl":"Slovenian","so":"Somali","es":"Spanish","sw":"Swahili","sv":"Swedish","ta":"Tamil","te":"Telugu","th":"Thai","tr":"Turkish","uk":"Ukrainian","ur":"Urdu","vi":"Vietnamese","cy":"Welsh","yi":"Yiddish","yo":"Yoruba","zu":"Zulu"},"al":{}})

gorhill commented 10 years ago

The part that may worry users (though that is not what this issue is about) is whether the key parameter can be used to identify uniquely a user. Source code: https://code.google.com/p/chromium/codesearch#chromium/src/components/translate/core/browser/translate_url_util.cc&sq=package:chromium&l=24&type=cs&rcl=1391908708

GuardianMajor commented 10 years ago

For the record, Google Chrome is built with A LOT of phone home operations and interactions with Google services/servers that are NOT exposed to any meaningful capture. This is what makes it "insecure" in the sense that it allows Google to snoop on EVERYTHING you do.

This is why projects such as Comodo's Dragon and SRWare's Iron build directly from the Chromium source ripping out much of that chatty phone home behavior and reporting allowing for a more secure operation. Even these have to allow a very small amount of benign and non-privacy related communication for purposes of the Play Store, Translation feature, Sync, etc.

One point of note though, while Comodo goes to great length to embed their own custom code which some may see just as bad, although many trust Comodo more than Google (personally I think they are equally bad), SRWare's Iron doesn't do that beyond setting the extension gallery to their own custom "store" (but you can still use the store regularly by going there directly) and setting their own homepage which can be easily removed. If you like your hand held, go with Dragon and if you are and advanced user who is pretty independent, then just use Iron.

gorhill commented 10 years ago

The part that may worry users (though that is not what this issue is about) is whether the key parameter can be used to identify uniquely a user.

Answering to my own question, regarding Chromium: found out the API key is a single key shared by all users of Ubuntu (or derived), as seen on line 1539 of (warning, big file) https://launchpadlibrarian.net/163981042/buildlog_ubuntu-saucy-amd64.chromium-browser_32.0.1700.102-0ubuntu0.13.10.1~20140128.970.1_UPLOADING.txt.gz.

cd /build/buildd/chromium-browser-32.0.1700.102/src && GYP_GENERATORS=make GYP_DEFINES="disable_sse2=1 use_third_party_translations=1 werror= sysroot= disable_nacl=1 linux_use_gold_binary=0 linux_use_gold_flags=0 enable_webrtc=1 logging_like_official_build=1 target_arch=x64 component=shared_library linux_use_tcmalloc=0 remove_webcore_debug_symbols=1 linux_dump_symbols=1 use_gnome_keyring=1 linux_link_gnome_keyring=1 use_gconf=1 use_gio=1 google_api_key='AIzaSyAQfxPJiounkhOjODEO5ZieffeBv6yft2Q' google_default_client_id='424119844901.apps.googleusercontent.com' google_default_client_secret='AIienwDlGIIsHoKnNHmWGXyJ' proprietary_codecs=1 build_ffmpegsumo=0 "  python build/gyp_chromium build/all.gyp  -Dgoogle_api_key='AIzaSyAQfxPJiounkhOjODEO5ZieffeBv6yft2Q' -Dgoogle_default_client_id='424119844901.apps.googleusercontent.com' -Dgoogle_default_client_secret='AIienwDlGIIsHoKnNHmWGXyJ'

So it can't be used to uniquely track a computer as far as Chromium is concerned.

requiredregistration commented 10 years ago

no one builds and releases any chromium or chromium-based builds that don't connect to google or somewhere else on their own.

the only solutions are modifying the source code and building it on our own or modifying the binary code.

GuardianMajor commented 10 years ago

@requiredregistration

no one builds and releases any chromium or chromium-based builds that don't connect to google or somewhere else on their own

That's what I said in no uncertain terms.

requiredregistration commented 10 years ago

@GuardianMajor you said to people to get dragon and iron:

If you like your hand held, go with Dragon and if you are and advanced user who is pretty independent, then just use Iron.

gorhill commented 10 years ago

the only solutions are modifying the source code and building it on our own or modifying the binary code

I wouldn't be surprised if there is one or more switches in there to prevent these auto-connections: http://peter.sh/experiments/chromium-command-line-switches/. That would be quite simpler than maintaining a fork.

Of interest: http://peter.sh/experiments/chromium-command-line-switches/#google-apis-url

GuardianMajor commented 10 years ago

@requiredregistration Yeah I did but before that I also said:

This is why projects such as Comodo's Dragon and SRWare's Iron build directly from the Chromium source ripping out much of that chatty phone home behavior and reporting allowing for a more secure operation. Even these have to allow a very small amount of benign and non-privacy related communication for purposes of the Play Store, Translation feature, Sync, etc.

You seem to pick and choose what you want to hear.

requiredregistration commented 10 years ago

@GuardianMajor read the description of the issue and my last two messages again.

you also said:

Even these have to allow a very small amount of benign and non-privacy related communication for purposes of the Play Store, Translation feature, Sync, etc.

that means connections to somewhere behind-the-scene.

gorhill commented 10 years ago

Comodo's Dragon and SRWare's Iron build directly from the Chromium source ripping out much of that chatty phone home behavior and reporting allowing for a more secure operation. Even these have to allow a very small amount of benign and non-privacy related communication for purposes of the Play Store, Translation feature, Sync

I don't understand... Given what I've seen, this is exactly the way my Chromium behaves after I disable the appropriate privacy-related settings, so I don't know what "chatty phone home behavior" Comodo or SWare have "ripped out" from Chromium (can you be more specific?). @requiredregistration's has been talking specifically about what you call "benign and non-privacy related communication for purposes of the Play Store, Translation feature, Sync". There is nothing HTTPSB can do about these, but I think the point is that we would like that even these "benign" connections to be controlled by the user.

GuardianMajor commented 10 years ago

@requiredregistration You once again are missing the point that you CANNOT build off of Chromium without some benign (meaning Play Store, Updates, Sync) features being allowed to work. If you ripped out all that, you will have nothing more than a browser that has NO other functionality. If that's what you want, yes it can be done and good luck doing it as you'd be the only one using it. Iron has achieved the most security without crippling functionality and including their own code, and Dragon has achieved the same but with the inclusion of a lot of their own code.

GuardianMajor commented 10 years ago

@gorhill Ray, in a way he/she is talking about that, because those "benign" communications are the ones that your extension or any other extension cannot intercept as part of the behind-the-scene functionality and that's the point I was trying to make. As their username suggests they are not serious userbase and I am done wasting good effort on trying to explain things to them. its up to you to continue if you feel its worth it.

GuardianMajor commented 10 years ago

@gorhill

I don't understand... Given what I've seen, this is exactly the way my Chromium behaves after I disable the appropriate privacy-related settings, so I don't know what "chatty phone home behavior" Comodo or SWare have "ripped out" from Chromium (can you be more specific?).

The ones you are able to disable are just the public facing options. The code by default tracks what sites you see, what you search, what extensions you have installed, how often you use which, scans your bookmarks for targeted ads based on your browsing behavior and generally snooping functionality that goes to serve Google's pushing of services. For example if you are using Skydrive or Outlook.com you will notice a lot more search results that highlight Google Drive and Gmail. And so on.

requiredregistration commented 10 years ago

@GuardianMajor HTTPSB is a security and privacy extension. think about it.

GuardianMajor commented 10 years ago

@requiredregistration

HTTPSB is a security and privacy extension. think about it.

and once again, your point?

requiredregistration commented 10 years ago

@GuardianMajor when it connects, it sends and receives data, and that without user permission.

gorhill commented 10 years ago

@GuardianMajor

generally snooping functionality that goes to serve Google's pushing of services

I believe you are talking about Chrome here (for which this would certainly not be unexpected).

I am talking about Chromium. As far as I am aware, there is no snooping in Chromium. Currently the issue here is that it connects to Google server at start and then every 2 hours I've observed (presumably for updates) without the user being able to control this (although I may try these switches when I have time).

We can't conflate that whatever Chrome does, Chromium does it too. Chromium is all open source, and whatever speculated claim can be validated by looking at the code. To say that Google "snoops" through Chromium on one's browsing history is such a serious claim, I need a URL to the piece of code which does that (I've check another claim re. Chromium, like the RLZ id, and found it to be unfounded).

GuardianMajor commented 10 years ago

@gorhill

I am talking about Chromium. As far as I am aware, there is no snooping in Chromium. Currently the issue here is that it connects to Google server at start and then every 2 hours I've observed (presumably for updates) without the user being able to control this (although I may try these switches when I have time).

You are correct, that is Chrome. Although Chromium's code is developed by Google as well, and while they don't snoop as much as the Google Chrome build, they do package some of the communication which can be stripped out to minimize it to the basic functionality I have previously stated as "benign" to take place.