search

Lusito / forget-me-not

Make the browser forget website data, except for the data you want to keep.
zlib License
226 stars 25 forks source link

Cookie rules don't apply to all cookies on the domain #124

Closed stevesaliman closed 5 years ago

stevesaliman commented 5 years ago

When I add a rule to Forget Me Not, it doesn't seem to apply to all cookies from that domain. This is making it impossible to login to sites like Yahoo Mail.

Steps to reproduce:

  1. Clear the Forget Me Not log.
  2. Create a rule for *.yahoo.com. My rule is set to "Forget"
  3. Go to mail.yahoo.com
  4. Look at the Forget Me Not log.

What I expect to see are a couple of domains listed with cookies that are allowed. What I actually see is a cookie that was allowed from login.yahoo.com and a blocked cookie from yahoo.com AS. I suspect that the issue lies with the AS that shows up after the domain. I've also seen T, NID, and CGIC after domains, but I have no idea what they mean. I tried looking at the source code, but that didn't shed any light on it either. What do those letters after the domain mean?

Lusito commented 5 years ago

That is certainly weird. Tho for some reason, I can't reproduce, I have a lead where to look. Will get back to you when I have more information.

Lusito commented 5 years ago

Is your fallback rule black? Otherwise I can't see how this would affect your login, since the only code that uses the domain as seen in the log is the cookie blocking feature.

Lusito commented 5 years ago

So, my research says, that in a deprecated standard for the set-cookie header, it was allowed to set multiple cookies in one set-cookie header. This is not allowed anymore, but some gateways may do this to compress http headers in order to reduce data. Since on my end, yahoo sends me multiple set-cookie headers, instead of one, I can only assume, that your gateway is doing this.

Obviously, even tho the gateway should be fixed, I will have to add a fix for FMN as well (since others might have this issue). I will try to implement a fix for this for the next version, but maybe you can check your router if there is some kind of header compression in place for a quick fix.

stevesaliman commented 5 years ago

My fallback rule is indeed "block". I like having precise control over who gets to set a cookie, and most sites don't need them unless I'm logging in to something.

It's interesting that you aren't seeing this on your end - I've seen it both at work and at home - two different gateways. I tried it again on my end, and It looks like I get 2 set-cookie headers, but they each seem to be setting more than one cookie. I'm running dd-wrt at home, I'll see if it does anything with headers.

stevesaliman commented 5 years ago

Upon further review, I don't think I'm getting multiple cookies in a header - I was fooled by cookie values that had a bunch of key value pairs separated by ampersands and confusing things with url parameters :-)

I tried again and have more information, which may or may not help...

The initial load works fine, it is only when I try to enter my password that things get interesting. The password is submitted in a POST, and the response contains several set-cookie headers, one of which sets a cookie named "T". My guess is that when FMN logs domain access for a blocked cookie, it also lets me know which cookie it blocked. Other cookies from the domain get set correctly, just not "T". It is an httpOnly cookie, but so are other cookies that appear to get set. Even more confusing: When I go to preferences, Firefox tells me I have 2 cookies set from yahoo.com, uMatrix tells me I have 4, and the response from my POST has 10.

Lusito commented 5 years ago

This is how the extension blocks cookies: It hooks into the web request, intercepting the HTTP headers and looks for set-cookie headers. The header is then parsed to get the name, the value and the domain for the cookie to be set. The domain and the cookie name are used to find a matching rule.

Multiple key/value pairs in one set-cookie header are separated by comma, at least according to an outdated RFC. After the key-value pairs, a number of extra attributes are separated by semicolon

So a set-cookie header with multiple key/value pairs looks something like: set-cookie: key1=value1, key2=value2; path=/; domain=login.yahoo.com; secure; HttpOnly

I split this by ;, so I would get:

Since you get yahoo.com AS, that would mean that the set-cookie header looks like this: set-cookie: key1=value1, key2=value2; path=/; domain=login.yahoo.com AS; secure; HttpOnly

Which doesn't make much sense to me, so maybe it isn't about multiple key/value pairs after all. Could you either show me other websites where this happens, where I can reproduce this, or show me the http headers in question? Strip out the value parts of it before posting! I.e. everything between the first = and the first ;. You can mail the headers to core [at] lusito.info as well if you don't want to post it online. If there are multiple key=value, separated by comma, keep the key= part including comma as well.

Thanks for your patience with this.

practik commented 5 years ago

I just tried logging in to Yahoo and am only seeing yahoo.com and mail.yahoo.com in the FMN log. If I check cookies, I see there is one from Yahoo named T, and in the past I know I've seen also seen ones named AS and NID, but I've never seen anything in the FMN log like what you're describing, @stevesaliman. I don't have a special FMN rule for Yahoo, though; it's covered by my default rule, which is Forget.

stevesaliman commented 5 years ago

Yahoo works for me as well when the default rule is "Forget", but when I add a rule for *.yahoo.com and change the default rule to "Block", Yahoo mail stops working.

I'll send you the headers I get when I get a chance. I'll focus on the "T" cookie since that is one that seems to be giving us problems.

stevesaliman commented 5 years ago

Here are all the set-cookie headers I see in the response to the POST of the password:

set-cookie: B=value; expires=Wed, 08-Jan-2020 21:50:32 GMT; path=/; domain=.yahoo.com
set-cookie: T=vaue; expires=Wed, 08-Jan-2020 21:50:32 GMT; path=/; domain=.yahoo.com; secure; HttpOnly
set-cookie: F=value; expires=Wed, 08-Jan-2020 21:50:32 GMT; path=/; domain=.yahoo.com; HttpOnly
set-cookie: PH=value; expires=Wed, 08-Jan-2020 21:50:32 GMT; path=/; domain=.yahoo.com
set-cookie: Y=value; expires=Wed, 08-Jan-2020 21:50:32 GMT; path=/; domain=.yahoo.com; secure
set-cookie: FS=value; expires=Wed, 08-Jan-2020 21:50:32 GMT; path=/; domain=.login.yahoo.com; secure; HttpOnly
set-cookie: SSL=value; expires=Thu, 01 Jan 1970 22:00:00 GMT; path=/; domain=.yahoo.com; secure; HttpOnly
set-cookie: AO=value; expires=Sat, 08 Jan 2039 18:05:00 GMT; path=/; domain=.yahoo.com
set-cookie: GUCS=value; Expires=Tue, 8 Jan 2019 22:20:47 GMT; Domain=.yahoo.com; Path=/; Secure
set-cookie: AS=value; path=/; domain=login.yahoo.com; secure; HttpOnly

Before cutting out the values of each cookie, I verified that there were no commas in them.

Lusito commented 5 years ago

All of these get parsed correctly. "yahoo.com AS" should not have happened with the headers above, making this really confusing. Maybe the value somehow causes issues while parsing, but I can't imagine any reason how it could.

I will have to take a fresh look at this issue tomorrow.

Lusito commented 5 years ago

What Firefox Version are you on?

stevesaliman commented 5 years ago

I've tried this on two different boxes. The one that generated the headers above is version 60.4.0esr on a Linux box. The one at home is on a Windows 10 box. I'll look at the version number when I get home.

It also seems that the cookie getting blocked changes. It was the "yahoo.com AS" cookie when I tried it at home when I first reported the issue, it is the "yahoo.com T" cookie today.

stevesaliman commented 5 years ago

I just tried logging into yahoo from the home PC. It is running Firefox 61.0.1 in Windows 10, and I'm seeing the same behavior I saw this afternoon from the Linux box - the "yahoo.com T" cookie is being blocked.

And I noticed something else: The "T" cookie's value is 552 characters long, and "AS" is 738. No other cookie's values reach 200 characters. Could there be some sort of length issue?

Lusito commented 5 years ago

Max cookie length is 4k on most browsers, so even 738 should be just fine.

Could you check if the issue is fixed with the following version? Please read #126 first for instructions http://lusito.info/fmn-sch-2.0.0.xpi

This uses an external library (used by many others) to parse the set-cookie header. So if there is something wrong with my parser, this should fix it. It might not be the most efficient approach tho, so if this works I might need to give this a closer look.

stevesaliman commented 5 years ago

I didn't install the developer version of Firefox, hopefully that is ok, also I had to set xpinstall.signatures.required to false instead of true to get the plugin to install.

I no longer see the "AS" or "T" entries in the FMN log, but I still can't login to Yahoo mail unless my default rule is "On Leave". It doesn't make much sense. When I login with "On Leave" as the default rule, I see 8 cookies for yahoo.com, 2 for login.yahoo.com, and 1 for mail.yahoo.com. When I make the default rule "Instantly" with separate rules for .yahoo.com, I only see one cookie for the yahoo.com domain. Adding separate rules for .mail.yahoo.com and *.login.yahoo.com doesn't change anything. The FMN log doesn't show anything blocked, but I only see one cookie for yahoo.com. It would seem that something is being blocked that doesn't get logged, but I can't figure out what it might be.

For completeness, I added *.oath.com to the rule set because I've seen that domain popup with Yahoo mail before.

Any ideas? Anything in particular I could check in the headers?

Lusito commented 5 years ago

Well, maybe then you are just missing a domain that needs to be set. Some websites use third-party domains to keep login information and without them you'll get logged out.

Try the following: Install a cookie manager extension, which allows you to list and delete cookies, for example: https://addons.mozilla.org/en-US/firefox/addon/cookie-quick-manager/

Restart the browser and check which cookies are set.. I'm guessing with your rule-set that it's probably very few, if any at all. Then make your fallback rule white or gray and log in to yahoo. Now check what cookies are set. There might be cookies, which are on another domain, which yahoo still needs to log in correctly. Try deleting the ones you think are not required and see if you are still logged in. At some point you won't be logged in anymore and you'll know which domain is required to stay logged in.

Lusito commented 5 years ago

About the missing cookies you noticed: I'm guessing, that this is due to yahoo deleting them upon logout.

If the above doesn't help, I could try to create another test version, which shows you popups for blocked cookies, just like it does with removed cookies.

stevesaliman commented 5 years ago

The mystery deepens :-)

I installed quick cookie manager and logged into yahoo mail. While logged in, I looked at the cookies and saw the same 11 cookies I saw yesterday when I used the "manage data" option in Firefox itself. The only thing I can think of is that there is some cookie being set while I'm logging in that gets cleared before the login is complete. I'll have another look at the headers and see if anything jumps out at me.

I don't think we need a popup for blocked cookies - aren't all the domains that have cookies blocked listed in the FMN log window?

stevesaliman commented 5 years ago

I tried to do a deeper analysis of the cookies used to get to my Yahoo inbox. I did the following:

  1. Clear the FMN log window and the Firefox Developer Tools Network tab.
  2. Go to mail.yahoo.com
  3. Enter my username and hit "Next".
  4. Enter my password and hit "Sign In".
  5. When my inbox came up, I looked at cookies and headers using different tools.

Here are the results: FMN's log window shows 4 domains: yahoo.com, mail.yahoo.com, login.yahoo.com, and guce.yahoo.com. All 4 of these domains have explicit rules set to "On Leave"

Firefox's own "Manage Data" window shows 1 cookie for mail.yahoo.com, 2 for login.yahoo.com, and 8 for yahoo.com.

Quick cookie Manager shows the same counts, BUT the domains are subtly different. It shows mail.yahoo.com, .login.yahoo.com, and .yahoo.com - the last two start with dots.

I then examined all of the response headers for all requests made during my test. I found the following cookies set by the following domains:

I noticed a few things when I looked at the data from all the sources.

  1. The SSL cookie for .yahoo.com is present in a response header, but does not show up in any of the other tools. It is set httpOnly=true and secure=true, but so is the T cookie, and it shows up elsewhere.
  2. FMN's log has an entry for guce.yahoo.com domain, but no cookies were set for that domain. There is a request for that domain, and it sets the GUC cookie, but with a domain of yahoo.com.
  3. 9 of the cookie domains start with a dot. I can't help but wonder if that dot has something to do with this.
  4. When I change my default rule to "Immediately", the only cookie that gets set is the "B" cookie for .yahoo.com, and I don't get past the post for my username. I have "On Leave" rules for all 4 domains mentioned in the FMN log.
  5. All requests went to one of the following domains: mail.yahoo.com, login.yahoo.com, and s.yimg.com - which doesn't set any cookies.

Hopefully this info helps out...

Lusito commented 5 years ago

Thanks for the information. The dot in front of the domain simply says, that cookies can be read by subdomains. For example:

All this trial and error just shows me that the the current logging method isn't as helpful as it should be. My current idea to change it to an action-based log like this:

I'm sorry this takes so long, but without having a method of reproducing the situation on my end, it's hard to identify the issue. I will give this another go tomorrow. My time is currently a bit limited because I am moving to a different city in about two weeks (again..).

stevesaliman commented 5 years ago

No problem, I appreciate the help with the issue.

I am curious, though - do you have a Yahoo account, and if so, can you login to it with a default rule of "Immediate"? If so, it would be interesting to figure out the difference in our environments.

Lusito commented 5 years ago

No, I don't have an Yahoo account. I'm hesitant to setting one up since it requires me to enter my phone number to proceed and I didn't have the nerve to read the extralong eula in order to avoid hidden costs. I guess I'm a bit paranoid in that way since I've seen friends get burned by "free" mail providers.

stevesaliman commented 5 years ago

I completely understand. I won't give them my phone number either - I got my account long before they started requiring phone numbers :-)

Lusito commented 5 years ago

FIY, I wasn't able to spend more time on this and for the next couple of days I'll be moving. Not sure when my internet will be ready. I'll try to get back on this as soon as possible.

Sorry about this.

remitamine commented 5 years ago

same problem happend for me on LinkedIn, the server actually send separate Set-Cookie headers for every cookie but for some reason the parseSetCookieHeader function receive multiple cookies separed by line feeds like this:

KEY1=VAL1; Path=/\nKEY2=VAL2; Path=/; Domain=.www.linkedin.com\nlang=v=2&lang=en-us; Path=/; Domain=linkedin.com\nKEY3=\"VAL3\"; domain=.linkedin.com; Path=/; Expires=Sun, 07-Feb-2021 21:12:53 GMT\nKEY4=\"VAL4\"; domain=.www.linkedin.com; Path=/; Secure; Expires=Sun, 07-Feb-2021 21:12:53 GMT; HttpOnly\nKEY5=\"VAL4\"; Expires=Sat, 09 Feb 2019 09:33:06 GMT; domain=.linkedin.com; Path=/

this breaks the matching for the domain and also discard the rest of the cookies, for example the matched domain will be .www.linkedin.com\nlang instead of .www.linkedin.com, and KEY2 to KEY5 are not extracted either. i'm testing directly on the latest develop branch on Firefox Developer Edition 66.0b5, just modified the npm-run-all version to ^4.1.5 to fix this error:

npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/event-stream/-/event-stream-3.3.6.tgz
npm ERR! 404 
npm ERR! 404  'event-stream@3.3.6' is not in the npm registry.
npm ERR! 404 Your package name is not valid, because 
npm ERR! 404  1. name can only contain URL-friendly characters
npm ERR! 404 It was specified as a dependency of 'forget-me-not-web-extension-builder'
npm ERR! 404 
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

npm ERR! A complete log of this run can be found in:
remitamine commented 5 years ago

a hacky fix for the problem:

diff --git a/src/background/headerFilter.ts b/src/background/headerFilter.ts
index a2e260e..251b996 100644
--- a/src/background/headerFilter.ts
+++ b/src/background/headerFilter.ts
@@ -52,9 +52,12 @@ export class HeaderFilter {
         return responseHeaders.filter((x) => {
             if (x.name.toLowerCase() === "set-cookie") {
                 if (x.value) {
-                    const cookieInfo = parseSetCookieHeader(x.value, fallbackDomain);
-                    if (cookieInfo && this.shouldCookieBeBlocked(tabId, cookieInfo))
-                        return false;
+                    x.value = x.value.split('\n').filter((x) => {
+                        const cookieInfo = parseSetCookieHeader(x, fallbackDomain);
+                        if (cookieInfo && this.shouldCookieBeBlocked(tabId, cookieInfo))
+                            return false;
+                        return true;
+                    }).join('\n');
                 }
             }
             return true;
Lusito commented 5 years ago

Thanks for the info, a linkedin account I do have and I do see the cookies on multiple lines in the headerFilter!

Two weird things I noticed though:

  1. inspecting the raw headers in the Firefox developer tools network tab shows each set-cookie on its own line with set-cookie prefix.
  2. I do not see the anything in the log like the yahoo.com AS that @stevesaliman had in his yahoo example.
Lusito commented 5 years ago

Just tried it on linkedin, splitting by line seems to fix the issue.

@stevesaliman could you try this version to see if it fixes your issue on yahoo as well? http://lusito.info/fmn-sch-2.0.0.xpi

stevesaliman commented 5 years ago

This latest version of the add-on appears to fix the Yahoo issue as well, and the log no longer shows yahoo.com AS like it did before, just the normal domains we'd expect to see. Like you I also saw each cookie in its own line.

Thank you for your efforts on this issue.

Lusito commented 5 years ago

Thanks for your patience. Credits go to @remitamine though for solving this puzzle.