search

uBlockOrigin / uMatrix-issues

This is the community-maintained issue tracker for uMatrix
https://github.com/gorhill/uMatrix
123 stars 17 forks source link

Cookies are still deleted despite lower domain-level allowance #277

Open Kein opened 4 years ago

Kein commented 4 years ago

Prerequisites

Description

Cookies deleted on timed manner even for allowed domains

A specific URL where the issue occurs

Any. In this particular case forums.unity.com

Steps to Reproduce

  1. Go to forums.unity.com (by any resource, really)
  2. Login
  3. Close the tab and monitor the logger for info events

Ruleset

Global scope level: domain

cname-reveal: * true
https-strict: behind-the-scene false
matrix-off: about-scheme true
matrix-off: behind-the-scene true
matrix-off: chrome-extension-scheme true
matrix-off: chrome-scheme true
matrix-off: localhost true
matrix-off: moz-extension-scheme true
matrix-off: opera-scheme true
matrix-off: vivaldi-scheme true
matrix-off: wyciwyg-scheme true
no-workers: * true
no-workers: youtube.com false
noscript-spoof: * true
referrer-spoof: * true
referrer-spoof: behind-the-scene false
* * * block
* * css allow
* * frame block
* * image allow
* 1st-party * allow
* 1st-party cookie block
* 1st-party frame allow
* 1st-party script block
127.0.0.1 1st-party script allow
unity.com 1st-party cookie allow
unity.com 1st-party script allow

Supporting evidence

image

image

Your environment

Additional notes

According to the wiki and option description:

Blacklisted cookies are not prevented by uMatrix from entering your browser. However they are prevented from leaving your browser, which is what really matters. Not blocking cookies before they enter your browser gives you the opportunity to be informed that a site tried to use cookies, and furthermore to inspect their contents if you wish.
Once these blacklisted cookies have been accounted for by uMatrix, you can ask uMatrix to remove them from your browser if you wish so: just check the setting "Delete blocked cookies" in the Privacy tab.

I read it multiple times trying to imagine all possible meanings I might be missing (which is already very bad thing to begin with - option description shouldn't call for a meeting of wise men to debate what this or that means) but in the end I always come back to what seems logical and most reasonable - these domains that do not have rule allowing cookies out will be cleared. My understanding is that when this feature was developed it sure was thought thru to flatten-out rules to make sure all checks are passed not just top level * * * block, so if I block cookies globally, but then allow for domain.com via 1st party rule - this will be the final flattened rule that is taken into account and therefore on the next iteration of a clean-up the cookie will be kept.
However, according to my observation and logger data that is not the case. Local storage also suffers form the same issue.

P.S. It would be nice to add a clarification how are allowed 3d party cookies processed. For example, if I allow google.com 3dparty cookies for youtube.com but block 1st party cookies on google.com - what the action will be performed during deletion, which will be kept or deleted? P.P.S. Screenshot of the switchboard says 1.4.1b0 - ignore that, the shots are made on that version but later after an update on b6 the same issues were observed so it was tested on b6 as well.

gwarser commented 4 years ago

These blue entries in the logger are not what uMatrix is doing. These are informative entries - they log everything is happening with cookies. Cookies removed by uM will have uppercase COOKIE in type column and will be red.

Also, your screenshot shows "{session-cookie:...}". Session cookies are temporary - they should disappear when you close browser at the latest (end of browsing "session"), but may also be removed when tab is closed.

Kein commented 4 years ago

The only extension I have that manages cookies is uMatrix. I tested it extensively with other extensions and standalone for about 2 weeks.
Last 2 days I started to test how cookie management works. This is the first time this year I've got logged out from majority of domains (including inoreader which is like never expires for centuries) despite the fact ALL of them were allowed for 1st party cookies.

If you are stating I'm wrong in my conclusion do elaborate. You are saying my conclusion is invalid despite the fact the only thing that's changed is uMatrix global option for cookie deletion. If I did the test wrong tell me how could I test it properly? Is there a way for me to trigger the clean up even from extension background page console? Any other way?

The issue is real on my end and I'm willing to put time to investigate it.

Kein commented 4 years ago

Cookies removed by uM will have uppercase COOKIE in type column and will be red.

Are you sure? I'm seeing only one message for cookie log:

https://github.com/gorhill/uMatrix/blob/master/src/js/cookies.js#L270 https://github.com/gorhill/uMatrix/blob/9b292304d33a44465922200efa5f8b378d0f9604/src/_locales/en/messages.json#L311

UPDATE: I went through all source code related to cookies and all locale messages and the deletion messages and nothing of what you say matches current codebase. There is a generic cookie removal line for every cookie removal. cookie.js manager fallbacks for some vAPIfunctional but it is pretty basic.

Given the circumstances I'd appreciate if you could unlock the issue instead of straight jumping into assumptions. Thank you.

gwarser commented 4 years ago

nothing of what you say matches current codebase

You can be right.

uMatrix version: 1.4.10b6

BTW, I was talking about this: https://github.com/gorhill/uMatrix/blob/0bcb7669e77adc958ee66a97fe9172898cb8131d/src/js/cookies.js#L496

gorhill commented 4 years ago

The logger reports cookies which may or may not have been removed by uMatrix -- you can verify this by deleting the cookies using the browser's own UI. The COOKIE entries is to denote removed outgoing COOKIE headers.

So far I have been unable to reproduce the reported issue on my side. I am not using unity.com to try to reproduce, but I tried with logged in https://news.ycombinator.com/news and not logged in https://github.com/brave/brave-browser/issues/9929 (which still delivers session cookeis).

Kein commented 4 years ago

BTW, I was talking about this: https://github.com/gorhill/uMatrix/blob/0bcb7669e77adc958ee66a97fe9172898cb8131d/src/js/cookies.js#L496

I saw this, yeah. Problem is, I have a tons of cookies ready to be expired yet the only cookies that are being removed in the log is the one I've visited recently. I've got logged out out of Inoreader.com again just as I'e been active here, on in this issue, even though I've explicitly set every first party domain to allow. From my understanding this SHOULD not happen.

Here are my cookies for Inoreader.com

image

The logger for All Tabs with filter info had only one message entry stating that only consents_cookie_use was removed, but it makes no sense, that cookie expires in like 2 years, I have no more extensions that delete or manage cookies and browser settings set to always keep 1st party cookies. I think it might have swallowed other log entries, not sure. It really makes no sense to me.

userSettings:

alwaysDetachLogger: false
autoUpdate: true
clearBrowserCache: true
clearBrowserCacheAfter: 120
cloudStorageEnabled: false
collapseBlacklisted: true
collapseBlocked: false
colorBlindFriendly: false
deleteCookies: true
deleteLocalStorage: false
deleteUnusedSessionCookies: false
deleteUnusedSessionCookiesAfter: 60
displayTextSize: "13px"
externalHostsFiles: []
externalRecipeFiles: []
iconBadgeEnabled: true
noTooltips: false
popupCollapseAllDomains: false
popupCollapseBlacklistedDomains: false
popupScopeLevel: "domain"
processHyperlinkAuditing: true
selectedHostsFiles: Array(6)
0: "malware-0"
1: "malware-1"
2: "dpollock-0"
3: "hphosts"
4: "mvps-0"
5: "plowe-0"
selectedRecipeFiles: Array(1)
0: "recipes_en-0"
length: 1
userHosts:
content: ""
enabled: false
userRecipes:
content: ""
enabled: false
gorhill commented 4 years ago

What is your setting regarding "Delete non-blocked session cookies [60] minutes after the last time they have been used"?

Kein commented 4 years ago

It is false, I've updated previous comment with raw saved values straight from background page. As a matter of fact it never was enabled, even for testing (didnt come to this yet).

gorhill commented 4 years ago

What are your rules re. inoreader.com?

Kein commented 4 years ago

Base:

cname-reveal: * true
https-strict: behind-the-scene false
matrix-off: about-scheme true
matrix-off: behind-the-scene true
matrix-off: chrome-extension-scheme true
matrix-off: chrome-scheme true
matrix-off: localhost true
matrix-off: moz-extension-scheme true
matrix-off: opera-scheme true
matrix-off: vivaldi-scheme true
matrix-off: wyciwyg-scheme true
no-workers: * true
no-workers: youtube.com false
noscript-spoof: * true
referrer-spoof: * true
referrer-spoof: behind-the-scene false
* * * block
* * css allow
* * frame block
* * image allow
* 1st-party * allow
* 1st-party cookie block
* 1st-party frame allow
* 1st-party script block
127.0.0.1 1st-party script allow

Were at the moment of creation of this issue:

inoreader.com 1st-party cookie allow
inoreader.com 1st-party script allow

Now I've explicitly set every single 1st party domain

inoreader.com inoreader.com cookie allow
inoreader.com www.inoreader.com cookie allow

However I had .steampowered.com being axed as well despite explicitly allowed.

gorhill commented 4 years ago

Is stepping into the code using the debugger something you are comfortable to do? There are only two places where uMatrix explicitly asks the browser to remove cookies, so maybe putting a breakpoint at these lines will provide useful additional info:

Kein commented 4 years ago

Yes, I could do that, will do when I get chance, I will update the issue.
I'm trying to understand this tho: https://github.com/gorhill/uMatrix/blob/b26b3bb9609f9eae1e3ace48170b5da1aa2fdf98/src/js/cookies.js#L404 and it looks like there is some redundancy that makes it a bit confusing.

Kein commented 4 years ago

I opened browser today and got logged out from forums.unity.com again. Rules havent changed: image

Hit a few breakpoints but mustAllow always returned true for scrHostname = forums.unity.com and cookieHostname = unity.com image

I will keep looking

gorhill commented 4 years ago

If you want to put a debugger instruction, probably best to put it before both loops (after const cookieHostname...), something like if ( cookieHostname.endsWith('unity.com') ) { debugger; } since true will be returned when no match is found, which is what you are trying to track down.

Kein commented 4 years ago

Haven't caught anything with breakpoint watch, but caught this one removed today again:
14:32:27​cookie deleted: https://store.steampowered.com/{session-cookie:steamLoginSecure}
image
Rules:
image

This shouldnt happen until at least a month when "Remember Login" is set. This never happened without uMatrix.

gorhill commented 4 years ago

It could be the site's own code which remove the cookie as a result of blocking something.

Kein commented 4 years ago

It could be the site's own code which remove the cookie as a result of blocking something.

Hmm, I see how that can make sense, but shouldnt there be a way with uMatrix to figure out what causes that ?

gorhill commented 4 years ago

Not really, any javascript code anywhere can do this. Are you spoofing referrer? This could be the issue.

Kein commented 4 years ago

I can see how referer could be some kind of issue with unity forums as they use unityID portal for ID, but definitely not with few others I had the same issue with, like steam or some very basic plain phpBB forums.

I will disable and test without referrer spoof for these.

gwarser commented 4 years ago

But this is again session cookie - it can be removed by browser at any time after you close the tab.

Kein commented 4 years ago

Sure, but these are resources I visit the most frequent and on daily basis and the aforemention issue never manifested without uMatrix. So this is either cookie cleanup logic or something else with uMatrix not accounted for in terms of permisisons (?).

gorhill commented 4 years ago

something else with uMatrix

As already stated:

There are only two places where uMatrix explicitly asks the browser to remove cookies [...]

If your breakpoint was not hit and if we are going to speculate, it's best to speculate the server itself or the site's client code is doing this. I have seen case of referrer-spoofing causing logging out of accounts in the past.

Kein commented 4 years ago

Nope, not the referrer, got logged out again from steampowered.com and unity.com

Kein commented 4 years ago

¯\_(ツ)_\/¯ Okay, 24hrs test with any cookie deletion mechanism off and the issue is still here.
I will test now referrer off, cache deletion off and cookie deletion off. If it still persist the I suspect it might be related to some late (?) cookie processing by uMatrix when the resource is not receiving it in time fast enough and thus issue new cookie? So if I had session I saved but it didnt get it by the document_end then it might reissues new temp and log me out.

Remu-rin commented 4 years ago

If it still persist the I suspect it might be related to some late (?) cookie processing by uMatrix when the resource is not receiving it in time fast enough and thus issue new cookie?

Then try testing with cookie column allowed globally.

Kein commented 4 years ago

Test that and tested this:

Then try testing with cookie column allowed globally.

Still the same issue.

To reiterate, while the problem is extremely annoying, I dont think that for me it quite tips yet the scales between "fine-grained control of what I allow on web-page" vs "I now have to login every time on half of resources". I can live with that. That being said, I still think this is valid issue, even if not an obvious and easy to track one. In my opinion, just because something is complex in nature does not mean it by definition should be pain in ass to use ("pleb filter") or should encourage to tolerate such small/medium sized annoyances because "all big boys endure, aren't you big boy?".

Keep this issue open until someone else figures out the way to trace it?

gorhill commented 4 years ago

One thing I could improve is reporting in the logger whether uBO itself removed a cookie or that a cookie was removed without uBO.

Kein commented 4 years ago

That would be welcome addition either way, yes, however I think all the testing I've done shows it is no longer related to cookie deletion as I thought initially (neither referrer related), see https://github.com/uBlockOrigin/uMatrix-issues/issues/277#issuecomment-657079821

Kein commented 4 years ago

Small update to the issue - someone else who uses uMatrix mentioned that they always have their cookies preferences reset on https://www.g2a.com At the bottom of the page there is a way to setup language and currency and with rules like this:

g2a.com 1st-party cookie allow
g2a.com 1st-party script inherit

when they return to the site after a day or two they would get these value reset to default detected by the IP origin. Normal behavior is that regardless if you are logged in or not the settings persist. May be this can be useful testing source, I dont know.