search

uBlockOrigin / uBlock-issues

This is the community-maintained issue tracker for uBlock Origin
https://github.com/gorhill/uBlock
944 stars 80 forks source link

uBlock applying blocking rules for websites it is turned off #3083

Closed ParticleCore closed 6 months ago

ParticleCore commented 9 months ago

Prerequisites

I tried to reproduce the issue when...

Description

This issue is in keep with recent user reports on social media stating that even after turning off uBlock for a specific website it is still being detected, very likely as a direct result of this bug that is causing the extension to still apply blocking rules for websites it was turned off.

Currently using uBlock 1.55.0 with default settings and rules on Firefox 121.0.1 and 123.0a1

I have it turned off for youtube.com, for example image

I have created a brand new profile with the only extension installed being uBlock

With uBlock turned off for youtube.com the ads are still being blocked

Debugging the page confirms that it is still injecting blocking rules: image

Disabling the extension in the browser addon manager and refreshing youtube resolves the problem, no more blocking rules appear in the website.

A specific URL where the issue occurs.

https://www.youtube.com/watch?v=oACoGd4AZwk

Steps to Reproduce

Install uBlock on a fresh profile Open any url video Open the debugger and look for the scriplet blocking rules

Expected behavior

When uBlock is turned of for a specific website no blocking rules should be applied

Actual behavior

When uBlock is turned of for a specific website, blocking rules are still being applied

uBO version

1.55.0

Browser name and version

Firefox 121.0.1 and 123.0a1

Operating System and version

Windows 10

gorhill commented 9 months ago

Unable to reproduce, when I turn off uBO and reload the Youtube page, I don't see any uBlockOrigin +js() scriptlets in the debugger:

uBO enabled: image

uBO disabled and page reloaded: image

What is your "4 results" screenshot showing? How did you get this?

Similarly I cannot reproduce with a simpler scenario:

gwarser commented 9 months ago

Injected scriptlets will not disappear if you navigate from one video to another. YT is a long living page - content is dynamically updated by JS, you need to manually reload or open in new tab to load fresh scripts without modifications.

gorhill commented 9 months ago

Regarding the issue with a link to here:

In order to confirm if Iridium is the reason for any slowness/lag/performance issues on YouTube

Firefox has a good profiler tool to sort out performance issues: https://profiler.firefox.com/

I did experience at some point what appeared to be performance issue in Firefox, with no extension enabled. I submitted a profiling session, see https://bugzilla.mozilla.org/show_bug.cgi?id=1874862.

More such profiling data need to be submitted if we want this issue to be investigated by Firefox devs.

ParticleCore commented 9 months ago

Unable to reproduce, when I turn off uBO and reload the Youtube page, I don't see any uBlockOrigin +js() scriptlets in the debugger:

uBO enabled: image

uBO disabled and page reloaded: image

What is your "4 results" screenshot showing? How did you get this?

Similarly I cannot reproduce with a simpler scenario:

* Add `example.com##+js(set, ubo, true)` in "My filters"

* Open `http://example.com/`

* Open dev console

* Verify that `ubo` is `true`

* Disable uBO, reload page

* `ubo` is `undefined`

Does not happen all the time, in my test runs (about 50) it happened 80% of the times. Will keep an eye on it to see if there is a way to trigger it.

ParticleCore commented 9 months ago

Injected scriptlets will not disappear if you navigate from one video to another. YT is a long living page - content is dynamically updated by JS, you need to manually reload or open in new tab to load fresh scripts without modifications.

Although I did not explicitly stated in the first steps, I did force refresh every time in between toggling the setting in uBlock

ParticleCore commented 9 months ago

Regarding the issue with a link to here:

In order to confirm if Iridium is the reason for any slowness/lag/performance issues on YouTube

Firefox has a good profiler tool to sort out performance issues: https://profiler.firefox.com/

I did experience at some point what appeared to be performance issue in Firefox, with no extension enabled. I submitted a profiling session, see https://bugzilla.mozilla.org/show_bug.cgi?id=1874862.

More such profiling data need to be submitted if we want this issue to be investigated by Firefox devs.

Did consider that, but as soon as uBlock was disabled in the addons manager the performance issues were gone. Unless the browser itself was causing the issues while the extension was enabled, then I can't think of any other way to explain it.

Was not able to correctly confirm (YT scripts keep changing every now and then) but the behavior of that issue made me believe that the way uBlock was tampering with the ad data on the website was causing the YT scripts to enter into a error/retry loop which would eventually time out, causing the video to stop loading and show the circle icon on the video.

From my experience playing with their code, it felt like part of their ad logic has instructions on how many are loaded and when to show an ad, and when it tries to pull the information for that ad (say 30s into the video) it never finds it, but since it is expecting to find something then it stays in that suspended state forever.

Problem is, since their scripts keep changing almost daily, and with it some of their logic, it is hard to pinpoint the actual issue.

stephenhawk8054 commented 9 months ago

You need to provide profiler data if you see performance issue. Otherwise, there's nothing to investigate.

gorhill commented 9 months ago

as soon as uBlock was disabled in the addons manager the performance issues were gone

If you experience performance issue you think are caused by uBO, please provide profiling data, just presuming doesn't lead anywhere except endless misdagnosis.

gorhill commented 9 months ago

Does not happen all the time, in my test runs (about 50) it happened 80% of the times

What is the test run? You (enable uBO => reload page => disable uBO => reload page) repeatedly? Or do you just disable uBO once then reload repeatedly?

ParticleCore commented 9 months ago

What is your "4 results" screenshot showing? How did you get this?

Forgot to address this. When the issue started happening I noticed the ad data was gone, when I tried to inspect it I stumbled on the ad data points overridden with constants, which then led me to those scriplets still running after the extension was turned off for that website and still persisted after page refresh. Only after disabling the extension in the addons manager the scriplets were no longer present.

Just searched for parts of the code to try figure out where it was coming from, nothing special: image

If you experience performance issue you think are caused by uBO, please provide profiling data, just presuming doesn't lead anywhere except endless misdagnosis.

Sorry, hard to scope that one. The lag occurs briefly in the first few seconds of the video playback until it hits the 30s-ish mark then the video stops playing, does not buffer anymore, and the loading circle shows up on the player and stays like that. Most of the times when I am fast enough, I can tell when the issue is happening by checking in the player stats for dropped frames, it always drops frames when the issue occurs.

I don't believe the performance problems are being caused by uBlock directly, but I do believe its scriplets/code modifications made by it induces the YT code into a state that causes performance issues, at the very least with the YT scripts that were loading for me at that time.

What is the test run? You (enable uBO => reload page => disable uBO => reload page) repeatedly? Or do you just disable uBO once then reload repeatedly?

Open YT Install uBlock Refresh YT Turn off uBlock for YT Reload YT disable uBlock for YT

Sometimes it looks like the issue is not happening in the first 30s, but some of those times it happens 1:30s in. This one just happened right now (with uBlock enabled for YT) to me, and stays like this: 1

Relevant YT scripts loaded for the above specific test are as follows:

player https://www.youtube.com/s/player/787e9b63/player_ias.vflset/en_US/base.js ui builder https://www.youtube.com/s/desktop/80338919/jsbin/desktop_polymer.vflset/desktop_polymer.js navigation https://www.youtube.com/s/desktop/80338919/jsbin/spf.vflset/spf.js

Those might change at any time though, but just in case they can be helpful I included them here.

stephenhawk8054 commented 9 months ago

Profiling data means using https://profiler.firefox.com/

ParticleCore commented 9 months ago

Profiling data means using https://profiler.firefox.com/

I meant to say it is hard to pin the performance issue on uBlock since it is most likely YT that is having performance issues as a result of uBlock modifications.

stephenhawk8054 commented 9 months ago

That's why you need profiler data. It will show what the website is loading. It's NOT just for pointing the issue to uBO.

gorhill commented 9 months ago

Alright I am going to try to gather all the detailed steps to repro in one place, tell me if I get this right:

  1. New Firefox profile
  2. Open https://www.youtube.com/watch?v=oACoGd4AZwk
  3. Install uBO
  4. uBO should be enabled for YT (default after install)
  5. Reload the YT tab
  6. Disable uBO for youtube.com
  7. Reload the YT tab
  8. Let the video play

Result: Eventually issue when repeating 1 to 8? Or is it 4 to 8?

Correct?

ParticleCore commented 9 months ago

Alright I am going to try to gather all the detailed steps to repro in one place, tell me if I get this right:

1. New Firefox profile

2. Open `https://www.youtube.com/watch?v=oACoGd4AZwk`

3. Install uBO

4. uBO should be enabled for YT (default after install)

5. Reload the YT tab

6. Disable uBO for `youtube.com`

7. Reload the YT tab

8. Let the video play

Result: Eventually issue when repeating 1 to 8? Or is it 4 to 8?

Correct?

The current issue for which I opened was the uBlock modifications were still being applied after it was turned off via this setting: image

And refreshing YT afterwards.

The other issue that made me find this was the performance issue, which was happening almost every time I opened YT with uBlock on (or off, but the blocking rules were somehow still being applied to YT).

Unfortunately I am now having a hard time making it happen again after trying to use the profiler recorder.

One last thing is yesterday this was not a problem at all, I worked for over 12h and never noticed this happening once using the exact same setup as I used today - uBlock was working correctly, no YT performance issues and no blocking rules were "sticking" on the website after turning it off in uBlock.

ParticleCore commented 9 months ago

I did experience at some point what appeared to be performance issue in Firefox, with no extension enabled. I submitted a profiling session, see https://bugzilla.mozilla.org/show_bug.cgi?id=1874862.

Beginning to wonder if it is all tied to that one somehow.

gorhill commented 9 months ago

And refreshing YT afterwards

The part that is not clear is you said:

in my test runs (about 50)

What were the exact repeated steps in these runs? It's what I am trying to figure before spending time to try and reproduce the issue.

ParticleCore commented 9 months ago

And refreshing YT afterwards

The part that is not clear is you said:

in my test runs (about 50)

What were the exact repeated steps in these runs? It's what I am trying to figure before spending time to try and reproduce the issue.

Apologies, that completely went over my head.

  1. Opened Firefox with a temporary new profile
  2. Opened YT
  3. Installed uBlock
  4. Refreshed YT
  5. Confirmed uBlock is blocking the ads
  6. Turned off uBlock for YT
  7. Refreshed YT
  8. Scriplets still showing in the debugger, ads being blocked as a result
  9. Disable uBlock in the Firefox addons manager
  10. Refreshed YT
  11. Ads no longer being blocked and scriplets no longer show in the debugger

So for me this is basically what I see: uBlock enabled and opening YT, can confirm the scriplets are present

1

Turn off uBlock for YT

2

Refresh YT (notice the blue icon in the refresh button in sources search, those results are in cache despite refreshing YT)

3

Search for the scriplets again (no longer cached search results) and they are still present

4

gorhill commented 9 months ago

Thanks. So if I can't reproduce the first time, which steps do I repeat to try to reproduce it again? 1 to 10? Or a subset of these steps?

ParticleCore commented 9 months ago

Thanks. So if I can't reproduce the first time, which steps do I repeat to try to reproduce it again? 1 to 10? Or a subset of these steps?

Usually just enabling uBlock, refreshing YT, disabling uBlock, refreshing YT was sufficient for most cases.

gorhill commented 9 months ago

Are you refreshing your search after you reload the page? If I reload the page after I disable uBO without refreshing the search term (I delete the last s than add s again), they stay there, the browser doesn't re-search on its own. If I force a re-search, the result is empty.

ParticleCore commented 9 months ago

Are you refreshing your search after you reload the page? If I reload the page after I disable uBO without refreshing the search term (I delete the last s than add s again), they stay there, the browser doesn't re-search on its own. If I force a re-search, the result is empty.

Yes, that is what I meant by the blue icon in the images, when that icon is showing then those results are cached from a previous load, the search must be done again in order to confirm the scriplets are there or not after refreshing the page.

image

gorhill commented 9 months ago

I tried tens of times and still can't reproduce. Maybe it's a timing issue and my PC is not as fast/slow as yours? What's the rough specs of your PC?

If scriptlets are still injected after disabling, I would think the issue is in the content-script registering code. This is where content scripts are unregistered when uBO is disabled for a site:

https://github.com/gorhill/uBlock/blob/1.55.0/src/js/scriptlet-filtering.js#L103-L109

ParticleCore commented 9 months ago

I tried tens of times and still can't reproduce. Maybe it's a timing issue and my PC is not as fast/slow as yours? What's the rough specs of your PC?

8gb ram i7 11800h Nvidia RTX 3050

If scriptlets are still injected after disabling, I would think the issue is in the content-script registering code. This is where content scripts are unregistered when uBO is disabled for a site:

https://github.com/gorhill/uBlock/blob/1.55.0/src/js/scriptlet-filtering.js#L103-L109

Any chance a specific condition might not be ironclad when it comes to injecting while checking if it is turned off for a specific website? Timing issue or a disallowed list that might not have time to populate beforehand, etc.

garry-ut99 commented 9 months ago

In no mean I want to interrupt yours conversation, and I don't know whether the both issues are related or not, but I think it's better to inform about an old similiar issue, even if it will turn out to be unrelated to the current one, than to not inform about it, and waste a theoretical possibility to help in the current issue :

ParticleCore commented 9 months ago

In no mean I want to interrupt yours conversation, and I don't know whether the both issues are related or not, but I think it's better to inform about an old similiar issue, even if it will turn out to be unrelated to the current one, than to not inform about it, and waste a theoretical possibility to help in the current issue :

* [Scriptlet injected on whitelisted page #2896](https://github.com/uBlockOrigin/uBlock-issues/issues/2896)

Sounds very reasonable. It does feel like the setting is being ignored for some odd reason, hopefully that will help in figuring something out, thanks.

gorhill commented 9 months ago

If more people can try to reproduce as per repro steps in https://github.com/uBlockOrigin/uBlock-issues/issues/3083#issuecomment-1898716293, this will help maybe identify what else is needed to reproduce.

gorhill commented 9 months ago

2896 is unrelated, it was caused by not having a listener for changes to reset registered content scripts.

It's there now and I verified that it is called immediately upon disabling uBO, @ParticleCore you can verify this by putting a breakpoint at this line and verfying that it executes through this.flush(msg.hostname).

gwarser commented 9 months ago

@ParticleCore can you record video?

garry-ut99 commented 9 months ago

I can reproduce, but only when logged in Youtube.

gwarser commented 9 months ago

Try setting breakpoint as gorhill said and capture few screenshots with stack and local variables.

gorhill commented 9 months ago

I could reproduce while being logged in:

image

The scriptlets are injected into accounts.youtube.com, while www.youtube.com is being trusted.

I think the issue is that when trusting www.youtube.com, only the scriptlets registered for www.youtube.com are being unregistered, i.e. the content scripts registered for accounts.youtube.com are not un-registered.

However the repro steps state that this was reproducible in a new profile, which means without being logged in into Youtube.

Anyway, unregistering all content scripts is not going to solve the issue, and only no longer relying on registered content scripts can work, because the same embedded frame in one context for which uBO is disabled can be used in another context for which uBO is not disabled -- contentScripts.register() knows nothing about top context.

stephenhawk8054 commented 9 months ago

Does this affect other sites with iframe embedded?

gorhill commented 9 months ago

Does this affect other sites with iframe embedded?

Yes, if you trust a.com which embeds b.com, whatever scriptlets were injected in b.com will still be injected because they are registered and uBO was not told to flush them, it was told to unregistered for a.com.

Note that this sort of issue is also a problem with MV3, it's an issue without a solution: https://github.com/uBlockOrigin/uBOL-home/issues/20.

gorhill commented 9 months ago

Also, not sure it's the issue reported because the screenshots in this comment shows that the context is www.youtube.com, for which the registered content scripts should definitely have been evicted.

ParticleCore commented 9 months ago

Also, not sure it's the issue reported because the screenshots in this comment shows that the context is www.youtube.com, for which the registered content scripts should definitely have been evicted.

I think it is, that was my fault, I thought I was using an empty new profile every single time but it was pulling a basic one to save me time from logging into accounts, but other than that it would be a fresh profile for everything else as far as I could see.. My nightly is connected to my profile via account sync, and even though it doesn't look like I am logged into the website:

image

I definitely have a session active in the background:

image

And if I go to the login page it showed that I was already logged in, and logging out cleared the login session:

image

But I will pin this one on Nightly, YT, or both acting up though

Now after logging out on Nightly I cannot replicate the issue again on Nightly, at least not so far. Sounds like a promising find for sure. Same on stable Firefox (you can see I was logged in on stable too because of the subscribed button on gif in https://github.com/uBlockOrigin/uBlock-issues/issues/3083#issuecomment-1898614980), I logged out of my account too and the issue has yet to happen again.

Strangely enough, even if I login again on YT and try to cause the issue to happen I am no longer able to, but I was also only able to do a few test runs. So far all videos play just fine with no lag, no scriplets show up after turning uBlock off anymore either.

Edit: now I know why it was throwing me off; sync was enabled but somehow the profile icon was not showing: image

This is what I am used to seeing when sync is on

image

ParticleCore commented 9 months ago

@ParticleCore can you record video?

Sorry for the late call, if I can make it happen again I will post a recording

krystian3w commented 9 months ago

Maybe duplicate of #3046.

Will be nice repair both with same commits.

garry-ut99 commented 9 months ago

https://github.com/uBlockOrigin/uBlock-issues/issues/3083#issuecomment-1899349892 : gorhill : However the repro steps state that this was reproducible in a new profile, which means without being logged in into Youtube.

I tried as logged in, because in the OP's initial comment https://github.com/uBlockOrigin/uBlock-issues/issues/3083#issue-2088282126 on the first screenshoot there is a rounded green icon with white letter "P" at top-right corner, which means he is logged in, (also by the way, it's worth to mention that the icon is partially obstructed by a text baloon) :

logged-in

However, on screenshoots in his next comment https://github.com/uBlockOrigin/uBlock-issues/issues/3083#issuecomment-1898716293 he isn't logged in anymore, because instead of the green icon, there is "Sign in" button :

logged-out

(I've only said what is on the screenshoots, not that it always reflects reality, given what explained in : https://github.com/uBlockOrigin/uBlock-issues/issues/3083#issuecomment-1899425549)

ParticleCore commented 9 months ago

An update; I've had the issue happening again (normal FF profile on stable FF version while logged in to YT and uBlock turned off for YT) and as soon as I added accounts.google.com to uBlock's trusted site (given this finding https://github.com/uBlockOrigin/uBlock-issues/issues/3083#issuecomment-1899349892) the issue immediately went away after refreshing YT.

It has not happened once while logged out as far as I can recall.

ParticleCore commented 9 months ago

Another update, seems like it wasn't enough adding the accounts.google.com to the trusted site list, it is happening again. There is not much to the recording other than to turn off uBlock for YT, add accounts.google.com as a trusted site and refreshing YT as many times as possible while logged in.

Recording showing uBlock filtering rules still being applied on YT despite being turned off and accounts.google.com included as a trusted site. The first refresh is done using the browser refresh button, the second is using Ctrl+Shift+R

https://github.com/uBlockOrigin/uBlock-issues/assets/9222661/7892882c-6b37-4fb6-8a59-1dfc65149281

And this is what happens after I disable uBlock in the addons manager:

https://github.com/uBlockOrigin/uBlock-issues/assets/9222661/9fc87f25-63f8-403c-8d1b-d145de92d9c3

@gwarser Hope the recordings help

garry-ut99 commented 9 months ago

@ParticleCore I see on your videos that you're still using uBO 1.55 (main channel), can you try on uBO dev version https://github.com/gorhill/uBlock/releases/tag/1.55.1b11 ? There were some improvements to the issue, see : https://github.com/gorhill/uBlock/commit/13dcd844a7.

ParticleCore commented 9 months ago

@ParticleCore I see on your videos that you're still using uBO 1.55 (main channel), can you try on uBO dev version https://github.com/gorhill/uBlock/releases/tag/1.55.1b11 ? There were some improvements to the issue, see : gorhill/uBlock@13dcd844a7.

Using that version and the issue still happens.

Turning uBlock off in the addons manager, refreshing YT, and then turning it back on seems to solve the issue temporarily, but eventually it happens again.

Nevermind, for some reason Firefox did not replace 1.55.0 even after restarting, so I uninstalled stable and installed the beta, will keep an eye on it now.

ParticleCore commented 9 months ago

I can confirm now the issue is still happening with 1.55.1b11 too.

gorhill commented 9 months ago

When I follow the steps at this comment, signed in (which is only how I could reproduce them), I cannot reproduce with 1.55.1b11 -- so back to being unable to reproduce. There has already been a lot of wrong information regarding the steps to reproduce, until it was identified that being signed in was important. So for now I consider the issue correctly mitigated, except the know edge cases I know this could happen, which the repro steps are not.

ParticleCore commented 9 months ago

@gorhill Is there any way to check which hosts might be causing the rules to be applied in the current tab? For example, you found out accounts.gooogle.com was one such case, is there a way I can see for myself if there could be another one causing this when it happens again?

Edit: Adding www.gstatic.com to trusted site seems to have helped too.

gorhill commented 9 months ago

Firefox shows the host on my side:

image

ParticleCore commented 9 months ago

Firefox shows the host on my side:

image

Thank you, I will pay attention to that area if it happens again. So far after adding www.gstatic.com the issue did not return, but the same thing happened with accounts.gooogle.com and eventually came back.

Yuki2718 commented 9 months ago

@ParticleCore What exact "issue" you're referring? Injecting scriptlet rules into accounts.youtube.com should not cause ads being blocked or detection. So if I understand correctly, the original claim has not been confirmed or reproduced by anyone other than you.

ParticleCore commented 9 months ago

@ParticleCore What exact "issue" you're referring? Injecting scriptlet rules into accounts.youtube.com should not cause ads being blocked or detection.

Because of that situation it affects ads on the page which it was suppose to be disabled on, and as a result it causes adblock detection as well. It means that despite appearing to be turned off for YouTube, it is infact still enabled.