Open joanjerezt opened 5 years ago
Which version of Adblock Plus is that?
Also could you please attach a trace-level log from geckodriver? Read more about reporting actionable bugs in our contribution guidelines.
Which version of Adblock Plus is that?
Adblock Plus 3.4.3
Also could you please attach a trace-level log from geckodriver? Read more about reporting actionable bugs in our contribution guidelines.
What I forgot the last time, can you please make sure that the preference extensions.logging.enabled
is set to true?
What I forgot the last time, can you please make sure that the preference
extensions.logging.enabled
is set to true?
Now, I set this preference to true in my custom profile to see add-ons debug messages on the log. It's attached.
It seems that when using a Marionette, Firefox loads in safe mode. Maybe this is why this extension is not working.
"path":"{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}.xpi","runInSafeMode":false
Otherwise, when I code:
firefox_capabilities = DesiredCapabilities.FIREFOX
firefox_capabilities['marionette'] = False
Adblock Plus icon appears in the bar, but obviously Firefox doesn't load any webpage coded in the script.
I don't know if this a expected behaviour, and if it's expected, how to workaround this to achieve my desired result, Adblock Plus working with custom configuration.
So I can't actually see something in the logs which would indicate a malfunction of the extension. But have a look at the very first line of the log:
1551960953830 mozrunner::runner INFO Running command: "/home/juan/firefox/firefox-bin" "-marionette" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofile.KZNO8kKxmoFo"
I'm not sure how you tell geckodriver to use the already existent profile, but here Firefox gets started with a temporary one. So are you sure that geckodriver picks up the one you already preconfigured?
I'm not sure how you tell geckodriver to use the already existent profile, but here Firefox gets started with a temporary one. So are you sure that geckodriver picks up the one you already preconfigured?
Yes, it's a copy of the same profile:
juan@debian:~/selenium-python$ shasum /tmp/rust_mozprofile.CyMYXp4cgs8o
da39a3ee5e6b4b0d3255bfef95601890afd80709 /tmp/rust_mozprofile.CyMYXp4cgs8o
juan@debian:~/selenium-python$ shasum /home/juan/.mozilla/firefox/0yikf5wi.selenium
da39a3ee5e6b4b0d3255bfef95601890afd80709 /home/juan/.mozilla/firefox/0yikf5wi.selenium
Please check manually by copying the profile to a different location like under /tmp and starting firefox with the -profile
to point to that profile. Does that work?
Please check manually by copying the profile to a different location like under /tmp and starting firefox with the
-profile
to point to that profile. Does that work?
Yes, it works by loading Firefox from tmp profile copied by selenium, in marionette mode and in normal mode. But not using selenium python (and NodeJS) bindings to launch browser.
juan@debian:~/firefox$ ./firefox-bin --profile /tmp/rust_mozprofile.dnDrTcors3SE --marionette
juan@debian:~/firefox$ ./firefox-bin --profile /tmp/rust_mozprofile.dnDrTcors3SE
New log attached.
geckodriver.log
Interesting. So it may be a preference geckodriver sets additionally to Marionette which might cause this problem. When you check the log you can find lines with Setting recommended pref
. Try to define those in your Python code and set it to the default value, which would create an user pref geckodriver cannot override again.
Or wait. Maybe this is also related to the settings here: https://searchfox.org/mozilla-central/rev/b2d35912da5b2acecb0274eb113777d344ffb75e/testing/geckodriver/src/prefs.rs#86-89
Here a description: https://mike.kaply.com/2012/02/21/understanding-add-on-scopes/
Can you try to set extensions.enabledScopes
to 15
please?
Interesting. So it may be a preference geckodriver sets additionally to Marionette which might cause this problem. When you check the log you can find lines with
Setting recommended pref
. Try to define those in your Python code and set it to the default value, which would create an user pref geckodriver cannot override again.
Yes, I tried it but I see the same behaviour. Adblock Plus is not working.
profile.set_preference("apz.content_response_timeout", 400)
profile.set_preference("browser.download.panel.shown", True)
# profile.set_preference("browser.pagethumbnails.capturing_disabled", 60000)
# profile.set_preference("browser.tabs.disableBackgroundZombification", 60000)
profile.set_preference("browser.tabs.warnOnCloseOtherTabs", True)
profile.set_preference("browser.urlbar.suggest.searches", True)
profile.set_preference("datareporting.policy.dataSubmissionPolicyAccepted", 0)
profile.set_preference("dom.disable_beforeunload", False)
# profile.set_preference("dom.file.createInChild", 60000)
profile.set_preference("extensions.getAddons.cache.enabled", True)
profile.set_preference("extensions.webservice.discoverURL", "https://discovery.addons.mozilla.org/%LOCALE%/firefox/discovery/pane/%VERSION%/%OS%/%COMPATIBILITY_MODE%")
profile.set_preference("network.http.prompt-temp-redirect", False)
profile.set_preference("network.http.speculative-parallel-limit", 6)
profile.set_preference("security.notification_enable_delay", 500)
profile.set_preference("signon.autofillForms", True)
profile.set_preference("toolkit.cosmeticAnimations.enabled", True)
Commented preferences don't actually exist.
Can you try to set
extensions.enabledScopes
to15
please?
Yes, but it don't work. It should be noted that extensions.autoDisableScopes
property is fixed to 10
value and it cannot be changed by Python script (by default is 15
).
I'm having the same issue (actually an eerily similar setup, I'm also copying profiles to /tmp) and I note that if you go into the about:addons
menu and click disable, then click enable, it does seem to work. It seems like toggling that value fixes this problem, although that value was already set to "enable" when I started toggling it.
I can see that ublock starts up fine, but is disabled before the first page starts loading. I don't know what mechanism is being used to disable it...
Can this not be reproduced by developers?
@joanjerezt, for example in python I have to force the value of extensions.autoDisableScopes
doing the following undocumented hack:
profile.DEFAULT_PREFERENCES['frozen']['extensions.autoDisableScopes'] = 0
profile.set_preference('extensions.enabledScopes', 15)
thanks to a hint I saw here: https://stackoverflow.com/questions/38316910/python-selenium-what-are-possible-keys-in-firefox-webdriver-profile-preference , and doing a review of selenium python code.
To solve the same problem of #1501 ("WARN Add-on ... is not correctly signed."), but with another add-on "HAR Export Trigger", I used the configuration above and, in addition, I put the .xpi file here /home/seluser/.mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/harexporttrigger@getfirebug.com.xpi
, and the add-on automatically was loaded when Firefox starts. (You can put multiple add-ons in that path if you want, for more information see: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Distribution_options/Sideloading_add-ons
Hope this help you.
Note that there was also filed bug 1563845.
This seems to be working for me as of 68.0.1
. Can anyone else confirm?
At least in 69.0b5 the problem was present.
Error started back up in mine, very strange. On my profile about:addons
opened to the "recommendations` tab, so I had to modify your workaround.
def retoggleAllTheAddons(driver):
driver.get("about:addons")
driver.find_element_by_id("category-extension").click()
driver.execute_script("""
let hb = document.getElementById("html-view-browser");
let al = hb.contentWindow.window.document.getElementsByTagName("addon-list")[0];
let cards = al.getElementsByTagName("addon-card");
for(let card of cards){
card.addon.disable();
card.addon.enable();
}
""")
That does take care of this problem though.
On my profile about:addons opened to the "recommendations` tab, so I had to modify your workaround.
I just disabled them in about:config. extensions.htmlaboutaddons.discover.enabled
@joanjerezt, for example in python I have to force the value of
extensions.autoDisableScopes
doing the following undocumented hack:profile.DEFAULT_PREFERENCES['frozen']['extensions.autoDisableScopes'] = 0 profile.set_preference('extensions.enabledScopes', 15)
thanks to a hint I saw here: https://stackoverflow.com/questions/38316910/python-selenium-what-are-possible-keys-in-firefox-webdriver-profile-preference , and doing a review of selenium python code.
To solve the same problem of #1501 ("WARN Add-on ... is not correctly signed."), but with another add-on "HAR Export Trigger", I used the configuration above and, in addition, I put the .xpi file here
/home/seluser/.mozilla/extensions/{ec8030f7-c20a-464f-9b0e-13a3a9e97384}/harexporttrigger@getfirebug.com.xpi
, and the add-on automatically was loaded when Firefox starts. (You can put multiple add-ons in that path if you want, for more information see: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/Distribution_options/Sideloading_add-onsHope this help you.
Yes, this code solves this bug. Thanks.
@mitsukuri it's hard to say something as long as you don't attach a trace log.
And the code to generate it is
from selenium.webdriver import Firefox, FirefoxProfile from selenium.webdriver.firefox.options import Options profile = FirefoxProfile( "c:/firefox/profiles/test") profile.DEFAULT_PREFERENCES['frozen']['extensions.autoDisableScopes'] = 0 profile.set_preference('extensions.enabledScopes', 15) opts = Options() opts.log.level = "trace" ff = Firefox( firefox_profile=profile, options=opts) ff.get( "https://google.com") ff.quit()
The extension (UMatrix in my case) is pre-installed to the test profile.
I also have the same issue where it is not working 68.2.0 macos 10.14.6 selenium 3.141.0
Error started back up in mine, very strange. On my profile
about:addons
opened to the "recommendations` tab, so I had to modify your workaround.def retoggleAllTheAddons(driver): driver.get("about:addons") driver.find_element_by_id("category-extension").click() driver.execute_script(""" let hb = document.getElementById("html-view-browser"); let al = hb.contentWindow.window.document.getElementsByTagName("addon-list")[0]; let cards = al.getElementsByTagName("addon-card"); for(let card of cards){ card.addon.disable(); card.addon.enable(); } """)
That does take care of this problem though.
@traverseda thank you for this. saved me the trouble of figuring this out myself.
@mitsukuri sorry for the delay in my answer. I just had a look at your tracelog and noticed that I forgot to tell that you also have to set the preference extensions.logging.enabled
to true
for the Firefox profile to use. Can you please do that and provide another trace log? Thanks.
Is the addon actually get disabled? Which one is it exactly for you? Please have a look into the Addon manager. The log doesn't actually have the wanted information. :/
This issue has been automatically locked since there has not been any recent activity after it was closed. If you have run into an issue you think is related, please open a new issue.
Sorry, that this issue was locked by the lock bot. We should actually keep it open until we figured out the reason.
@mitsukuri could you maybe attach a pre-configured profile (as zip file) which exhibits the problem? It would help me for testing, so that I won't have to create it on my own. Please also include any other necessary changes. Also if possible a small Python test could be good to have. Thanks.
Sorry, that this issue was locked by the lock bot. We should actually keep it open until we figured out the reason.
You shouldn't have introduced that f*****g bot at all.
I think I've nailed it.
What I did:
- Cleaned the test profile in question by deleting all the files in corresponding folder
- Launched it in a new browser from
about:profiles
.- Installed an extenstion into it (uMatrix, but could be just any).
- Closed the browser with that profile.
- Ran the selenium script -> Everything's fine, uMatrix was up and working.
- Closed the script-instantiated brower.
- [Crucial] Launched the test profile again from
about:profiles
.- Repeated 4 and 5: Gotcha! The uMatrix flashed for a second and disappeared.
At this point it became apparent that step 7 ruins the config in some way.
I cleaned the test profile once again and recreated it, this time saving a backup. Then I repeated steps 2..7 and diffed the profile with its backup. It turned out that some change to
prefs.js
was the culprit.After some fiddling I found that it was a
user_pref("extensions.lastAppBuildId", "20191202093317")
entry that did the trick. If I remove or decrement it, eveything's fine. If it's there, the extension will misbehave and disappear.Now, I've tried to keep that in place but increment other instances of
20191202093317
across theprefs.js
, wondering if that will do, but to no avail.I also converted
2019-12-02 09-33-17
to unix epochs (1575279197
GMT /1575268397
local time) but those are nowhere to be found inprefs.js
I'm not sure where to go from here but at least I'm 100% positive that mere deletion or decrementing of
extensions.lastAppBuildId
to a lesser value does resolve the problem for me.fresh prefs.js prefs.js modified in step 7
Happy new year everyone!
Can confirm this partially I think. I also decremented this number by 1 and it worked fine.
Wow, that's some great progress. Much more reproducible now.
The workaround works like a charm.
Trying to further contribute to this issue. I experience the same behavior on Ubuntu 18.04, with selenium, gecko, marionette and Firefox. I've set the profile path to the relevant .mozilla/firefox/[profile_code]
.
Upon browser startup, the add-ons appear briefly on the task bar, than they disappear (and do not seem to work). I can go and disable then re-enable them from the config window, which causes them to re-appear on the taskbar.
A previous version of Firefox was working fine (probably 1 year ago).
I've tried changing extensions.enabledScopes
from 5 to 15, but the issue wasn't solved (extensions appearing in the first instance means that they are already within scope).
I've enable extensions logging. Here are some relevant log entries for the ghostery
addon (as example):
1585572667108 addons.xpi DEBUG Loading bootstrap scope from jar:file:///home/jeanmonetuser/.mozilla/firefox/w0lgybv7.default/extensions/firefox@ghostery.com.xpi!/
1585572667108 addons.xpi DEBUG Calling bootstrap method startup on firefox@ghostery.com version 8.4.6
1585572667120 addons.manager DEBUG Starting provider: PluginProvider
1585572667120 addons.manager DEBUG Registering shutdown blocker for PluginProvider
1585572667121 addons.manager DEBUG Provider finished startup: PluginProvider
1585572667124 addons.manager DEBUG Completed startup sequence
1585572668113 addons.xpi DEBUG Changed add-on firefox@ghostery.com in app-profile
1585572669280 addons.xpi-utils DEBUG Add-on firefox@ghostery.com modified in app-profile
1585572670511 addons.xpi-utils DEBUG Make addon app-profile:firefox@ghostery.com visible
1585572670511 addons.xpi DEBUG Updating XPIState for {"id":"firefox@ghostery.com","syncGUID":"{23916279-ac32-48de-ab69-7800e08369cb}","version":"8.4.6","type":"extension","loader":null,"updateURL":null,"optionsURL":"app/templates/hub.html","optionsType":3,"optionsBrowserStyle":true,"aboutURL":null,"defaultLocale":{"name":"Ghostery †Privacy Ad Blocker","description":"Ghostery is a powerful privacy extension. Block ads, stop trackers and speed up websites.","creator":"Ghostery","developers":null,"translators":null,"contributors":null},"visible":true,"active":false,"userDisabled":false,"appDisabled":false,"embedderDisabled":false,"installDate":1540865709000,"updateDate":1585572666000,"applyBackgroundUpdates":1,"path":"/tmp/rust_mozprofile.BzZMF0f1AiOe/extensions/firefox@ghostery.com.xpi","skinnable":false,"sourceURI":null,"releaseNotesURI":null,"softDisabled":false,"foreignInstall":false,"strictCompatibility":true,"locales":[{"name":"Ghostery ........","creator":"Ghostery","developers":null,"translators":null,"contributors":null,"locales":["nl"]}],"targetApplications":[{"id":"toolkit@mozilla.org","minVersion":"52.0","maxVersion":null}],"targetPlatforms":[],"signedState":2,"seen":true,"dependencies":[],"incognito":"spanning","userPermissions":{"permissions":["webNavigation","webRequest","webRequestBlocking","cookies","tabs","storage"],"origins":["http://*/*","https://*/*","https://www.ghostery.com/*","https://apps.ghostery.com/*","https://staging-apps.ghostery.com/*","https://gcache.ghostery.com/*","https://staging-gcache.ghostery.com/*","https://account.ghostery.com/*","https://account.ghosterystage.com/*"]},"icons":{"16":"app/images/icon16.png","48":"app/images/icon48.png","128":"app/images/icon128.png"},"iconURL":null,"blocklistState":0,"blocklistURL":null,"startupData":null,"hidden":false,"installTelemetryInfo":null,"recommendationState":{"validNotAfter":1736713913000,"validNotBefore":1578925913000,"states":["recommended"]},"rootURI":"jar:file:///tmp/rust_mozprofile.BzZMF0f1AiOe/extensions/firefox@ghostery.com.xpi!/","location":"app-profile"}
# in above line note: "active":false
# LINE A
1585572670518 addons.xpi DEBUG Calling bootstrap method shutdown on firefox@ghostery.com version 8.4.6
# LINES B & C
1585572670642 addons.xpi DEBUG Loading bootstrap scope from jar:file:///tmp/rust_mozprofile.BzZMF0f1AiOe/extensions/firefox@ghostery.com.xpi!/
1585572670642 addons.xpi DEBUG Calling bootstrap method update on firefox@ghostery.com version 8.4.6
1585572670694 addons.xpi DEBUG Updating for XPIState {"id":"firefox@ghostery.com","syncGUID":"{23916279-ac32-48de-ab69-7800e08369cb}","version":"8.4.6","type":"extension","loader":null,"updateURL":null,"optionsURL":"app/templates/hub.html","optionsType":3,"optionsBrowserStyle":true,"aboutURL":null,"defaultLocale":{"name":"Ghostery.. Privacy Ad Blocker","description":"Ghostery is a powerful privacy extension. Block ads, stop trackers and speed up websites.","creator":"Ghostery","developers":null,"translators":null,"contributors":null},"visible":true,"active":true,"userDisabled":false,"appDisabled":false,"embedderDisabled":false,"installDate":1540865709000,"updateDate":1585572666000,"applyBackgroundUpdates":1,"path":"/tmp/rust_mozprofile.BzZMF0f1AiOe/extensions/firefox@ghostery.com.xpi","skinnable":false,"sourceURI":null,"releaseNotesURI":null,"softDisabled":false,"foreignInstall":false,"strictCompatibility":true,"locales":[{"name":"Ghostery .............","creator":"Ghostery","developers":null,"translators":null,"contributors":null,"locales":["nl"]}],"targetApplications":[{"id":"toolkit@mozilla.org","minVersion":"52.0","maxVersion":null}],"targetPlatforms":[],"signedState":2,"seen":true,"dependencies":[],"incognito":"spanning","userPermissions":{"permissions":["webNavigation","webRequest","webRequestBlocking","cookies","tabs","storage"],"origins":["http://*/*","https://*/*","https://www.ghostery.com/*","https://apps.ghostery.com/*","https://staging-apps.ghostery.com/*","https://gcache.ghostery.com/*","https://staging-gcache.ghostery.com/*","https://account.ghostery.com/*","https://account.ghosterystage.com/*"]},"icons":{"16":"app/images/icon16.png","48":"app/images/icon48.png","128":"app/images/icon128.png"},"iconURL":null,"blocklistState":0,"blocklistURL":null,"startupData":null,"hidden":false,"installTelemetryInfo":null,"recommendationState":{"validNotAfter":1736713913000,"validNotBefore":1578925913000,"states":["recommended"]},"rootURI":"jar:file:///tmp/rust_mozprofile.BzZMF0f1AiOe/extensions/firefox@ghostery.com.xpi!/","location":"app-profile"}
# in above line note only change from previous similar call: "active":true
1585572670698 addons.xpi-utils DEBUG Updating add-on states
Applying the suggested monkey-patch
If I set firefox_profile.set_preferences('extensions.lastAppBuildId', '[number from prefs.js - 1]')
, as suggested above (FIX), Selenium/Firefox seems to load and keep extensions active. If I then remove this setting and re-launch, the extensions remain active.
HOWEVER, if I manually launch Firefox (ie not via Selenium, but via OS), which uses the same profile, close Firefox, then re-launch Selenium/Firefox, extensions appear and disappear again.
If I re-implement FIX, extensions remain visible again.
Note, when I apply FIX, lines A, B and C (see log extract above) seem to disappear from the log.
Hope this helps for finding a permanent fix.
Is there any update on this issue. Workaround is working fine so far
var options = new FirefoxOptions();
options.SetPreference("extensions.lastAppBuildId", "<apppID> -1 ");
I'm sorry if offtopic but maybe it will help somebody. I migrated Firefox profile from Windows 7 Firefox 32bit to Windows 10 Firefox 64bit and failed to get addons working. I removed extensions.lastAppBuildId
pref, restarted FF and now addons work.
Apologies for the necro, but I'm facing the same issue on firefox 102. Tried the workaround however nothing changes. Also tried to use the install_addon function but that is now deprecated from selenium.. Is there a fix that I'm not aware of?
Using selenium with Python btw
System
Testcase
I developed a Javascript file to use an already created Firefox profile with one installed extension (Adblock Plus). The problem is that this extension is enabled but it’s not working.
As a workaround, one can install this extension after loading Firefox browser for first time but it doesn’t work for me because I want an already custom configuration for this extension (own filtering rules).
Unlike this bug report #1405, I’m using a NodeJS API and it doesn’t work after restarting the browser or script. I tried to develop a Python script, importing a custom profile, but the same thing happens.
Stacktrace
None.
Trace-level log