Issue Using Firefox Profiles With Selenium-Wire Driver

[jackbprescott]

Hi! I'm trying to use a selenium.webdriver.firefox.firefox_profile.FirefoxProfile object in order to set all the preferences I want for my Firefox plugins. I'm instantiating the selenium-wire driver using this profile object with the following line:

browser = webdriver.Firefox(seleniumwire_options={'verify_ssl': False}, firefox_profile=profile)

However, when I try to run browser.get('https://www.google.com/'), I get the following issue in the browser that opens:

**Did Not Connect: Potential Security Issue

Firefox detected a potential security threat and did not continue to www.google.com because this website requires a secure connection.

What can you do about it?

www.google.com has a security policy called HTTP Strict Transport Security (HSTS), which means that Firefox can only connect to it securely. You can’t add an exception to visit this site.

The issue is most likely with the website, and there is nothing you can do to resolve it.

If you are on a corporate network or using anti-virus software, you can reach out to the support teams for assistance. You can also notify the website’s administrator about the problem.**

In the Python console, I get the following issue:

Traceback (most recent call last): File "test.py", line 24, in <module> browser.get('https://www.google.com/') File "/Users/jackprescott/Documents/Python/Virtualenvs/QBS/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 333, in get self.execute(Command.GET, {'url': url}) File "/Users/jackprescott/Documents/Python/Virtualenvs/QBS/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute self.error_handler.check_response(response) File "/Users/jackprescott/Documents/Python/Virtualenvs/QBS/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.InsecureCertificateException: Message:

When I run the exact same code, but remove the profile argument from the webdriver instantiation, it works just fine. Do you have any suggestions on how to fix this/what might be going on? Alternatively, do you know any other ways we can set the plugin preferences in Firefox with Selenium without passing a Firefox profile to the webdriver?

[wkeeling]

Thanks for raising this.

Do you have the same problem with the Firefox profile if you use Selenium directly - i.e. if you use from selenium import webdriver ?

[jackbprescott]

The exact same code, except using a selenium webdriver instead of a selenium-wire webdriver, works as expected, using the profile properly and opening the browser with the correct preferences for my plug-ins, with no problems going to any websites I tried.

[wkeeling]

OK thanks. I'll see if I can reproduce.

[silviuFlorin]

I was able to reproduce it when having a proxy configured manually from about:preferences

[wkeeling]

@jackbprescott Sorry for the delay in coming back. Would you be able to to post the contents of the profile object you're passing? I appreciate that may contain plugin configuration specific to your environment, but I'm curious to see whether there may be anything in there that's causing the issue with Selenium Wire.

[woswos]

I have this issue as well. I am trying to control the Tor Browser, which is essentially Firefox. I need to specify socks ports to make it connect to Tor. I use the following lines to set the proxy. It works without any problems with the regular selenium but it doesn't work with selenium-wire. The code runs without any errors but the Tor Browser doesn't connect to Tor, which means the proxy wasn't set properly.

profile = FirefoxProfile('path/to/tor/browser/profile')
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.update_preferences()

driver = webdriver.Firefox(firefox_profile=profile)

Should I use seleniumwire_options to specify these instead?

Edit: I should have used seleniumwire_options :) Edit 2: I realized that using seleniumwire_options makes regular Firefox proxied with Tor. However, I want to run the Tor Browser itself and I need profiles for that, which doesn't work for some reason as mentioned in this issue Edit 3: I solved this issue and an example can be found here: https://gist.github.com/woswos/38b921f0b82de009c12c6494db3f50c5

[Hitkul]

I have been facing the same issue with firefox. Selenium works fine but Selenium-wire give the exception mentioned in the issue. any solution?

[wkeeling]

@Hitkul I'm still investigating, hopefully have an answer soon. Are you seeing the problem on the latest version 1.2.1 of Selenium Wire?

[Hitkul]

@wkeeling Yeh I am on the latest version. Looking forward to the solution. Thanks

[wkeeling]

Sorry it's taken a while to get to this. I've tried reproducing with the latest version of Selenium Wire, but I don't see the "Did Not Connect: Potential Security Issue" message. Google seems to load normally.

I'm using a simple profile:

profile = FirefoxProfile()
profile.set_preference('intl.accept_languages', 'no,en-us,en')
driver = webdriver.Firefox(firefox_profile=profile)

driver.get('https://www.google.com/')

Can you confirm whether this is still an issue on the latest version of Selenium Wire?

[JayPalm]

Having the same issue, getting the security issue message, even with allowing untrusted certs.

from seleniumwire import webdriver
from selenium.webdriver.common.keys import Keys

profile_path = '<path/to/profile>'
profile = webdriver.FirefoxProfile(profile_path)
profile.accept_untrusted_certs=True

driver = webdriver.Firefox(profile)
driver.get("https://oakland.overdrive.com/account/loans")

No issue if I import webdriver from selenium (instead of seleniumwire). Also don't have an issue when I remove the profile path, but then I don't get what I need.

[villasv]

I had this issue while using an existing profile too. It went away after I added the Selenium Wire certificate as an authority to that browser profile.

[converge]

hi, was seleniumwire webdriver.FirefoxProfile removed? how could I set accepted language pls?

[wkeeling]

@converge you'll need to import directly from the selenium package:

from selenium.webdriver.firefox.firefox_profile import FirefoxProfile

I'll look at making FirefoxProfile available in the selenium-wire specific webdriver package in a new release.

[jefcolbi]

Hi. I faced the same error with the latest selenium-wire

default_prof_path = list(FirefoxProfileLoader.get_profiles())[0].path
print(f"{default_prof_path = }")

options = FirefoxOptions()
options.accept_insecure_certs = True
options.set_preference("security.cert_pinning.enforcement_level", 0)
options.set_preference("network.stricttransportsecurity.preloadlist", False)

options.profile = FirefoxProfile(default_prof_path)
options.profile.set_preference("security.cert_pinning.enforcement_level", 0)
options.profile.set_preference("network.stricttransportsecurity.preloadlist", False)
options.profile.update_preferences()

return Firefox(options=options)

Traceback

/home/jefcolbi/Projets/Open/abf/connector/src/libraries/base_scraper.py:264: DeprecationWarning: firefox_profile has been deprecated, please use an Options object
  options.profile = FirefoxProfile(default_prof_path)
/home/jefcolbi/Projets/Open/abf/connector/src/libraries/base_scraper.py:264: DeprecationWarning: Setting a profile has been deprecated. Please use the set_preference and install_addons methods
  options.profile = FirefoxProfile(default_prof_path)
/home/jefcolbi/Projets/Open/abf/connector/src/libraries/base_scraper.py:265: DeprecationWarning: Getting a profile has been deprecated.
  options.profile.set_preference("security.cert_pinning.enforcement_level", 0)
/home/jefcolbi/Projets/Open/abf/connector/src/libraries/base_scraper.py:266: DeprecationWarning: Getting a profile has been deprecated.
  options.profile.set_preference("network.stricttransportsecurity.preloadlist", False)
/home/jefcolbi/Projets/Open/abf/connector/src/libraries/base_scraper.py:267: DeprecationWarning: Getting a profile has been deprecated.
  options.profile.update_preferences()
/home/jefcolbi/Projets/Open/abf/connector/.env/lib/python3.8/site-packages/selenium/webdriver/firefox/webdriver.py:148: DeprecationWarning: Getting a profile has been deprecated.
  if options.profile:
/home/jefcolbi/Projets/Open/abf/connector/.env/lib/python3.8/site-packages/selenium/webdriver/firefox/webdriver.py:149: DeprecationWarning: Getting a profile has been deprecated.
  self.profile = options.profile
Traceback with variables (most recent call last):
  File "<string>", line 244, in run
      self = <base_test.Runner object at 0x7ffbbd705df0>
      query = 'list_all_customers'
      e = InsecureCertificateException('', None, ['RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8', 'WebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:182:5', 'InsecureCertificateError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:303:5', 'checkReadyState@chrome://remote/content/marionette/navigate.sys.mjs:58:24', 'onNavigation@chrome://remote/content/marionette/navigate.sys.mjs:329:39', 'emit@resource://gre/modules/EventEmitter.jsm:160:20', 'receiveMessage@chrome://remote/content/marionette/actors/MarionetteEventsParent.sys.mjs:37:25', ''])
  File "<string>", line 63, in login
      self = <base_test.FortnoxScraper object at 0x7ffbbd705e50>
  File "/home/jefcolbi/Projets/Open/abf/connector/.env/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 441, in get
    self.execute(Command.GET, {'url': url})
      self = <seleniumwire.webdriver.Firefox (session="daea9dc7-91bb-4679-9dfc-b2ba2deed14a")>
      url = 'https://apps.domain.com/auth/auth/login.php'
  File "/home/jefcolbi/Projets/Open/abf/connector/.env/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
    self.error_handler.check_response(response)
      self = <seleniumwire.webdriver.Firefox (session="daea9dc7-91bb-4679-9dfc-b2ba2deed14a")>
      driver_command = 'get'
      params = {'url': 'https://apps.domain.com/auth/auth/login.php'}
      response = {'status': 400, 'value': '{"value":{"error":"insecure certificate","message":"","stacktrace":"RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8\\nWebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:182:5\\nInsecureCertificateError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:303:5\\ncheckReadyState@chrome://remote/content/marionette/navigate.sys.mjs:58:24\\nonNavigation@chrome://remote/content/marionette/navigate.sys.mjs:329:39\\nemit@resource://gre/modules/EventEmitter.jsm:160:20\\nreceiveMessage@chrome://remote/content/marionette/actors/MarionetteEventsParent.sys.mjs:37:25\\n"}}'}
  File "/home/jefcolbi/Projets/Open/abf/connector/.env/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
    raise exception_class(message, screen, stacktrace)
      self = <selenium.webdriver.remote.errorhandler.ErrorHandler object at 0x7ffbb76959d0>
      response = {'status': 400, 'value': '{"value":{"error":"insecure certificate","message":"","stacktrace":"RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8\\nWebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:182:5\\nInsecureCertificateError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:303:5\\ncheckReadyState@chrome://remote/content/marionette/navigate.sys.mjs:58:24\\nonNavigation@chrome://remote/content/marionette/navigate.sys.mjs:329:39\\nemit@resource://gre/modules/EventEmitter.jsm:160:20\\nreceiveMessage@chrome://remote/content/marionette/actors/MarionetteEventsParent.sys.mjs:37:25\\n"}}'}
      status = 'insecure certificate'
      value = {'error': 'insecure certificate', 'message': '', 'stacktrace': 'RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8\nWebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:182:5\nInsecureCertificateError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:303:5\ncheckReadyState@chrome://remote/content/marionette/navigate.sys.mjs:58:24\nonNavigation@chrome://remote/content/marionette/navigate.sys.mjs:329:39\nemit@resource://gre/modules/EventEmitter.jsm:160:20\nreceiveMessage@chrome://remote/content/marionette/actors/MarionetteEventsParent.sys.mjs:37:25\n'}
      message = ''
      screen = None
      stacktrace = ['RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8', 'WebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:182:5', 'InsecureCertificateError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:303:5', 'checkReadyState@chrome://remote/content/marionette/navigate.sys.mjs:58:24', 'onNavigation@chrome://remote/content/marionette/navigate.sys.mjs:329:39', 'emit@resource://gre/modules/EventEmitter.jsm:160:20', 'receiveMessage@chrome://remote/content/marionette/actors/MarionetteEventsParent.sys.mjs:37:25', '']
      value_json = '{"value":{"error":"insecure certificate","message":"","stacktrace":"RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8\\nWebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:182:5\\nInsecureCertificateError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:303:5\\ncheckReadyState@chrome://remote/content/marionette/navigate.sys.mjs:58:24\\nonNavigation@chrome://remote/content/marionette/navigate.sys.mjs:329:39\\nemit@resource://gre/modules/EventEmitter.jsm:160:20\\nreceiveMessage@chrome://remote/content/marionette/actors/MarionetteEventsParent.sys.mjs:37:25\\n"}}'
      json = <module 'json' from '/usr/lib/python3.8/json/__init__.py'>
      exception_class = <class 'selenium.common.exceptions.InsecureCertificateException'>
      st_value = 'RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8\nWebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:182:5\nInsecureCertificateError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:303:5\ncheckReadyState@chrome://remote/content/marionette/navigate.sys.mjs:58:24\nonNavigation@chrome://remote/content/marionette/navigate.sys.mjs:329:39\nemit@resource://gre/modules/EventEmitter.jsm:160:20\nreceiveMessage@chrome://remote/content/marionette/actors/MarionetteEventsParent.sys.mjs:37:25\n'
selenium.common.exceptions.InsecureCertificateException: Message: 
Stacktrace:
RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8
WebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:182:5
InsecureCertificateError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:303:5
checkReadyState@chrome://remote/content/marionette/navigate.sys.mjs:58:24
onNavigation@chrome://remote/content/marionette/navigate.sys.mjs:329:39
emit@resource://gre/modules/EventEmitter.jsm:160:20
receiveMessage@chrome://remote/content/marionette/actors/MarionetteEventsParent.sys.mjs:37:25

No error until now?