Cannot Add Prefs as Experimental Option

[metalpacoforeal]

Hi there!

I'm trying to download a PDF file using uc version 3.1.3 and I'm unable to add an experimental option to download PDF files externally.

This code works fine with selenium's webdriver, but not with UC.

Here's the code snippet...

` from selenium import webdriver import undetected_chromedriver as uc

if name == 'main':

Set up a headless chromedriver

options = webdriver.ChromeOptions()

# Set pdf download preference
prefs = {"plugins.always_open_pdf_externally": True}
options.add_experimental_option("prefs", prefs)

# Initialize driver
driver = uc.Chrome(options=options, version_main=98)
driver.get('www.barchart.com')

`

And here's the traceback:

Traceback (most recent call last): File "/Users/jrodriguez/Library/Application Support/JetBrains/PyCharm2021.3/scratches/scratch_3.py", line 13, in <module> driver = uc.Chrome(options=options, version_main=98) File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/undetected_chromedriver/__init__.py", line 369, in __init__ super(Chrome, self).__init__( File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 70, in __init__ super(WebDriver, self).__init__(DesiredCapabilities.CHROME['browserName'], "goog", File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 93, in __init__ RemoteWebDriver.__init__( File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 268, in __init__ self.start_session(capabilities, browser_profile) File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/undetected_chromedriver/__init__.py", line 552, in start_session super(selenium.webdriver.chrome.webdriver.WebDriver, self).start_session( File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 359, in start_session response = self.execute(Command.NEW_SESSION, parameters) File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 424, in execute self.error_handler.check_response(response) File "/Users/jrodriguez/PycharmProjects/BespokeReportScraper/venv/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: cannot parse capability: goog:chromeOptions from invalid argument: unrecognized chrome option: prefs Stacktrace: 0 chromedriver 0x0000000100796ee9 chromedriver + 5013225 1 chromedriver 0x00000001007221d3 chromedriver + 4534739 2 chromedriver 0x00000001002f8a68 chromedriver + 170600 3 chromedriver 0x00000001003112b7 chromedriver + 271031 4 chromedriver 0x00000001003157c5 chromedriver + 288709 5 chromedriver 0x000000010030de9b chromedriver + 257691 6 chromedriver 0x000000010034f1c8 chromedriver + 524744 7 chromedriver 0x000000010034ec6d chromedriver + 523373 8 chromedriver 0x000000010035030a chromedriver + 529162 9 chromedriver 0x000000010034a6d3 chromedriver + 505555 10 chromedriver 0x000000010032076e chromedriver + 333678 11 chromedriver 0x0000000100321745 chromedriver + 337733 12 chromedriver 0x0000000100752efe chromedriver + 4734718 13 chromedriver 0x000000010076ca19 chromedriver + 4839961 14 chromedriver 0x00000001007721c8 chromedriver + 4862408 15 chromedriver 0x000000010076d3aa chromedriver + 4842410 16 chromedriver 0x0000000100747a01 chromedriver + 4688385 17 chromedriver 0x0000000100788538 chromedriver + 4953400 18 chromedriver 0x00000001007886c1 chromedriver + 4953793 19 chromedriver 0x000000010079e225 chromedriver + 5042725 20 libsystem_pthread.dylib 0x00007fff2071a8fc _pthread_start + 224 21 libsystem_pthread.dylib 0x00007fff20716443 thread_start + 15

[morjun]

struggling with the same issue here

[sebdelsol]

Chrome is started before its driver and misses all desired capabilities pushed by the driver. Here's a solution that works for me:

import json
import os
import tempfile
from functools import reduce

import undetected_chromedriver as webdriver

class ChromeWithPrefs(webdriver.Chrome):
    def __init__(self, *args, options=None, **kwargs):
        if options:
            self._handle_prefs(options)

        super().__init__(*args, options=options, **kwargs)

        # remove the user_data_dir when quitting
        self.keep_user_data_dir = False

    @staticmethod
    def _handle_prefs(options):
        if prefs := options.experimental_options.get("prefs"):
            # turn a (dotted key, value) into a proper nested dict
            def undot_key(key, value):
                if "." in key:
                    key, rest = key.split(".", 1)
                    value = undot_key(rest, value)
                return {key: value}

            # undot prefs dict keys
            undot_prefs = reduce(
                lambda d1, d2: {**d1, **d2},  # merge dicts
                (undot_key(key, value) for key, value in prefs.items()),
            )

            # create an user_data_dir and add its path to the options
            user_data_dir = os.path.normpath(tempfile.mkdtemp())
            options.add_argument(f"--user-data-dir={user_data_dir}")

            # create the preferences json file in its default directory
            default_dir = os.path.join(user_data_dir, "Default")
            os.mkdir(default_dir)

            prefs_file = os.path.join(default_dir, "Preferences")
            with open(prefs_file, encoding="latin1", mode="w") as f:
                json.dump(undot_prefs, f)

            # pylint: disable=protected-access
            # remove the experimental_options to avoid an error
            del options._experimental_options["prefs"]

if __name__ == "__main__":
    prefs = {
        "profile.default_content_setting_values.images": 2,
        # "download.default_directory": "d:/temp",
        # "plugins.always_open_pdf_externally": True,
    }
    options = webdriver.ChromeOptions()
    options.add_experimental_option("prefs", prefs)

    # use the derived Chrome class that handles prefs
    driver = ChromeWithPrefs(options=options)

The idea is to create a "Default/Preferences" json file from the ChromeOptions().experimental_options["prefs"] key and remove it to avoid the error. A way to do it is to create and use a derived class of Chrome that handles this.

Disclaimer:

• I don't know Chrome and Selenium enough to be sure it's the best way to do it and that there's no caveats.
• I don't know if this would work with the others Chrome specific capabilities. Hence I limited it to the "prefs" key.

[metalpacoforeal]

Thanks, @sebdelsol! I'll give that a shot and report back. 🙂

[metalpacoforeal]

So @sebdelsol, your code works for adding prefs (yay!)... but for some reason, now I can't download the PDF due to "insufficient permissions." Much closer to my script doing what it's supposed to do, though, so thank you! 🙂

[sebdelsol]

Make sure your Downloads folder has proper permissions. Btw you can now change it with prefs = {"download.default_directory": "your path"}

[sebdelsol]

This might be a fix for some issues : #517, #495, #493, #477, #452, #338, #309, #260, #213, #211 and #194

[metalpacoforeal]

It works! You were absolutely right about the permissions on the folder, @sebdelsol. Thanks so much!

[bhargav-sarvaria]

Chrome is started before its driver and misses all desired capabilities pushed by the driver. Here's a solution that works for me:

import json
import os
import tempfile
from functools import reduce

import undetected_chromedriver as webdriver

class ChromeWithPrefs(webdriver.Chrome):
    def __init__(self, *args, options=None, **kwargs):
        if options:
            self._handle_prefs(options)

        super().__init__(*args, options=options, **kwargs)

        # remove the user_data_dir when quitting
        self.keep_user_data_dir = False

    @staticmethod
    def _handle_prefs(options):
        if prefs := options.experimental_options.get("prefs"):
            # turn a (dotted key, value) into a proper nested dict
            def undot_key(key, value):
                if "." in key:
                    key, rest = key.split(".", 1)
                    value = undot_key(rest, value)
                return {key: value}

            # undot prefs dict keys
            undot_prefs = reduce(
                lambda d1, d2: {**d1, **d2},  # merge dicts
                (undot_key(key, value) for key, value in prefs.items()),
            )

            # create an user_data_dir and add its path to the options
            user_data_dir = os.path.normpath(tempfile.mkdtemp())
            options.add_argument(f"--user-data-dir={user_data_dir}")

            # create the preferences json file in its default directory
            default_dir = os.path.join(user_data_dir, "Default")
            os.mkdir(default_dir)

            prefs_file = os.path.join(default_dir, "Preferences")
            with open(prefs_file, encoding="latin1", mode="w") as f:
                json.dump(undot_prefs, f)

            # pylint: disable=protected-access
            # remove the experimental_options to avoid an error
            del options._experimental_options["prefs"]

if __name__ == "__main__":
    prefs = {
        "profile.default_content_setting_values.images": 2,
        # "download.default_directory": "d:/temp",
        # "plugins.always_open_pdf_externally": True,
    }
    options = webdriver.ChromeOptions()
    options.add_experimental_option("prefs", prefs)

    # use the derived Chrome class that handles prefs
    driver = ChromeWithPrefs(options=options)

The idea is to create a "Default/Preferences" json file from the ChromeOptions().experimental_options["prefs"] key and remove it to avoid the error. A way to do it is to create and use a derived class of Chrome that handles this.

Disclaimer:

• I don't know Chrome and Selenium enough to be sure it's the best way to do it and that there's no caveats.
• I don't know if this would work with the others Chrome specific capabilities. Hence I limited it to the "prefs" key.

THANK YOU SO MUCH! I WANT TO GIVE YOU A BIG HUG!

[xerobia-test]

This doesn't work for mobileEmulation though - even after replacing all the pref parts with mobileEmulation. Can anyone help?

[sebdelsol]

As stated it works only with prefs experimental options because they are actually stored in the Profile/Default/Preferences file. I don't know whether this other experimental option you need is stored in Chrome profile and/or it's a mere capability... To find out you should begin with a simple Selenium driver and check how it's handled with a debugger and check thoroughly the Selenium repo.

EDIT : By quickly searching on the Selenium repo, it seems you only need to add the correct capabilities. Please check the Selenium documentation how to do that.

EDIT2 : @xerobia-test, I'm only a contributor. I needed prefs experimental options and suggested a pull request to do that and it was merged. If you have further needs, feel free to do the same.

EDIT3 : Those option are not that easy to implement since you can't rely on your driver forwarding them with a mere request when launching the browser. In undetected-chromdriver case, Chrome is spawned before the driver (that's half the trick how it remains undetectable) hence restricting further requests for experimental options. I haven't found any reliable accessible documentations for how each were actually implemented, so it was for me a trial and error process (at least faster than finding my way in Chromium code).

[xerobia-test]

I see. Are there any plans to get the add_experimental_options option working again in the near future?

[xerobia-test]

That's ok. Thanks for your efforts! I've been messing around with it for hours now - I think I'll just settle for now.

[tmb5cg]

I don't understand how/where you guys learn everything you know but @sebdelsol thank you.

@xerobia-test I am also settling. I am trying to post instagram stories, which only works by emulating mobile via chrome dev tools for some reason. I'm going to search for a chrome extension that maybe can emulate mobile, but if I understood what sebdelsol said, it seems like mobile emulation is an entirely separate operation that is done somewhere special in between the driver/sending requests - who knows

[kaliiiiiiiiii]

This doesn't work for mobileEmulation though - even after replacing all the pref parts with mobileEmulation. Can anyone help?

have a look at Selenium-Profiles

Repo still in progress // working on btw

[sebdelsol]

The prefs experimental options are not correctly handled since those could be nested dict and can't be properly merged with a mere dict.update or the ** operator in a dict comprehension as used in the code above. Please check #869 for a workaround till the author merge the relevant pull requests.

[Bohdanloz]

Hi there, i try to use your code in my project - automation of google meet joining and it still asks me for micro and video camera access. How can I fix this?

from selenium import webdriver
from selenium.webdriver import DesiredCapabilities
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
import undetected_chromedriver as uc
import time
import os
import json
import tempfile
from functools import reduce

class ChromeWithPrefs(uc.Chrome):
    def __init__(self, *args, options=None, **kwargs):
        if options:
            self._handle_prefs(options)

        super().__init__(*args, options=options, **kwargs)

        # remove the user_data_dir when quitting
        self.keep_user_data_dir = False

    @staticmethod
    def _handle_prefs(options):
        if prefs := options.experimental_options.get("prefs"):
            # turn a (dotted key, value) into a proper nested dict
            def undot_key(key, value):
                if "." in key:
                    key, rest = key.split(".", 1)
                    value = undot_key(rest, value)
                return {key: value}

            # undot prefs dict keys
            undot_prefs = reduce(
                lambda d1, d2: {**d1, **d2},  # merge dicts
                (undot_key(key, value) for key, value in prefs.items()),
            )

            # create an user_data_dir and add its path to the options
            user_data_dir = os.path.normpath(tempfile.mkdtemp())
            options.add_argument(f"--user-data-dir={user_data_dir}")

            # create the preferences json file in its default directory
            default_dir = os.path.join(user_data_dir, "Default")
            os.mkdir(default_dir)

            prefs_file = os.path.join(default_dir, "Preferences")
            with open(prefs_file, encoding="latin1", mode="w") as f:
                json.dump(undot_prefs, f)

            # pylint: disable=protected-access
            # remove the experimental_options to avoid an error
            del options._experimental_options["prefs"]

prefs = {
    "profile.default_content_setting_values.media_stream_mic": 2,
    "profile.default_content_setting_values.media_stream_camera": 2,
    "profile.default_content_setting_values.geolocation": 2,
    "profile.default_content_setting_values.notifications": 2
}
#user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36'
opt = uc.ChromeOptions()
opt.add_argument('--no-sandbox')
#opt.add_argument('--headless=chrome')
opt.add_argument("--disable-extensions")
opt.add_argument("--disable-infobars")
opt.add_argument("user_agent=DN")
#opt.add_argument(f'user-agent={user_agent}')
opt.add_argument("--proxy-server='direct://'")
opt.add_argument("--proxy-bypass-list=*")
opt.add_argument('--disable-gpu')
opt.add_argument("--window-size=1920x1080")
opt.add_argument("--allow-insecure-localhost")
opt.add_argument('--ignore-certificate-errors')
opt.add_argument('--allow-running-insecure-content')
opt.add_argument('--disable-dev-shm-usage')
opt.add_experimental_option("prefs", prefs)
opt.add_argument('--start-maximized')

def join_google(link, mail_address, password):
    driver = ChromeWithPrefs(options=opt)

Here is how my code looks like. Can you please say me what I am doing wrong? Cause all the time it asks me for micro and video access and I cannot close this popup message cause it's covered with another one.

[asapsmc]

I'm having a problem with the (above) @sebdelsol workaround. When I don't have to set experimental options (thus I don't have the need to use this workaround) my code works perfectly (opens a specific chrome profile) with this

options = webdriver.ChromeOptions()
options.headless=False
options.add_argument("--log-level=DEBUG")
options.add_argument("--user-data-dir=/Users/machine/Library/Application Support/Google/Chrome")
driver = uc.Chrome(options=options)

However, when I use the above workaround code, when I try to do this

prefs = {
    "profile.default_content_setting_values.images": 2,
     # "download.default_directory": "d:/temp",
     # "plugins.always_open_pdf_externally": True,
}
options2 = webdriver.ChromeOptions()
options2.add_experimental_option("prefs", prefs)
driver2 = ChromeWithPrefs(options=options2)

I get The chromedriver version (114.0.5735.90) detected in PATH at /usr/local/bin/chromedriver might not be compatible with the detected chrome version (120.0.6099.199); currently, chromedriver 120.0.6099.109 is recommended for chrome 120.*, so it is advised to delete the driver in PATH and retry.

So, it seems that with the "regular" webdriver.ChromeOptions(), Selenium (I'm using Selenium 4.16 and undetected_chromedriver 3.5.3) finds the correct ChromeDriver version, otherwise it doesn't.

Can anyone help me identify where's the issue? Why can't this approach find the correct driver automatically?

[HakanBi]

Add options.add_argument('--handle_prefs') to handle it

[asapsmc]

Do you mean options.add_argument('--handle_prefs=prefs')? I have tried options2.add_argument('--handle_prefs') options2.add_experimental_option("prefs", prefs) driver2 = uc.Chrome(options=options2), but I get an error:

Message: invalid argument: cannot parse capability: goog:chromeOptions
from invalid argument: unrecognized chrome option: prefs