[BUG/QUESTION] header_overrides works differently for headless and headful chrome

[jaroszan]

I have a switch in my script which looks as follows:

gui_mode = False
if not gui_mode:
     chrome_options.add_argument("--log-level=3")
     chrome_options.add_argument("--headless")
     chrome_options.add_argument("--window-size=2560,1440")
...
self.driver = webdriver.Chrome(
     chrome_options=chrome_options,
     seleniumwire_options=seleniumwire_options,
)
...
self.driver.header_overrides = {
     'cookie': 'COOKIE'
}

If I set gui mode to False header_overrides works as expected, however when I set it to True cookie header is not being added. I am using it to bypass certain authentication flow which involves MFA. I am not checking headers per se, but I can see headless mode working by checking screenshots generated during session. Is there a difference between how headless and headful mode affect selenium-wire?

[wkeeling]

There shouldn't be any difference in running the browser in headless mode or not. Have you tried using a request interceptor instead of the header_overrides mechanism? The header_overrides mechanism has been deprecated in favour of request interceptors.

For example:

def interceptor(request):
    del request.headers['cookie']  # Remove any existing cookie
    request.headers['cookie'] = 'COOKIE'

self.driver.request_interceptor = interceptor

You can verify whether it's working by pointing the webdriver at https://httpbin.org/headers which will echo the headers back to you.

[jaroszan]

Thanks for suggestion. In my case it made no difference. My setup involved making screenshots and saving html on failure and when in headful mode I get MFA screen, in headless mode I get 502 Bad Gateway (which looks like https://github.com/wkeeling/selenium-wire/issues/113). What is weird is that headless mode for Chrome worked 100% before and it was good enough for me. Chrome in itself seems to differentiate between how it works in headless and headful modes, eg. https://bugs.chromium.org/p/chromium/issues/detail?id=765245

Firefox on the hand consistently fails both in headless and headful on redirection to authentication mechanism, but different step than when using Chrome.

I have tried other automating solutions (puppeteer and playwright) and injecting header in them works consistently - cookie is recognized successfully by the target application. Which means I am either misusing selenium-wire or there are underlying issues. I will keep trying to make it work in selenium-wire for now.

Another important detail is that I am using mitmproxy as backend. With default proxy I received bunch of random errors which were gone as soon as I switched to mitmproxy. But may be this backend has issues of its own.

[jaroszan]

@wkeeling is it the correct way to use request interceptor?

self.driver = webdriver.Chrome(
    chrome_options=chrome_options,
    seleniumwire_options=seleniumwire_options,
)

def interceptor(request):
            print("REQUEST INTERCEPTED")
            del request.headers['cookie']
            request.headers['cookie'] = 'cookie'
            print(request.headers)

self.driver.request_interceptor = interceptor
self.driver.get("https://httpbin.org/headers")

I see nothing being printed in the terminal and can't see added header in browser. Or am I doing it in the wrong way?

[wkeeling]

That looks fine. What version of Selenium Wire are you using?

[jaroszan]

selenium 3.141.0 selenium-wire 4.2.4

[wkeeling]

I notice that your code is running within a class (the self reference). Are you able to share all the code - just in case there's something strange happening that we can't see? Obviously obfuscate anything sensitive.

[jaroszan]

I have put it everything inside the setUp function of python unittest:

class BaseTestClass(unittest.TestCase):
    def setUp(self):
        init()
        chrome_options = webdriver.ChromeOptions()
        chrome_options.add_argument('--no-sandbox')
        chrome_options.add_argument("--disable-setuid-sandbox")
        chrome_options.add_argument("--disable-dev-shm-usage")
        seleniumwire_options = {
            'backend': 'mitmproxy',
            'verify_ssl': False
        }
        gui_mode = True
        if not gui_mode:
            chrome_options.add_argument("--log-level=3")
            chrome_options.add_argument("--headless")
            chrome_options.add_argument("--window-size=2560,1440")

        self.driver = webdriver.Chrome(
            chrome_options=chrome_options,
            seleniumwire_options=seleniumwire_options,
        )

        def interceptor(request):
            print("REQUEST INTERCEPTED")
            del request.headers['cookie']
            request.headers['cookie'] = 'COOKIE'
            print(request.headers)

        self.driver.request_interceptor = interceptor
        self.driver.get("https://httpbin.org/headers")
        self.driver.maximize_window()
        time.sleep(30)
        self.verificationErrors = []

[wkeeling]

The code looks good. I ran it myself and it seemed to set the cookie header as expected. I had to add a test_() method to the class in order for the test runner to execute the setUp(), but this is what it gave me:

Perhaps worth creating a clean environment and trying with that, in case there's something unexpected happening in your existing environment perhaps?

[jaroszan]

Simplifying test case to the extreme didn't help in my case, so I moved over to a different Windows 10 machine where I have unlimited freedoms and it worked like a charm. Looks like some of the security policies (on which I have no impact) on my main work machine interfere somehow into how selenium-wire is supposed to work. Thanks and sorry for confusion.