[🐛 Bug]: Fail to work on headless mode

[Alperen-ylmaz]

What happened?

Hi I am trying to scrape links from websites however I got the error below. When run my script without headless mode it works but it doesnt work with headless mode.

[2024-08-08T10:53:33.515Z] Error processing request: Message: [2024-08-08T10:53:33.530Z] Stacktrace: [2024-08-08T10:53:33.530Z] 0 chromedriver 0x0000000100cd10b8 cxxbridge1$str$ptr + 1887276 [2024-08-08T10:53:33.530Z] 1 chromedriver 0x0000000100cc9794 cxxbridge1$str$ptr + 1856264 [2024-08-08T10:53:33.530Z] 2 chromedriver 0x00000001008d882c cxxbridge1$string$len + 88524 [2024-08-08T10:53:33.530Z] 3 chromedriver 0x000000010091c834 cxxbridge1$string$len + 367060 ... my driver setup below

def setup_driver(): options = Options() options.add_argument('--no-sandbox') options.add_argument('--disable-dev-shm-usage') options.add_argument('--window-size=1920x1080') options.add_argument('--incognito') options.add_argument('--headless') # Run in headless mode options.add_argument('--disable-gpu') driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)
return driver

Thanks in advance for your help! I really appreciate any advice you can give.

How can we reproduce the issue?

def setup_driver():
    options = Options()
    options.add_argument('--no-sandbox')
    options.add_argument('--disable-dev-shm-usage')
    options.add_argument('--window-size=1920x1080')
    options.add_argument('--incognito')
    options.add_argument('--headless')  # Run in headless mode
    options.add_argument('--disable-gpu')
    driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)    
    return driver

Relevant log output

[2024-08-08T10:53:33.515Z] Error processing request: Message:
[2024-08-08T10:53:33.530Z] Stacktrace:
[2024-08-08T10:53:33.530Z] 0   chromedriver                        0x0000000100cd10b8 cxxbridge1$str$ptr + 1887276
[2024-08-08T10:53:33.530Z] 1   chromedriver                        0x0000000100cc9794 cxxbridge1$str$ptr + 1856264
[2024-08-08T10:53:33.530Z] 2   chromedriver                        0x00000001008d882c cxxbridge1$string$len + 88524
[2024-08-08T10:53:33.530Z] 3   chromedriver                        0x000000010091c834 cxxbridge1$string$len + 367060

Operating System

Mac

Selenium version

python

What are the browser(s) and version(s) where you see this issue?

Chrome

What are the browser driver(s) and version(s) where you see this issue?

Chromedriver

Are you using Selenium Grid?

No response

[github-actions[bot]]

@Alperen-ylmaz, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[pmartinez1]

I have been working with selenium using python recently and am interested in helping out. Would it be fine if I took a look at this issue, is that fine or do is there a process (for triaging or resolving issues) I should follow? Thanks.

[pmartinez1]

Hi @Alperen-ylmaz , What version of Selenium are you running? Starting in version 4.11.X, you no longer need to use ChromeDriverManager, because Selenium added support for chromedriver. I was able to quickly get this working with the modifications below. Hoping this helps.

def setup_driver(): options = Options() options.add_argument('--no-sandbox') options.add_argument('--disable-dev-shm-usage') options.add_argument('--window-size=1920x1080') options.add_argument('--incognito') options.add_argument('--headless') # Run in headless mode options.add_argument('--disable-gpu') driver = webdriver.Chrome(options=options, service=Service())
return driver

[pujagani]

Can you try adding the argument "--headless=new"? Please refer to https://www.selenium.dev/blog/2023/headless-is-going-away/ for the explanation. That might help.

[Roopesh-Bharatwaj-K-R]

Thanks, I too have the same issue I will check for the same and will let you know.