ERR_NAME_NOT_RESOLVED when visiting https://webdriver.modheader.com/add?...

[ndtreviv]

 modheader version

I downloaded it from here: https://github.com/modheader/modheader_selenium/raw/main/chrome-modheader/modheader.crx today.

What I expected to happen:

I expected the call to webdriver.modheader.com to be intercepted by the extension and not to get a ERR_NAME_NOT_RESOLVED error.

Code snippet:

            WebDriverManager.chromedriver().setup();

            List<String> appliedOptions = new ArrayList<>();
            appliedOptions.addAll(DEFAULT_WEBDRIVER_PROPERTIES);

            ChromeOptions chromeOptions = new ChromeOptions();
            chromeOptions.addArguments(appliedOptions);
            chromeOptions.addExtensions(new File("/modheader.crx"));
            chromeOptions.setExperimentalOption("excludeSwitches", new String[]{"enable-automation"});
            chromeOptions.setExperimentalOption("useAutomationExtension", false);

            driver = new ChromeDriver(chromeOptions);
            driver.get("https://webdriver.modheader.com/add?Referer=" + URLEncoder.encode(referrer, Charset.defaultCharset()));

Logs:

2022-09-21 19:00:24.514  INFO 77077 --- [nio-8081-exec-2] i.g.bonigarcia.wdm.WebDriverManager      : Using chromedriver 105.0.5195.52 (resolved driver for Chrome 105)
2022-09-21 19:00:24.519  INFO 77077 --- [nio-8081-exec-2] i.g.bonigarcia.wdm.WebDriverManager      : Exporting webdriver.chrome.driver as /Users/me/.cache/selenium/chromedriver/mac-arm64/105.0.5195.52/chromedriver
Starting ChromeDriver 105.0.5195.52 (412c95e518836d8a7d97250d62b29c2ae6a26a85-refs/branch-heads/5195@{#853}) on port 54901
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
[1663783224.875][WARNING]: Deprecated chrome option is ignored: useAutomationExtension
[1663783224.875][WARNING]: Deprecated chrome option is ignored: useAutomationExtension
2022-09-21 19:00:25.465  INFO 77077 --- [ null to remote] o.o.selenium.remote.ProtocolHandshake    : Detected upstream dialect: W3C
2022-09-21 19:00:25.526  WARN 77077 --- [nio-8081-exec-2] o.o.selenium.devtools.CdpVersionFinder   : Unable to find an exact match for CDP version 105, so returning the closest version found: 102
2022-09-21 19:00:25.526  INFO 77077 --- [nio-8081-exec-2] o.o.selenium.devtools.CdpVersionFinder   : Found CDP implementation for version 105 of 102
2022-09-21 19:00:25.758  WARN 77077 --- [nio-8081-exec-2] c.c.s.services.FeatureExtractorService   : Couldn't prepare the webdriver: unknown error: net::ERR_NAME_NOT_RESOLVED
  (Session info: chrome=105.0.5195.125)
Build info: version: '4.2.1', revision: 'ac4d0fdd4a'
System info: host: 'mylaptop.lan', ip: 'fe80:0:0:0:145b:86e7:ade4:4635%en0', os.name: 'Mac OS X', os.arch: 'aarch64', os.version: '12.5.1', java.version: '17.0.3'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [c16bef64676a3c4892a554cc6afc06d6, get {url=https://webdriver.modheader.com/add?Referer=https%3A%2F%2Fsomesite.com}]
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 105.0.5195.125, chrome: {chromedriverVersion: 105.0.5195.52 (412c95e51883..., userDataDir: /var/folders/yc/p1gcr6s917b...}, goog:chromeOptions: {debuggerAddress: localhost:57302}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: MAC, proxy: Proxy(), se:cdp: ws://localhost:57302/devtoo..., se:cdpVersion: 105.0.5195.125, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
Session ID: c16bef64676a3c4892a554cc6afc06d6

org.openqa.selenium.WebDriverException: unknown error: net::ERR_NAME_NOT_RESOLVED
  (Session info: chrome=105.0.5195.125)

[hao1300]

Can you put a breakpoint in your test code before it crashes, then go to chrome://extensions in the test browser, enable "Developer mode" (top right corner), then click on ModHeader's "background page" link. This should open up the "Console". Can you send me the errors that you see there?

[ndtreviv]

It clicking the background page link doesn't do anything, but there is an error if I click the Update button after enabling Developer Mode (found after clicking around trying to figure out why it wasn't opening the background page):

If I go to here: chrome-extension://jciacimpfapbkckkegdfngcnbdpabaoi/_generated_background_page.html which is where I think the background page should be based on this then I get a blank page.

When I move past the point where it tries to load the header page I get a background page (iframe) link:

That link shows me this:

At this point the background page link is still empty.

[ndtreviv]

Man, I stepped through all that without --headless on and it all worked 😅 Does it not work in headless mode?

[hao1300]

--headless does not work, but --headless=chrome should work. Give that a shot.

[ndtreviv]

The plot thickens. If I'm not debugging (ie stepping through it slowly) even without --headless then it doesn't work. The page gets stuck on the background page and doesn't go any further, I guess because of the error.

I tried --headless=chrome but no joy. Here's the total set of options:

            "--start-maximized",
            "--kiosk",
            "--headless=chrome",
            "--disable-dev-shm-usage",
            "--ignore-certificate-errors",
            "--disable-blink-features=AutomationControlled",
            "--log-level=DEBUG",
            "--no-proxy-server",
            "--whitelisted-ips",
            "--no-sandbox",
            "--disable-gpu",
            "--disable-software-rasterizer"

[ndtreviv]

Wait...it's enabling developer mode that makes it work. If I don't manually go to chrome://extensions and flick that switch then it doesn't work, headless or not. Do I need another option in my chromedriver usage to make this work?

[hao1300]

I am not sure what is going on there. I tried running my example code (https://github.com/modheader/modheader_selenium/blob/main/examples/javawebdriver/ChromeExample.java) with no options and with your options, and they both work just fine.

I am running on Windows though. Not sure if there is something specific to running Chrome webdriver on Mac.

[ndtreviv]

I tried including this to enable developer mode on extensions by default:

            Map<String, Object> prefs = new HashMap<>();
            prefs.put("extensions.ui.developer_mode", true);
            chromeOptions.setExperimentalOption("prefs", prefs);

and it still gets stuck on background page. I'll try and get a screen recording to see if that helps.

[ndtreviv]

Here's a screen recording of it. As you can see, it goes to a page not found error briefly before going to the ModHeader page.

The page I'm trying to get is: http://httpbin.org/get which would show me my headers.

https://user-images.githubusercontent.com/1530653/191596632-47b441fa-4db1-465e-891b-343c06052fbc.mp4

[hao1300]

Can you try adding a small delay (maybe 1-2 seconds wait) before navigating to webdriver.modheader.com.

I am guessing what is happening underneath the hood is that the extension is still initializing.

[ndtreviv]

That works now, both without --headless and with --headless=chrome.

Thanks! Is there a more elegant way of waiting for the plugin to init other than a hard 2 second wait?

eg: In selenium I could wait for an element on page, for example?

[hao1300]

You can try using WebDriverWait until webdriver.modheader.com loads successfully.

On Wed, Sep 21, 2022, 2:58 PM Nathan Trevivian @.***> wrote:

That works now, both without --headless and with --headless=chrome.

Thanks! Is there a more elegant way of waiting for the plugin to init other than a hard 2 second wait?

— Reply to this email directly, view it on GitHub https://github.com/modheader/modheader_selenium/issues/36#issuecomment-1254166308, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGC3PX7XSALLYZXLQ24T4ATV7NSG5ANCNFSM6AAAAAAQSLRPOQ . You are receiving this because you commented.Message ID: @.***>

[ndtreviv]

--headless does not work, but --headless=chrome should work. Give that a shot.

--headless=chrome isn't truly headless. On the mac, for example, I get another chrome showing up in the app switcher (command-tab). When I use --headless I don't get that. I'm worried that this isn't doing what it's supposed to do in my linux-docker environment.

[ndtreviv]

Yeah, it's an issue. EG: https://stackoverflow.com/a/62943578/498391

I'm getting the same error ([SEVERE]: bind() failed: Cannot assign requested address (99)) on my linux environment when using --headless=chrome

[hao1300]

Unfortunately that is beyond my scope. You may want to reach out to Chromium / Selenium for helps on --headless=chrome. I think --headless itself will not support installing extensions.

[abaldwin-op]

@hao1300 Figured I'd share that I'm running into this issue (ERR_NAME_NOT_RESOLVED) on MacOS as well, but not running headless. What information would be most useful to troubleshoot? Is the debug crx still valid?

I can remove the (endless retry attempts..) if I revert to the old crx and domain. Seems to start working w/in 1 second, but not always.

            driver.get("https://webdriver.modheader.com/add?example=header");

            // Try again if the extension wasn't ready:
            //   https://github.com/modheader/modheader_selenium/issues/36
            while (driver.getCurrentUrl().contains("chrome-extension") == false) {
                log().info("waiting for modheader_selenium to initialize");
                try {
                    Thread.sleep(500);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
                driver.get("https://webdriver.modheader.com/add?example=header");
            }

            // Ensure the headers are loaded before continuing
            new WebDriverWait(driver, Duration.ofSeconds(5))
                    .until(ExpectedConditions.titleIs("Done"));

[hao1300]

@abaldwin-op - Did you download the CRX file from here? https://docs.modheader.com/advanced/selenium-webdriver - Click on "Download for Chrome"

When you see the ERR_NAME_NOT_RESOLVED error, are you able to bring up the ModHeader popup manually? And if so, are you able to modify header manually using ModHeader (without going through the webdriver.modheader.com URL)

[abaldwin-op]

@hao1300 - Appreciate the reply.

Odd. That's the one I have in my branch (md5) and it's working without the retry hack.

These are the two I tried before my hack:

• https://github.com/modheader/modheader_selenium/tree/main/chrome-modheader/modheader.crx
• https://drive.google.com/file/d/1zeI_FYYtDOxUgTW3d3cRfIV5Jl87RKq9/view

What's even more odd is the two CRX above also work now. I added proper URL encoding and a "Done" wait in troubleshooting, so it's possible I didn't catch that I resolved the issue that way...?

We can consider this resolved for now, I think. I'll open a new issue for further troubleshooting if I run into again.