Selenium WebDriver cannot identify element on remote GitHub Ubuntu server

[aaptwo]

What happened?

The following script runs on IntelliJ IDE and on command line. It fails when I run it on remote GitHub Ubuntu server.

How can we reproduce the issue?

import io.github.bonigarcia.wdm.WebDriverManager;
//import org.apache.tools.ant.types.LogLevel;
import org.openqa.selenium.By;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.logging.LogEntries;
import org.openqa.selenium.logging.LogEntry;
import org.openqa.selenium.logging.LogType;
import org.openqa.selenium.logging.LoggingPreferences;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
import org.testng.Assert;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;

import java.time.Duration;
import java.util.Date;
import java.util.List;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import java.util.logging.Level;

public class HelloJavaTest {
    WebDriver driver;

    @BeforeClass
    public void setup() {
        WebDriverManager.chromedriver().setup();
        ChromeOptions chromeOptions = new ChromeOptions();
        chromeOptions.addArguments("--headless", "--disable-gpu", "--disable-infobars", "--ignore-certificate-errors","--disable-extensions","--no-sandbox","--disable-dev-shm-usage", "--window-size=1920,1080", "--remote-allow-origins=*");
        driver = new ChromeDriver(chromeOptions);
        driver.get("https://www.beha-amprobe.com/en/");
        driver.manage().window().maximize();
        driver.manage().timeouts().implicitlyWait(15, TimeUnit.SECONDS);
    }

    @Test
    public void testTopNavigation() {
        WebDriverWait wait = new WebDriverWait(driver, 30);
        wait.until(ExpectedConditions.visibilityOfAllElements(driver.findElements(By.xpath("//ul[@class='tb-megamenu-nav nav level-0 items-36']/li"))));
        List<WebElement> topNav = driver.findElements(By.xpath("//ul[@class='tb-megamenu-nav nav level-0 items-36']/li"));
        for(WebElement elem : topNav) {
            System.out.println(elem.getText());
        }
    }
}

Relevant log output

Actual output:
org.openqa.selenium.TimeoutException: Expected condition failed: waiting for visibility of all [[[ChromeDriver: chrome on LINUX (82aa2c95b7304dd2e1e70b49d3c2c6b5)] -> xpath: //ul[@class='tb-megamenu-nav nav level-0 items-36']/li]] (tried for 30 second(s) with 500 milliseconds interval)
Build info: version: '4.6.0', revision: '79f1c02ae20'
System info: os.name: 'Linux', os.arch: 'amd64', os.version: '5.15.0-1041-azure', java.version: '11.0.19'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 114.0.5735.198, chrome: {chromedriverVersion: 114.0.5735.90 (386bc09e8f4f..., userDataDir: /tmp/.com.google.Chrome.4ybJY9}, goog:chromeOptions: {debuggerAddress: localhost:33647}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: LINUX, proxy: Proxy(), se:cdp: ws://localhost:33647/devtoo..., se:cdpVersion: 114.0.5735.198, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}
Session ID: 82aa2c95b7304dd2e1e70b49d3c2c6b5
    at org.openqa.selenium.support.ui.WebDriverWait.timeoutException(WebDriverWait.java:87)
    at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:231)
    at BehaAmprobe.HomePage.validateTopNav(HomePage.java:165)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
... Removed 36 stack frames

Expected output:
Home
Products
Where to Buy
Support & Service
Announcements
About Us

Operating System

Ubuntu

Selenium version

Java 4.6

What are the browser(s) and version(s) where you see this issue?

Chrome 114.0.5735.198

What are the browser driver(s) and version(s) where you see this issue?

ChromeDriver 114.0.5735.90

Are you using Selenium Grid?

No

[github-actions[bot]]

@aaptwo, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[titusfortner]

1. We only support the latest version of Selenium (4.10)
2. Try using --headless=new
3. If the code works in one context but not another, any bugs are likely to be with chromedriver

[github-actions[bot]]

Hi, @aaptwo. This issue has been determined to require fixes in ChromeDriver.

You can see if the feature is passing in the Web Platform Tests.

If it is something new, please create an issue with the ChromeDriver team. Feel free to comment the issues that you raise back in this issue. Thank you.

[aaptwo]

Actually, I was using Selenium 4.6. I upgraded to Selenium 4.10 and using '--headless=new' (instead of '--headless'), but I get the same errors as before.

[titusfortner]

Does it work when you don't run it headless?

[aaptwo]

Without '--headless' or --headless-new' it won't run at all (no tests will run), I think because on remote server there is no display attached to it. With '--headless' or --headless-new', some tests pass, others fail because it cannot identify elements.

[titusfortner]

You can try taking a screenshot and seeing what is on the page. It could be a network /proxy issue where not everything actually loads. But not sure what to tell you here. There's no bug for us to fix, just something unexpected in your setup. Also, explicit waits and implicit waits don't work well together. Try removing the implicit wait.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.