Closed githubuser100007 closed 3 years ago
Updated to reflect OS was Windows.
This is what is being printed from Selenium Beta-2 (note the pageLoad and script timouts disregard my setting, and what is weird is that the outcome is timing out at 180000 ms, not the 600000 that was set, or the 300000 that is shown in these logs)
Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [472f80a2d4250c338c8fb9b8ade2c9be, getPageSource {}]
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 89.0.4389.90, chrome: {chromedriverVersion: 88.0.4324.96 (68dba2d8a0b14..., userDataDir: /tmp/.com.google.Chrome.374gW6}, goog:chromeOptions: {debuggerAddress:
localhost:39145}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: eager, platform: LINUX, platformName: LINUX, proxy: Proxy(), setWindowRect: true, strictFileInteractability: false, **timeouts: {implicit: 0, pageL
oad: 300000, script: 30000},** unhandledPromptBehavior: dismiss, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
Session ID: 472f80a2d4250c338c8fb9b8ade2c9be
Here is more stack dump, again happens at 3 minutes instead of the 10 minute timeout I set:
org.openqa.selenium.TimeoutException: java.util.concurrent.TimeoutException
Build info: version: '4.0.0-beta-2', revision: 'Unknown'
System info: host: 'DEV01', ip: '172.30.0.127', os.name: 'Windows Server 2016', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_231'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Command: [12c302ddd6915400e72f0497133acb7c, get {url=https://www.walmart.com/}]
Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 89.0.4389.90, chrome: {chromedriverVersion: 89.0.4389.23 (61b08ee2c5002..., userDataDir: C:\Users\user\AppData\Loc...}, goog:chromeOptions: {debuggerAddress: localhost:55683}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: eager, platform: WINDOWS, platformName: WINDOWS, proxy: Proxy(manual, http=tst05.de..., setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss, webauthn:extension:largeBlob: true, webauthn:virtualAuthenticators: true}
Session ID: 12c302ddd6915400e72f0497133acb7c
at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:71)
at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
at org.openqa.selenium.remote.http.netty.NettyHttpHandler.execute(NettyHttpHandler.java:51)
at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42)
at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56)
at org.openqa.selenium.remote.http.netty.NettyClient.execute(NettyClient.java:103)
at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:184)
at org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:153)
at org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:128)
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:620)
at org.openqa.selenium.remote.RemoteWebDriver.get(RemoteWebDriver.java:334)
at com.util.SeleniumBrowser.browse(SeleniumBrowser.java:169)
Caused by: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:206)
at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:65)
... 13 more
Could you please provide a complete code example to understand better what is happening? Code snippets are not enough in this case. Would be great to see how the driver is created and the url being loaded.
Note that I use the webVitalsSnippet script injection on a driver without proxy and it works fine. So I don't think that has anything to do with it, but I included it here anyways, since that is what is happening.
private RemoteWebDriver createDriver() throws Exception {
ChromiumDriver driver = null;
ChromeOptions options = new ChromeOptions();
options.setBinary("/usr/bin/chrome");
//remove unexepectedalertexceptions
options.addArguments("--silent", "--headless", "--blink-settings=imagesEnabled=false", "--disable-blink-features", "--disable-blink-features=AutomationControlled", "--disable-gpu", "--no-sandbox", "--window-size=1920,1200", "--ignore-certificate-errors", "--disable-application-cache");
options.setCapability(CapabilityType.UNEXPECTED_ALERT_BEHAVIOUR, UnexpectedAlertBehaviour.DISMISS);
// no downloads
Map<String, Object> chromePrefs = new HashMap<String, Object>();
chromePrefs.put("download_restrictions", 3);
options.setExperimentalOption("prefs", chromePrefs);
LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
options.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
options.setCapability( "goog:loggingPrefs", logPrefs );
options.setPageLoadStrategy(PageLoadStrategy.EAGER);
DriverService.Builder serviceBuilder = new ChromeDriverService.Builder().withSilent(true);
ChromeDriverService chromeDriverService = (ChromeDriverService)serviceBuilder.build();
chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));
//running crawlera-headless-proxy on port 3128: https://support.zyte.com/support/solutions/articles/22000203564-using-zyte-smart-proxy-manager-with-selenium
Proxy proxy = new Proxy();
proxy.setHttpProxy("127.0.0.1:3128");
proxy.setSslProxy("127.0.0.1:3128");
options.setCapability(CapabilityType.PROXY, proxy);
driver = new ChromeDriver(chromeDriverService, options);
// Inject Core Web Vitals scripts
Map<String, Object> parameters = new HashMap<String, Object>();
parameters.put("source",webVitalsSnippet);
driver.executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", parameters);
// Stop any unwanted downloading of *.pdf *.crdownload et al. files
parameters = new HashMap<String, Object>();
parameters.put("behavior","deny");
driver.executeCdpCommand("Browser.setDownloadBehavior", parameters);
driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(600000));
driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(5000));
return driver;
}
public static void main(String[] args) throws Exception {
RemoteWebDriver driver = createDriver();
long timer = System.currentTimeMillis();
try {
driver.get("https://www.walmart.com");
} catch (TimeoutException te2) {
log.warn("Chrome Driver thread timed out: " + (System.currentTimeMillis() - timer) ); // prints approximately 3 minutes, no matter what
} finally {
driver.quit();
}
}
webVitalsSnippet:
var vitalsCLS = 'unset';
function logCLS({value}) {
vitalsCLS = value.toString();
console.log('CLS: ' + vitalsCLS);
}
var vitalsFID = 'unset';
function logFID({value}) {
vitalsFID = value.toString();
console.log('FID: ' + vitalsFID);
}
var vitalsLCP = 'unset';
function logLCP({value}) {
vitalsLCP = value.toString();
console.log('LCP: ' + vitalsLCP);
}
window.onload = function() {
var script = document.createElement('script');
script.src = 'https://unpkg.com/web-vitals';
script.onload = function() {
// When loading `web-vitals` using a classic script, all the public
// methods can be found on the `webVitals` global namespace.
webVitals.getCLS(logCLS, true);
webVitals.getFID(logFID, true);
webVitals.getLCP(logLCP, true);
}
document.body.appendChild(script);
}
Thank you for sharing the script and the stack trace. In my attempt to recreate the issue, I have been running the same example that is shared with a few tweaks (commenting out the proxy bit since it was mentioned in the follow-up comment that it is not a concern here).
public static void main(String[] args) throws Exception {
SetTimeout setTimeout = new SetTimeout();
RemoteWebDriver driver = setTimeout.createDriver();
long timer = System.currentTimeMillis();
try {
driver.get("https://www.walmart.com/");
System.out.println(driver.getTitle());
} catch (TimeoutException te2) {
System.out.println("Chrome Driver thread timed out: " + (System.currentTimeMillis() - timer) ); // prints approximately 3 minutes, no matter what
} finally {
driver.quit();
}
}
private RemoteWebDriver createDriver() throws Exception {
ChromiumDriver driver = null;
ChromeOptions options = new ChromeOptions();
// options.setBinary("/usr/bin/chrome");
//remove unexepectedalertexceptions
options.addArguments("--silent", "--blink-settings=imagesEnabled=false", "--disable-blink-features", "--disable-blink-features=AutomationControlled", "--disable-gpu", "--no-sandbox", "--window-size=1920,1200", "--ignore-certificate-errors", "--disable-application-cache");
options.setCapability(CapabilityType.UNEXPECTED_ALERT_BEHAVIOUR, UnexpectedAlertBehaviour.DISMISS);
// no downloads
Map<String, Object> chromePrefs = new HashMap<String, Object>();
chromePrefs.put("download_restrictions", 3);
options.setExperimentalOption("prefs", chromePrefs);
LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
options.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
options.setCapability( "goog:loggingPrefs", logPrefs );
options.setPageLoadStrategy(PageLoadStrategy.EAGER);
DriverService.Builder serviceBuilder = new ChromeDriverService.Builder().withLogLevel(ChromeDriverLogLevel.INFO);
ChromeDriverService chromeDriverService = (ChromeDriverService)serviceBuilder.build();
chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));
//running crawlera-headless-proxy on port 3128: https://support.zyte.com/support/solutions/articles/22000203564-using-zyte-smart-proxy-manager-with-selenium
// Proxy proxy = new Proxy();
// proxy.setHttpProxy("127.0.0.1:3128");
// proxy.setSslProxy("127.0.0.1:3128");
// options.setCapability(CapabilityType.PROXY, proxy);
driver = new ChromeDriver(chromeDriverService, options);
//Inject Core Web Vitals scripts
// Map<String, Object> parameters = new HashMap<String, Object>();
// parameters.put("source", "var vitalsCLS = 'unset';\n" +
// "function logCLS({value}) {\n" +
// " \tvitalsCLS = value.toString();\n" +
// " \tconsole.log('CLS: ' + vitalsCLS);\n" +
// "}\n" +
// "\n" +
// "var vitalsFID = 'unset';\n" +
// "function logFID({value}) {\n" +
// " \tvitalsFID = value.toString();\n" +
// " \tconsole.log('FID: ' + vitalsFID);\n" +
// "}\n" +
// "\n" +
// "var vitalsLCP = 'unset';\n" +
// "function logLCP({value}) {\n" +
// " \tvitalsLCP = value.toString();\n" +
// " \tconsole.log('LCP: ' + vitalsLCP);\n" +
// "}\n" +
// "\n" +
// "window.onload = function() {\n" +
// "\t\n" +
// "\tvar script = document.createElement('script');\n" +
// "\tscript.src = 'https://unpkg.com/web-vitals';\n" +
// " \tscript.onload = function() {\n" +
// " \t\n" +
// " // When loading `web-vitals` using a classic script, all the public\n" +
// " // methods can be found on the `webVitals` global namespace.\n" +
// "\twebVitals.getCLS(logCLS, true); \n" +
// " webVitals.getFID(logFID, true); \n" +
// " webVitals.getLCP(logLCP, true); \n" +
// " \n" +
// " }\n" +
// "\n" +
// " document.body.appendChild(script);\n" +
// "\n" +
// "} ");
// driver.executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", parameters);
//Stop any unwanted downloading of *.pdf *.crdownload et al. files
// parameters = new HashMap<String, Object>();
// parameters.put("behavior","deny");
// driver.executeCdpCommand("Browser.setDownloadBehavior", parameters);
WebDriver.Timeouts pageLoadTimeout = driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(600000));
System.out.println(pageLoadTimeout.getPageLoadTimeout().toMillis());
WebDriver.Timeouts scriptTimeout = driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(600000));
System.out.println(scriptTimeout.getScriptTimeout().toMillis());
return driver;
}
Logs:
Mar 30, 2021 7:39:27 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Mar 30, 2021 7:39:28 PM org.openqa.selenium.devtools.CdpVersionFinder findNearestMatch
INFO: Found exact CDP implementation for version 89
600000
600000
Verify your identity
When the page load timeout is set to 2 milliseconds as shown below :
WebDriver.Timeouts pageLoadTimeout = driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(2));
System.out.println(pageLoadTimeout.getPageLoadTimeout().toMillis());
WebDriver.Timeouts scriptTimeout = driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(600000));
System.out.println(scriptTimeout.getScriptTimeout().toMillis());
Logs:
Mar 30, 2021 7:46:25 PM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
Mar 30, 2021 7:46:25 PM org.openqa.selenium.devtools.CdpVersionFinder findNearestMatch
INFO: Found exact CDP implementation for version 89
2
600000
Chrome Driver thread timed out: 36
I have run the same two code samples and checked the debug logs and the timeout values are set as expected with the Chrome driver.
An early timeout is not the problem. It's when you set it to 10 minutes that is the problem.
I'm wondering if the chromedriver doesn't support changing its timeouts, so anything earlier than it's default of 3 minutes is okay but anything after 3 minutes the chrome driver times out no matter what at 3 minutes.
As Diego rightly pointed out, the issue seems to be due to the default values of HTTP client config. The read timeout defaults to 3 mins. Hence as you mentioned, the issue springs up when the timeout is set higher than 3 mins.
The issue is not with the Chrome driver since Chrome driver debug logs show the timeout being set correctly and when the value is set less than 3 mins the value is considered.
The solution for this is to configure the client timeout when creating the driver using the RemoteWebDriverBuilder.
private RemoteWebDriver createDriver() throws Exception {
RemoteWebDriver driver = null;
ChromeOptions options = new ChromeOptions();
// options.setBinary("/usr/bin/chrome");
//remove unexepectedalertexceptions
options.addArguments("--unexpectedAlertBehaviour=dismiss","--silent", "--blink-settings=imagesEnabled=false", "--disable-blink-features", "--disable-blink-features=AutomationControlled", "--disable-gpu", "--no-sandbox", "--window-size=1920,1200", "--ignore-certificate-errors", "--disable-application-cache");
// no downloads
Map<String, Object> chromePrefs = new HashMap<String, Object>();
chromePrefs.put("download_restrictions", 3);
options.setExperimentalOption("prefs", chromePrefs);
LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
//options.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
options.setCapability("goog:loggingPrefs", logPrefs );
options.setPageLoadStrategy(PageLoadStrategy.EAGER);
DriverService.Builder serviceBuilder = new ChromeDriverService.Builder();
ChromeDriverService chromeDriverService = (ChromeDriverService) serviceBuilder.build();
chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));
//running crawlera-headless-proxy on port 3128: https://support.zyte.com/support/solutions/articles/22000203564-using-zyte-smart-proxy-manager-with-selenium
Proxy proxy = new Proxy();
proxy.setHttpProxy("127.0.0.1:3128");
proxy.setSslProxy("127.0.0.1:3128");
//options.setProxy(proxy);
//driver = new ChromeDriver(chromeDriverService, options);
driver = (RemoteWebDriver) RemoteWebDriver
.builder()
.config(ClientConfig.defaultConfig()
.connectionTimeout(Duration.ofMinutes(10))
.readTimeout(Duration.ofMinutes(10)))
.withDriverService(chromeDriverService)
.addAlternative(options)
.setCapability("proxy", proxy)
.build();
//Inject Core Web Vitals scripts
// Map<String, Object> parameters = new HashMap<String, Object>();
// parameters.put("source", "var vitalsCLS = 'unset';\n" +
// "function logCLS({value}) {\n" +
// " \tvitalsCLS = value.toString();\n" +
// " \tconsole.log('CLS: ' + vitalsCLS);\n" +
// "}\n" +
// "\n" +
// "var vitalsFID = 'unset';\n" +
// "function logFID({value}) {\n" +
// " \tvitalsFID = value.toString();\n" +
// " \tconsole.log('FID: ' + vitalsFID);\n" +
// "}\n" +
// "\n" +
// "var vitalsLCP = 'unset';\n" +
// "function logLCP({value}) {\n" +
// " \tvitalsLCP = value.toString();\n" +
// " \tconsole.log('LCP: ' + vitalsLCP);\n" +
// "}\n" +
// "\n" +
// "window.onload = function() {\n" +
// "\t\n" +
// "\tvar script = document.createElement('script');\n" +
// "\tscript.src = 'https://unpkg.com/web-vitals';\n" +
// " \tscript.onload = function() {\n" +
// " \t\n" +
// " // When loading `web-vitals` using a classic script, all the public\n" +
// " // methods can be found on the `webVitals` global namespace.\n" +
// "\twebVitals.getCLS(logCLS, true); \n" +
// " webVitals.getFID(logFID, true); \n" +
// " webVitals.getLCP(logLCP, true); \n" +
// " \n" +
// " }\n" +
// "\n" +
// " document.body.appendChild(script);\n" +
// "\n" +
// "} ");
// driver.executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", parameters);
//Stop any unwanted downloading of *.pdf *.crdownload et al. files
// parameters = new HashMap<String, Object>();
// parameters.put("behavior","deny");
// driver.executeCdpCommand("Browser.setDownloadBehavior", parameters);
WebDriver.Timeouts pageLoadTimeout = driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(600000));
System.out.println(pageLoadTimeout.getPageLoadTimeout().toMillis());
WebDriver.Timeouts scriptTimeout = driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(600000));
System.out.println(scriptTimeout.getScriptTimeout().toMillis());
return driver;
}
Can you please try as shown above and see if changing the client config values helps solve the issue you are facing? Thank you! Please provide feedback on the same.
I'm getting the following error when building the driver the new way:
java.lang.IllegalArgumentException: Illegal key values seen in w3c capabilities: [loggingPrefs, unexpectedAlertBehaviour]
at org.openqa.selenium.remote.NewSessionPayload.lambda$validate$5(NewSessionPayload.java:203)
at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:372)
at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:373)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.stream.ReferencePipeline$11$1.accept(ReferencePipeline.java:373)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at org.openqa.selenium.remote.NewSessionPayload.validate(NewSessionPayload.java:207)
at org.openqa.selenium.remote.NewSessionPayload.
The W3C valid capabilities are shown here https://www.w3.org/TR/webdriver1/#capabilities
loggingPrefs
and unexpectedAlertBehaviour
are not, hence the error message.
Removed those capabilities and now getting an error trying to execute two of the critical settings on the driver for us:
// Inject Core Web Vitals scripts
Map<String, Object> parameters = new HashMap<String, Object>();
parameters.put("source",webVitalsSnippet);
((ChromiumDriver)driver).executeCdpCommand("Page.addScriptToEvaluateOnNewDocument", parameters);
// Stop any unwanted downloading of *.pdf *.crdownload et al. files
parameters = new HashMap<String, Object>();
parameters.put("behavior","deny");
((ChromiumDriver)driver).executeCdpCommand("Browser.setDownloadBehavior", parameters);
Error:
java.lang.ClassCastException: org.openqa.selenium.remote.RemoteWebDriver cannot be cast to org.openqa.selenium.chromium.ChromiumDriver
Please use as a base the example provided above by @pujagani, which works. Otherwise this issue will become a debugging session and for that it would be preferrable for you to join our Slack/IRC channel https://www.selenium.dev/support/
Is there no other way to issue CDP requests? This is a prerequisite for us.
I understand the issue is about the 3 minutes timeout, which is triggered since it is the default one. We provided an example above that shows how to increase the timeout. Am I missing something?
Removing functionality to increase the timeout does not equal a workaround. Is this related to: https://github.com/SeleniumHQ/selenium/issues/7914
Thank you for providing the feedback. I received the same error while working with it, but to ensure the options are set as desired, "options.addArguments("--unexpectedAlertBehaviour=dismiss"
and "options.setCapability("goog:loggingPrefs", logPrefs);
was added in the previous example.
ChromeDriver 75.0.3770.8 onwards the http://chromedriver.chromium.org/downloads states that it has "Renamed capability loggingPrefs to goog:loggingPrefs, as required by W3C standard". So both are covered.
Now the next concern is valid regarding the CDP commands. I am sharing an example below which a workaround to set a client config as well as run the CDP commands. Since currently, I was not able to figure a direct way to do both in a simpler manner. (Will discuss this with the Selenium contributors for future implementations)
public class SetTimeout {
public static void main(String[] args) throws Exception {
SetTimeout setTimeout = new SetTimeout();
RemoteWebDriver driver = setTimeout.createDriver();
long timer = System.currentTimeMillis();
try {
driver.get("https://www.walmart.com/");
System.out.println(driver.getTitle());
} catch (TimeoutException te2) {
System.out.println("Chrome Driver thread timed out: " + (System.currentTimeMillis() - timer)); // prints approximately 3 minutes, no matter what
} finally {
driver.quit();
}
}
private RemoteWebDriver createDriver() throws Exception {
RemoteWebDriver driver = null;
ChromeOptions options = new ChromeOptions();
//remove unexepectedalertexceptions
options.addArguments("--unexpectedAlertBehaviour=dismiss", "--silent", "--blink-settings=imagesEnabled=false", "--disable-blink-features", "--disable-blink-features=AutomationControlled", "--disable-gpu", "--no-sandbox", "--window-size=1920,1200", "--ignore-certificate-errors", "--disable-application-cache");
// no downloads
Map<String, Object> chromePrefs = new HashMap<String, Object>();
chromePrefs.put("download_restrictions", 3);
options.setExperimentalOption("prefs", chromePrefs);
LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
options.setCapability("goog:loggingPrefs", logPrefs);
options.setPageLoadStrategy(PageLoadStrategy.EAGER);
DriverService.Builder serviceBuilder = new ChromeDriverService.Builder();
ChromeDriverService chromeDriverService = (ChromeDriverService) serviceBuilder.build();
chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));
chromeDriverService.start();
//running crawlera-headless-proxy on port 3128: https://support.zyte.com/support/solutions/articles/22000203564-using-zyte-smart-proxy-manager-with-selenium
// Proxy proxy = new Proxy();
// proxy.setHttpProxy("127.0.0.1:3128");
// proxy.setSslProxy("127.0.0.1:3128");
//options.setProxy(proxy);
ClientConfig config = ClientConfig.defaultConfig().connectionTimeout(Duration.ofMinutes(10))
.readTimeout(Duration.ofMinutes(10)).baseUrl(chromeDriverService.getUrl());
Tracer tracer = OpenTelemetryTracer.getInstance();
CommandExecutor executor = new HttpCommandExecutor(
buildChromiumCommandMappings("goog"),
config,
new TracedHttpClient.Factory(tracer, HttpClient.Factory.createDefault()));
driver = new RemoteWebDriver(executor, options);
//Inject Core Web Vitals scripts
Map<String, Object> parameters = new HashMap<String, Object>();
parameters.put("source", "var vitalsCLS = 'unset';\n" +
"function logCLS({value}) {\n" +
" \tvitalsCLS = value.toString();\n" +
" \tconsole.log('CLS: ' + vitalsCLS);\n" +
"}\n" +
"\n" +
"var vitalsFID = 'unset';\n" +
"function logFID({value}) {\n" +
" \tvitalsFID = value.toString();\n" +
" \tconsole.log('FID: ' + vitalsFID);\n" +
"}\n" +
"\n" +
"var vitalsLCP = 'unset';\n" +
"function logLCP({value}) {\n" +
" \tvitalsLCP = value.toString();\n" +
" \tconsole.log('LCP: ' + vitalsLCP);\n" +
"}\n" +
"\n" +
"window.onload = function() {\n" +
"\t\n" +
"\tvar script = document.createElement('script');\n" +
"\tscript.src = 'https://unpkg.com/web-vitals';\n" +
" \tscript.onload = function() {\n" +
" \t\n" +
" // When loading `web-vitals` using a classic script, all the public\n" +
" // methods can be found on the `webVitals` global namespace.\n" +
"\twebVitals.getCLS(logCLS, true); \n" +
" webVitals.getFID(logFID, true); \n" +
" webVitals.getLCP(logLCP, true); \n" +
" \n" +
" }\n" +
"\n" +
" document.body.appendChild(script);\n" +
"\n" +
"} ");
Command addScriptCommand = new Command(driver.getSessionId(), ChromiumDriverCommand.EXECUTE_CDP_COMMAND, ImmutableMap.of("cmd", "Page.addScriptToEvaluateOnNewDocument", "params", parameters));
driver.getCommandExecutor().execute(addScriptCommand);
//Stop any unwanted downloading of *.pdf *.crdownload et al. files
parameters = new HashMap<String, Object>();
parameters.put("behavior", "deny");
Command setDownloadBehaviorCommand = new Command(driver.getSessionId(), ChromiumDriverCommand.EXECUTE_CDP_COMMAND, ImmutableMap.of("cmd", "Browser.setDownloadBehavior", "params", parameters));
driver.getCommandExecutor().execute(setDownloadBehaviorCommand);
// driver.executeCdpCommand("Browser.setDownloadBehavior", parameters);
WebDriver.Timeouts pageLoadTimeout = driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(600000));
System.out.println(pageLoadTimeout.getPageLoadTimeout().toMillis());
WebDriver.Timeouts scriptTimeout = driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(600000));
System.out.println(scriptTimeout.getScriptTimeout().toMillis());
return driver;
}
private Map<String, CommandInfo> buildChromiumCommandMappings(String vendorKeyword) {
String sessionPrefix = "/session/:sessionId/";
String chromiumPrefix = sessionPrefix + "chromium";
String vendorPrefix = sessionPrefix + vendorKeyword;
HashMap<String, CommandInfo> mappings = new HashMap<>();
mappings.put(ChromiumDriverCommand.LAUNCH_APP,
new CommandInfo(chromiumPrefix + "/launch_app", HttpMethod.POST));
String networkConditions = chromiumPrefix + "/network_conditions";
mappings.put(ChromiumDriverCommand.GET_NETWORK_CONDITIONS,
new CommandInfo(networkConditions, HttpMethod.GET));
mappings.put(ChromiumDriverCommand.SET_NETWORK_CONDITIONS,
new CommandInfo(networkConditions, HttpMethod.POST));
mappings.put(ChromiumDriverCommand.DELETE_NETWORK_CONDITIONS,
new CommandInfo(networkConditions, HttpMethod.DELETE));
mappings.put(ChromiumDriverCommand.EXECUTE_CDP_COMMAND,
new CommandInfo(vendorPrefix + "/cdp/execute", HttpMethod.POST));
// Cast / Media Router APIs
String cast = vendorPrefix + "/cast";
mappings.put(ChromiumDriverCommand.GET_CAST_SINKS,
new CommandInfo(cast + "/get_sinks", HttpMethod.GET));
mappings.put(ChromiumDriverCommand.SET_CAST_SINK_TO_USE,
new CommandInfo(cast + "/set_sink_to_use", HttpMethod.POST));
mappings.put(ChromiumDriverCommand.START_CAST_TAB_MIRRORING,
new CommandInfo(cast + "/start_tab_mirroring", HttpMethod.POST));
mappings.put(ChromiumDriverCommand.GET_CAST_ISSUE_MESSAGE,
new CommandInfo(cast + "/get_issue_message", HttpMethod.GET));
mappings.put(ChromiumDriverCommand.STOP_CASTING,
new CommandInfo(cast + "/stop_casting", HttpMethod.POST));
mappings.put(ChromiumDriverCommand.SET_PERMISSION,
new CommandInfo(sessionPrefix + "/permissions", HttpMethod.POST));
return unmodifiableMap(mappings);
}
static final class ChromiumDriverCommand {
private ChromiumDriverCommand() {
}
static final String LAUNCH_APP = "launchApp";
static final String GET_NETWORK_CONDITIONS = "getNetworkConditions";
static final String SET_NETWORK_CONDITIONS = "setNetworkConditions";
static final String DELETE_NETWORK_CONDITIONS = "deleteNetworkConditions";
static final String EXECUTE_CDP_COMMAND = "executeCdpCommand";
// Cast Media Router APIs
static final String GET_CAST_SINKS = "getCastSinks";
static final String SET_CAST_SINK_TO_USE = "selectCastSink";
static final String START_CAST_TAB_MIRRORING = "startCastTabMirroring";
static final String GET_CAST_ISSUE_MESSAGE = "getCastIssueMessage";
static final String STOP_CASTING = "stopCasting";
static final String SET_PERMISSION = "setPermission";
}
}
I hope this helps fix the issue.
I get an error running this code. See below for logs.
10 May 2021 14:23:58,474 ERROR [Thread-1] (WebdriverProxyFactory.java:54) - Unable to create driver.
org.openqa.selenium.SessionNotCreatedException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure.
Build info: version: '4.0.0-beta-2', revision: 'Unknown'
System info: host: 'DEV01', ip: '172.30.0.127', os.name: 'Windows Server 2016', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_231'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Command: [null, newSession {desiredCapabilities=Capabilities {browserName: chrome, goog:loggingPrefs: org.openqa.selenium.logging..., pageLoadStrategy: eager, proxy: Proxy(manual, http=tst05.de...}}]
Capabilities {}
at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:638)
at org.openqa.selenium.remote.RemoteWebDriver.startSession(RemoteWebDriver.java:252)
at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:174)
Caused by: java.io.UncheckedIOException: java.net.ConnectException: Connection refused: no further information: localhost/0:0:0:0:0:0:0:1:10560
Thank you for providing an update. However, with the same example after multiple runs, I am not able to reproduce this error. Can you provide some more details, please? Please also provide the code snippet used to reproduce the issue.
Here is the code that I am running. Please let me know if I've somehow missed something.
RemoteWebDriver driver = null;
ChromeOptions options = new ChromeOptions();
options.setBinary("/usr/bin/chrome");
//remove unexepectedalertexceptions
options.addArguments("--unexpectedAlertBehaviour=dismiss", "--silent", "--headless", "--blink-settings=imagesEnabled=false", "--disable-blink-features",
"--disable-blink-features=AutomationControlled", "--disable-gpu", "--no-sandbox", "--window-size=1920,1200", "--ignore-certificate-errors", "--disable-application-cache");
// no downloads
Map<String, Object> chromePrefs = new HashMap<String, Object>();
chromePrefs.put("download_restrictions", 3);
chromePrefs.put("download.default_directory", "/dev/null");
options.setExperimentalOption("prefs", chromePrefs);
LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
options.setCapability( "goog:loggingPrefs", logPrefs );
options.setPageLoadStrategy(PageLoadStrategy.EAGER);
DriverService.Builder serviceBuilder = new ChromeDriverService.Builder().withSilent(true);
ChromeDriverService chromeDriverService = (ChromeDriverService)serviceBuilder.build();
chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));
ClientConfig config = ClientConfig.defaultConfig().connectionTimeout(Duration.ofMinutes(10))
.readTimeout(Duration.ofMinutes(10)).baseUrl(chromeDriverService.getUrl());
Tracer tracer = OpenTelemetryTracer.getInstance();
CommandExecutor executor = new HttpCommandExecutor(
buildChromiumCommandMappings("goog"),
config,
new TracedHttpClient.Factory(tracer, HttpClient.Factory.createDefault()));
driver = new RemoteWebDriver(executor, options);
//Inject Core Web Vitals scripts
Map<String, Object> parameters = new HashMap<String, Object>();
parameters.put("source", "var vitalsCLS = 'unset';\n" +
"function logCLS({value}) {\n" +
" \tvitalsCLS = value.toString();\n" +
" \tconsole.log('CLS: ' + vitalsCLS);\n" +
"}\n" +
"\n" +
"var vitalsFID = 'unset';\n" +
"function logFID({value}) {\n" +
" \tvitalsFID = value.toString();\n" +
" \tconsole.log('FID: ' + vitalsFID);\n" +
"}\n" +
"\n" +
"var vitalsLCP = 'unset';\n" +
"function logLCP({value}) {\n" +
" \tvitalsLCP = value.toString();\n" +
" \tconsole.log('LCP: ' + vitalsLCP);\n" +
"}\n" +
"\n" +
"window.onload = function() {\n" +
"\t\n" +
"\tvar script = document.createElement('script');\n" +
"\tscript.src = 'https://unpkg.com/web-vitals';\n" +
" \tscript.onload = function() {\n" +
" \t\n" +
" // When loading `web-vitals` using a classic script, all the public\n" +
" // methods can be found on the `webVitals` global namespace.\n" +
"\twebVitals.getCLS(logCLS, true); \n" +
" webVitals.getFID(logFID, true); \n" +
" webVitals.getLCP(logLCP, true); \n" +
" \n" +
" }\n" +
"\n" +
" document.body.appendChild(script);\n" +
"\n" +
"} ");
Command addScriptCommand = new Command(driver.getSessionId(), ChromiumDriverCommand.EXECUTE_CDP_COMMAND, ImmutableMap.of("cmd", "Page.addScriptToEvaluateOnNewDocument", "params", parameters));
driver.getCommandExecutor().execute(addScriptCommand);
//Stop any unwanted downloading of *.pdf *.crdownload et al. files
parameters = new HashMap<String, Object>();
parameters.put("behavior", "deny");
Command setDownloadBehaviorCommand = new Command(driver.getSessionId(), ChromiumDriverCommand.EXECUTE_CDP_COMMAND, ImmutableMap.of("cmd", "Browser.setDownloadBehavior", "params", parameters));
driver.getCommandExecutor().execute(setDownloadBehaviorCommand);
driver.manage().timeouts().pageLoadTimeout(Duration.ofMillis(600000));
driver.manage().timeouts().setScriptTimeout(Duration.ofMillis(5000));
The error occurs here:
driver = new RemoteWebDriver(executor, options);
Thank you for sharing the code snippet.
You have missed adding chromeDriverService.start();
as shared earlier in the code example as a fix for this issue.
Since the service is not started and RemoteWebDriver is trying to use its URL, thejava.net.ConnectException: Connection refused
is thrown.
Please add that after chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));
and give it a try.
That works! Thanks for the workaround.
Reopening in case you want to fix the original issue.
Got a bunch of Too many open files errors, do I have to close the service separately from the driver?
Please make sure to be using the most recent releases (beta-3
is the most recent).
I believe you have been able to move forward and I'd like to avoid having this issue turned into a debugging session. If there is a concrete issue with code showing how to reproduce it, please open a new issue.
Please join our Slack/IRC channel for more questions, we'd be happy to reply there. https://www.selenium.dev/support/
I understand. However, after running several large scale tests after the initial unit test, it appears that the linux machine runs out of sockets. So I am trying to confirm that the chromedriverservice.start() call doesn't need to be closed when the driver is closed (this is used in the context of a connection pool), which was not a requirement before in our original code.
Does driver.close handle the chromedriverservice.close() call?
Thank you for the feedback and the details. Probably this will help, quit the RemoteWebDriver then close the ChromeDriver service. I stepped through the code to confirm that RemoteWebDriver quit will not close the ChromeDriver service. Thanks for bringing this to our notice.
public class SetTimeout {
ChromeDriverService chromeDriverService;
public static void main(String[] args) throws Exception {
SetTimeout setTimeout = new SetTimeout();
RemoteWebDriver driver = setTimeout.createDriver();
long timer = System.currentTimeMillis();
try {
driver.get("https://www.walmart.com/");
System.out.println(driver.getTitle());
} catch (TimeoutException te2) {
System.out.println("Chrome Driver thread timed out: " + (System.currentTimeMillis() - timer)); // prints approximately 3 minutes, no matter what
} finally {
driver.quit();
setTimeout.getChromeDriverService().stop();
}
}
public ChromeDriverService getChromeDriverService() {
return chromeDriverService;
}
private RemoteWebDriver createDriver() throws Exception {
RemoteWebDriver driver = null;
ChromeOptions options = new ChromeOptions();
//remove unexepectedalertexceptions
options.addArguments("--unexpectedAlertBehaviour=dismiss", "--silent", "--blink-settings=imagesEnabled=false", "--disable-blink-features", "--disable-blink-features=AutomationControlled", "--disable-gpu", "--no-sandbox", "--window-size=1920,1200", "--ignore-certificate-errors", "--disable-application-cache");
// no downloads
Map<String, Object> chromePrefs = new HashMap<String, Object>();
chromePrefs.put("download_restrictions", 3);
options.setExperimentalOption("prefs", chromePrefs);
LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
options.setCapability("goog:loggingPrefs", logPrefs);
options.setPageLoadStrategy(PageLoadStrategy.EAGER);
DriverService.Builder serviceBuilder = new ChromeDriverService.Builder();
chromeDriverService = (ChromeDriverService) serviceBuilder.build();
chromeDriverService.sendOutputTo(new FileOutputStream("/dev/null"));
chromeDriverService.start();
// Rest of the code is same as shared earlier
🐛 Bug Report
I am setting the timeout to 10 minutes because I am using https://github.com/scrapinghub/crawlera-headless-proxy
I am seeing it timeout after 3 minutes (still downloading all of the JS/CSS files)
To Reproduce
Set page load timeout to 10 minutes
Expected behavior
Expecting it to timeout after 10 minutes, not 3.
Test script or set of commands reproducing this issue
Environment
OS: Windows Browser: Chrome Browser version: Version 89.0.4389.90 (Official Build) (64-bit) Browser Driver version: ChromeDriver 89.0.4389.23 Language Bindings version: Java 8 Selenium Grid version (if applicable): Beta 2