Selenoid devtool APIS hang when run testcases parallely

[Manikandan-18]

I'm using selenoid 1.10.0 Linux binary Using chrome browser v81.0 docker image - selenoid/vnc_chrome:81.0 OS - Ubuntu - 18.04 Using cdt-java-client-2.1.0.jar file

I have run more than 500 test cases parallelly running with 20 to 50 browser, I'm taking a screenshot and navigate a page using selenoid devtool, if I run the single case or run the test cases parallelly below 10 browsers it working fine.

If I run the test cases parallelly using more than 10 browsers, below exception occurs some test case steps

1. Chrome devtool service creation API got hang
2. page.captureScreenshot(); API got hang
3. page.navigate(url); API got hang

that's why my test case is hanging infinitely

Note:

• I have run test case parallelly 20 browsers or more
• Single test case having a minimum of 20 steps every step I have to create a new devtool connection and take a screenshot

Is creating a devtool connection for every step is correct?

` In my code

public class DevTools {

public static ChromeDevToolsService connect(WebDriver driver, String driverIp) {
     try {
         // Init ChromeDevtools client
         URL url = new URL(driverIp);
         String webSocketUrl = String.format("ws://" + url.getHost() + ":" + url.getPort() + "/devtools/%s/page", SeCustomActions.getSessionId(driver));
         WebSocketService webSocketService = WebSocketServiceImpl.create(new URI(webSocketUrl));
         CommandInvocationHandler commandInvocationHandler = new CommandInvocationHandler();
         Map<Method, Object> commandsCache = new ConcurrentHashMap<>();
         ChromeDevToolsServiceConfiguration devtoolConf = new ChromeDevToolsServiceConfiguration();
         devtoolConf.setReadTimeout(300);
         ChromeDevToolsService devtools =
                 ProxyUtils.createProxyFromAbstract(
                         ChromeDevToolsServiceImpl.class,
                         new Class[] {WebSocketService.class, ChromeDevToolsServiceConfiguration.class},
                         new Object[] {webSocketService, devtoolConf},
                         (unused, method, args) ->  
                         commandsCache.computeIfAbsent(
                                 method,
                                 key -> {
                                     Class<?> returnType = method.getReturnType();
                                     return ProxyUtils.createProxy(returnType, commandInvocationHandler);
                                 }));
         commandInvocationHandler.setChromeDevToolsService(devtools);
         return devtools;
     }
     catch(Exception e) {
         e.printStackTrace();
     }
     return null;
}

public static void closeDevTool(ChromeDevToolsService devtool) {
    try {
        devtool.close();
    }
    catch(Exception e) {
        e.printStackTrace();
    }
}

public static String takeScreenshot(WebDriver driver, String driverIp) throws Exception {
    ChromeDevToolsService devtool = connect(driver, driverIp);
    Page page = devtool.getPage();
    String encodedScreenshot = page.captureScreenshot();
    encodedScreenshot = "data:image/png;base64," + encodedScreenshot;
    closeDevTool(devtool);
    return encodedScreenshot;
}

public static Navigate navigate(WebDriver driver, String driverIp, String url) throws Exception {
    ChromeDevToolsService devtool = connect(driver, driverIp);
    Page page = devtool.getPage();
    Navigate navigate = page.navigate(url);
    closeDevTool(devtool);
    return navigate;
}

public static void reload(WebDriver driver, String driverIp) throws Exception {
    ChromeDevToolsService devtool = connect(driver, driverIp);
    Page page = devtool.getPage();
    page.reload();
    closeDevTool(devtool);
}

} `

the below exception is thrown while creating devtool connection

com.github.kklisura.cdt.services.exceptions.WebSocketServiceException: Failed connecting to ws server {} at com.github.kklisura.cdt.services.impl.WebSocketServiceImpl.connect(WebSocketServiceImpl.java:120) at com.github.kklisura.cdt.services.impl.WebSocketServiceImpl.create(WebSocketServiceImpl.java:86) at automation.devtools.DevTools.connect(DevTools.java:34) at automation.devtools.DevTools.takeScreenshot(DevTools.java:70) Caused by: javax.websocket.DeploymentException: Handshake error. at org.glassfish.tyrus.client.ClientManager$3$1.run(ClientManager.java:679) at org.glassfish.tyrus.client.ClientManager$3.run(ClientManager.java:717) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.glassfish.tyrus.client.ClientManager$SameThreadExecutorService.execute(ClientManager.java:871) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:516) at org.glassfish.tyrus.client.ClientManager.connectToServer(ClientManager.java:378) at com.github.kklisura.cdt.services.impl.WebSocketServiceImpl.connect(WebSocketServiceImpl.java:98) ... 17 more Caused by: org.glassfish.tyrus.core.HandshakeException: Response code was not 101: 404. at org.glassfish.tyrus.client.TyrusClientEngine.processResponse(TyrusClientEngine.java:320) at org.glassfish.tyrus.container.grizzly.client.GrizzlyClientFilter.handleHandshake(GrizzlyClientFilter.java:346) at org.glassfish.tyrus.container.grizzly.client.GrizzlyClientFilter.handleRead(GrizzlyClientFilter.java:315) at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:526) at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) ... 1 more

the below exception is thrown while taking a screenshot

com.github.kklisura.cdt.services.exceptions.ChromeDevToolsInvocationException: Timeout expired while waiting for server response. at com.github.kklisura.cdt.services.impl.ChromeDevToolsServiceImpl.invoke(ChromeDevToolsServiceImpl.java:149) at com.github.kklisura.cdt.services.invocation.CommandInvocationHandler.invoke(CommandInvocationHandler.java:87) at com.sun.proxy.$Proxy21.captureScreenshot(Unknown Source)

Timeout expired i think i have set a 300 sec read timeout in chrome devtool configuration while creating a devtool connection ChromeDevToolsServiceConfiguration devtoolConf = new ChromeDevToolsServiceConfiguration(); devtoolConf.setReadTimeout(300);

Kindly check and give the solution for that Thanks in advance

[vania-pooh]

@Manikandan-Zoho Selenoid is just reverse-proxying websocket traffic, so could be Chrome issue.

[Manikandan-18]

@vania-pooh Now, I create devtool connections for every single step. Is this fine creating a single connection for a tab? Is there any problem with creating frequent devtool connections for every single step? Creating a parallel connection makes any problem with that? Please help me to solve this

[vania-pooh]

@Manikandan-18 I would expect to do one connection per Selenium session.

[Manikandan-18]

@vania-pooh Now I'm using one connection per tab, devtool connection creation working fine, but the below exception occurred sometimes.

page.captureScreenshot(); profiler.takePreciseCoverage();

timeout expired exception occurred, I think I was set the read timeout in chrome devtool service configuration due to the exception occur, before I set the read timeout the API hanged infinity. Please help me to solve this

com.github.kklisura.cdt.services.exceptions.ChromeDevToolsInvocationException: Timeout expired while waiting for server response. at com.github.kklisura.cdt.services.impl.ChromeDevToolsServiceImpl.invoke(ChromeDevToolsServiceImpl.java:149) at com.github.kklisura.cdt.services.invocation.CommandInvocationHandler.invoke(CommandInvocationHandler.java:87) at com.sun.proxy.$Proxy22.takePreciseCoverage(Unknown Source)

com.github.kklisura.cdt.services.exceptions.ChromeDevToolsInvocationException: Timeout expired while waiting for server response. at com.github.kklisura.cdt.services.impl.ChromeDevToolsServiceImpl.invoke(ChromeDevToolsServiceImpl.java:149) at com.github.kklisura.cdt.services.invocation.CommandInvocationHandler.invoke(CommandInvocationHandler.java:87) at com.sun.proxy.$Proxy28.captureScreenshot(Unknown Source)

[vania-pooh]

@vania-pooh yeah I have seen this, not sure whether this is a bug of Java client or of Chrome.

[Manikandan-18]

@vania-pooh How can I solve this issue? I need to take JS and CSS code coverage, any solution for this issue?

[vania-pooh]

@Manikandan-18 I would take thread dump of frozen code first.

[Manikandan-18]

Hi @vania-pooh any fix sent for the above issue? Now I'm using cdt-java-client-2.1.0.jar and using chrome browser 87 and 90 version, which cdt-java-client version I need to use?

[f1-work]

Keep in mind that the channel you opened with for devtools listen has a fixed capacity to buffer unread events. For example, if Chrome is sending events faster than your code can handle them, we have to allocate potentially unbounded memory to buffer all of the events.

Maybe this will solve the problem

[vania-pooh]

E.g. session.listen(devtools.network.ResponseReceived, buffer_size=1024)