Chrome start issues - headless flag?

[Lazarus404]

Hey guys,

When trying to run tests in Docker, I get the typical failure error below:

Process exited with result 0 and output { WebDriverError: unknown error: Chrome failed to 
start: exited abnormally.
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so 
ChromeDriver is assuming that Chrome has crashed.)
    at Object.throwDecodedError (/KITE/KITE-Phone-Test/js/node_modules/selenium-webdriver/lib/error.js:550:15)
    at parseHttpResponse (/KITE/KITE-Phone-Test/js/node_modules/selenium-webdriver/lib/http.js:565:13)
    at Executor.execute (/KITE/KITE-Phone-Test/js/node_modules/selenium-webdriver/lib/http.js:491:26)
    at process._tickCallback (internal/process/next_tick.js:68:7)
  name: 'WebDriverError',
  remoteStacktrace: '#0 0x5586b814a579 <unknown>\n' }
TypeError: Cannot read property 'quit' of undefined
    at Phone.testScript (/KITE/KITE-Phone-Test/js/Phone.js:37:25)
    at process._tickCallback (internal/process/next_tick.js:68:7)
wrote all data to file

This happens because Chrome isn't starting in headless mode, so without a display driver, it crashes. My question is; where should I be putting or setting a flag to run it as headless? I can't seem to find the correct location and I see no examples demonstrating this. There is also no mention of this in the README. Thanks :-)

[namvuCosmo]

Hello,

You can enable headless mode like so :

    {
      "browserName": "chrome",
      "version": "83",
      "platform": "LINUX",
      "headless": true
    }

I'll try to remember to include this in the readme in the future :).

[nemani]

I have the same issue, and even with setting the "headless": true in the test config I was unable to run it in headless mode. I think chrome has changed and needs different flags than the one supported by the KITE runner right now.

[Lazarus404]

That does seem like it's the case. Here are the versions I am running:

# chromedriver --version
ChromeDriver 83.0.4103.39 (ccbf011cb2d2b19b506d844400483861342c20cd-refs/branch-heads/4103@{#416})
# google-chrome --version
Google Chrome 83.0.4103.61

My config is:

  "browserName": "chrome",
  "version": "83",
  "platform": "LINUX",
  "headless": true,
  "audio": {
    "directory": "/KITE/KITE-Phone-Test/js/files",
    "filename": "sample.wav",
    "duration": "01:00:00",
    "type": "Audio"
  }

[Lazarus404]

So, I found the general problem. The Chrome options aren't being set. When console.log'ing the chromeOptions variable in WebDriverUtility.js, I get {}. I'll see if I can get to the root of the cause.

[Lazarus404]

So, I think the issue seems to be the config isn't parsed into the initial class properly. This is the content of this in the root JS class:

{
  name: 'Phone test',
  numberOfParticipant: '1',
  id: '0',
  uuid: 'dfd34c39c593491598311cf265cd2e5d',
  reportPath: './temp/Phone.js_2020-06-22-174209_a9225/0',
  capabilities:
   { app: false,
     jsonConfig:
      { headless: 'true',
        browserName: {},
        audio: [Object],
        version: {},
        platform: {} },
     capability:
      { headless: true,
        windowSize: '1920,1200',
        technologyPreview: false,
        useFakeMedia: true,
        flags: [],
        audio: [Object],
        extraCapabilities: {},
        maxInstances: 5,
        gateway: 'none' },
     kind: 'browser',
     logger:
      { name: 'org.webrtc.kite.config.client.Client',
        debugEnabled: true },
     paas:
      { availableSlots: 0,
        region: 'NC',
        type: 'local',
        url: 'http://localhost:4444/wd/hub' },
     count: 5,
     exclude: false,
     browserSpecs:
      { extension: '',
        profile: '',
        driverString: '-Dwebdriver.chrome.driver=./chromedriver',
        browserName: 'chrome',
        version: '83',
        deviceName: 'unknown',
        platform: 'LINUX' },
     clientIndex: 0,
     region: 'NC',
     fullReset: false },
  remoteUrl: undefined,
  timeout: 60,
  report:
   AllureTestReport {
     name: 'Phone test',
     stage: 'scheduled',
     start: 1592847731932,
     uuid: undefined,
     parameters: [],
     attachments: [],
     steps: [],
     status: 'passed',
     ignore: false,
     labels: [],
     links: [],
     historyId: 'a8dc4fcb-5adf-488d-a434-008b593c6c91' },
  reporter:
   Reporter {
     reportPath: './temp/Phone.js_2020-06-22-174209_a9225/0',
     containers: [],
     tests: [],
     attachments: [] },
  networkInstrumentation: undefined,
  payload:
   { url: 'https://phone.google-staging.com',
     testTimeout: 60,
     users: [ [Object] ],
     phoneNumbers: [ [Object] ] },
  url: 'https://phone.google-staging.com',
  takeScreenshot: undefined }

[namvuCosmo]

oh, I didn't realize it was the JS code, I'll take a look.

[namvuCosmo]

So I dug a round in the code.

It was another flag that messes up the chrome in headless mode, I think this is something new in recent chrome version.

For now, the temporary solution I can offer you is to comment out the code for this flag. It is in the file \js\node_modules\kite-common\wdmgmt\WebDriverUtility.js in your test directory, line 86 to 88. You can comment out these lines for the windowSize flag, it is not supported in headless mode.

//  if (capability.windowSize) {
//    chromeOptions['args'].push("window-size=" + capabilities.windowSize);
//  }

I'll update the library kite-common and push a new version, you can just update the version for this later as a more permanent solution, so that you don't have to worry when you recompile the whole project.

[Lazarus404]

Thank you. That got it a bit further, but Selenium is still not playing ball:

 Process exited with result 0 and output Error: ECONNREFUSED connect ECONNREFUSED 127.0.0.1:4444
    at ClientRequest.<anonymous> (/KITE/KITE-Phone-Test/node_modules/selenium-webdriver/http/index.js:262:15)
    at emitOne (events.js:116:13)
    at ClientRequest.emit (events.js:211:7)
    at Socket.socketErrorListener (_http_client.js:387:9)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at emitErrorNT (internal/streams/destroy.js:64:8)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
TypeError: Cannot read property 'quit' of undefined
    at Phone.testScript (/KITE/KITE-Phone-Test/js/Phone.js:39:25)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
wrote all data to file

If I log out the capabilities passed to the Selenium builder, I see:

 { browserName: 'chrome',
  version: '83',
  gateway: null,
  platformName: undefined,
  platform: 'LINUX',
  'goog:chromeOptions':
   { args:
      [ '--use-fake-device-for-media-stream',
        '--use-fake-ui-for-media-stream',
        '--headless' ] } }

The URL passed to the builder is 'http://localhost:4444/wd/hub'.

I will come back if I figure out the cause.

[namvuCosmo]

I have updated the kite-common library (v0.1.14), headless chrome should work now.

[Lazarus404]

I have updated the kite-common library (v0.1.14), headless chrome should work now.

I still get the error I posted literally the same time you did :-)

[namvuCosmo]

yeah I noticed haha. What address were you expecting to be passed? Did you have a hub running at http://localhost:4444 ?

[Lazarus404]

Doesn't the test runner launch the grid hub for you?

Aha! I see an error way up the deployment stack. The hub failed to launch due to lack of a display driver.

# Unable to init server: Could not connect: Connection refused
# Failed to parse arguments: Cannot open display:
Unable to init server: Could not connect: Connection refused
# Failed to parse arguments: Cannot open display:
Unable to init server: Could not connect: Connection refused
# Failed to parse arguments: Cannot open display:

[namvuCosmo]

We do have some other scripts that will help run a grid, but it's not included in the run command for the test itself (r config.json).

The test expects a grid exists already, and the address can be pass in the config file, for example:

  "grids": [
    {
      "type": "local",
      "url": "http://192.168.1.222:4444/wd/hub"
    }
  ],

The grid doesn't need to be on your local machine, as long as you can connect to your grid, the test will launch the webdriver on the there.

[Lazarus404]

Okay. This needs to be contained, so I need to figure out how to run it without a display driver. Thanks for your help.

[nemani]

Can we also start echoing the exact command used to launch chrome / Firefox?

The original error of this thread just indicates that chrome has crashed without indicating why.

Thanks for all your help!!

[namvuCosmo]

Do you plan to run this in docker container? @Lazarus404

[Lazarus404]

Yes, that's exactly what I'm doing. I've had to hack a lot of the scripts, to get around the Y|N requests. If I can get this working, I can then start to look at making this a feature of the tool itself.

[namvuCosmo]

@nemaniarjun as you can see above, the webdriver will try to launch the chrome with a certain configuration:

 { browserName: 'chrome',
  version: '83',
  gateway: null,
  platformName: undefined,
  platform: 'LINUX',
  'goog:chromeOptions':
   { args:
      [ '--use-fake-device-for-media-stream',
        '--use-fake-ui-for-media-stream',
        '--headless' ] } }

AFAIK, this is just equivalent to launching chrome with flags google-chrome --headless .... Sometimes this mode is picky and some flags will cause it to crash.

I haven't have any trouble running headless on either physical machines or VM on aws though, as long as the chromedriver is correct.

[namvuCosmo]

@Lazarus404 I have looked into that recently, and I tried with the java code because I'm not that good with JS. I didn't include the grid setup in the Dockerfile but the rest seemed pretty straightforward.

There are some old closed issues where someone tried with docker as well, they might be a bit dated but maybe still can be of use to you.

Let me know if you have any further issues, I would like to have some docker image for KITE as well, and you can probably help me save a lot of time on JS version :-)

[Lazarus404]

Absolutely. I'm close to having this working. I will create a PR one I have something worth submitting.

[Lazarus404]

@namvuCosmo I've got the hub running, but I'm receiving an error when starting the webdriver and not sure what's causing it. There also seems to be a number of red herrings:

12:35:46.672 INFO [ActiveSessionFactory.lambda$apply$11] - Matched factory org.openqa.selenium.grid.session.remote.ServicedSession$Factory (provider: org.openqa.selenium.chrome.ChromeDriverService)
12:35:46.689 DEBUG [UrlChecker.waitUntilAvailable] - Waiting for [http://localhost:8100/status]
12:35:46.690 DEBUG [UrlChecker.lambda$waitUntilAvailable$1] - Polling http://localhost:8100/status
12:35:46.708 DEBUG [UrlChecker.lambda$waitUntilAvailable$1] - Polling http://localhost:8100/status
[1592915746.706][SEVERE]: bind() failed: Cannot assign requested address (99)
Starting ChromeDriver 83.0.4103.39 (ccbf011cb2d2b19b506d844400483861342c20cd-refs/branch-heads/4103@{#416}) on port 8100
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.
12:35:46.724 DEBUG [HttpURLConnection.writeRequests] - sun.net.www.MessageHeader@5ef8c48d5 pairs: {GET /status HTTP/1.1: null}{User-Agent: Java/11.0.7}{Host: localhost:8100}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}
12:35:46.728 DEBUG [HttpURLConnection.getInputStream0] - sun.net.www.MessageHeader@5947a5784 pairs: {null: HTTP/1.1 200 OK}{Content-Length: 248}{Content-Type: application/json; charset=utf-8}{cache-control: no-cache}
12:35:47.138 INFO [ProtocolHandshake.createSession] - Detected dialect: W3C
12:35:47.163 DEBUG [RemoteWebDriver.log] - Executing: newSession [null, newSession {desiredCapabilities=Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 83.0.4103.61, chrome: {chromedriverVersion: 83.0.4103.39 (ccbf011cb2d2b..., userDataDir: /tmp/.com.google.Chrome.BmrWEm}, goog:chromeOptions: {debuggerAddress: localhost:33203}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:virtualAuthenticators: true}}]
12:35:47.164 DEBUG [RemoteWebDriver.log] - Executed: [null, newSession {desiredCapabilities=Capabilities {acceptInsecureCerts: false, browserName: chrome, browserVersion: 83.0.4103.61, chrome: {chromedriverVersion: 83.0.4103.39 (ccbf011cb2d2b..., userDataDir: /tmp/.com.google.Chrome.BmrWEm}, goog:chromeOptions: {debuggerAddress: localhost:33203}, networkConnectionEnabled: false, pageLoadStrategy: normal, platformName: linux, proxy: Proxy(), setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:virtualAuthenticators: true}}]
12:35:47.170 INFO [RemoteSession$Factory.lambda$performHandshake$0] - Started new session c05ae1cbe42c948b267c4ce0b9c3b2b5 (org.openqa.selenium.chrome.ChromeDriverService)
2020-06-23 12:35:47,193 INFO  - [Phone.js_0 ] ERROR: ReferenceError: driver is not defined
 2020-06-23 12:35:47,195 INFO  - [Phone.js_0 ]     at Phone.testScript (/KITE/KITE-Phone-Test/js/Phone.js:16:31)
 2020-06-23 12:35:47,197 INFO  - [Phone.js_0 ]     at <anonymous>
 2020-06-23 12:35:47,198 INFO  - [Phone.js_0 ]     at process._tickCallback (internal/process/next_tick.js:188:7)
 12:35:47.208 DEBUG [WebDriverServlet.handle] - Found handler: c05ae1cbe42c948b267c4ce0b9c3b2b5 (org.openqa.selenium.chrome.ChromeDriverService)
12:35:47.209 DEBUG [WebDriverServlet.lambda$handle$0] - Handler thread for session c05ae1cbe42c948b267c4ce0b9c3b2b5 (chrome): Executing DELETE on /session/c05ae1cbe42c948b267c4ce0b9c3b2b5 (handler: ServicedSession)
12:35:47.212 DEBUG [HttpURLConnection.writeRequests] - sun.net.www.MessageHeader@4ecc23148 pairs: {DELETE /session/c05ae1cbe42c948b267c4ce0b9c3b2b5 HTTP/1.1: null}{User-Agent: selenium/4.0.0-alpha.7 (js linux)}{Accept-Encoding: gzip}{Accept: application/json; charset=utf-8}{Cache-Control: no-cache}{Pragma: no-cache}{Host: localhost:8100}{Connection: keep-alive}
12:35:47.264 DEBUG [HttpURLConnection.getInputStream0] - sun.net.www.MessageHeader@441c1ae94 pairs: {null: HTTP/1.1 200 OK}{Content-Length: 14}{Content-Type: application/json; charset=utf-8}{cache-control: no-cache}
12:35:47.268 DEBUG [ReverseProxyHandler.execute] - To downstream: {"value":null}
12:35:47.273 INFO [ActiveSessions$1.onStop] - Removing session c05ae1cbe42c948b267c4ce0b9c3b2b5 (org.openqa.selenium.chrome.ChromeDriverService)
12:35:47.279 DEBUG [HttpURLConnection.writeRequests] - sun.net.www.MessageHeader@2226b98b7 pairs: {DELETE /session/c05ae1cbe42c948b267c4ce0b9c3b2b5 HTTP/1.1: null}{Cache-Control: no-cache}{Pragma: no-cache}{User-Agent: Java/11.0.7}{Host: localhost:8100}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}
12:35:47.281 DEBUG [HttpURLConnection.getInputStream0] - sun.net.www.MessageHeader@54458c504 pairs: {null: HTTP/1.1 200 OK}{Content-Length: 14}{Content-Type: application/json; charset=utf-8}{cache-control: no-cache}
12:35:47.282 DEBUG [ReverseProxyHandler.execute] - To downstream: {"value":null}
12:35:47.283 DEBUG [UrlChecker.waitUntilUnavailable] - Waiting for http://localhost:8100/shutdown
12:35:47.284 DEBUG [UrlChecker.lambda$waitUntilUnavailable$2] - Polling http://localhost:8100/shutdown
12:35:47.285 DEBUG [HttpURLConnection.writeRequests] - sun.net.www.MessageHeader@1bc497b95 pairs: {GET /shutdown HTTP/1.1: null}{User-Agent: Java/11.0.7}{Host: localhost:8100}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}
12:35:47.287 DEBUG [HttpURLConnection.getInputStream0] - sun.net.www.MessageHeader@78396f0d3 pairs: {null: HTTP/1.1 200 OK}{Content-Length: 40}{Content-Type: application/json; charset=utf-8}
2020-06-23 12:35:47,289 INFO  - [Phone.js_0 ] wrote all data to file
 12:35:47.298 DEBUG [UrlChecker.lambda$waitUntilUnavailable$2] - Polling http://localhost:8100/shutdown
Process exited with result 0 and output ERROR: ReferenceError: driver is not defined
    at Phone.testScript (/KITE/KITE-Phone-Test/js/Phone.js:16:31)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
wrote all data to file

Firstly, it suggests chromedriver is being connected from a non-local connection (which makes no sense), but it also seems theres a binding failure, though of what I don't know

[Lazarus404]

Ignore. I'm an idiot :-)

I've got it all running, now. I'll set this up in a PR sometime next week and submit it. I'm sure it will benefit others.

[namvuCosmo]

can't wait :)

[nemani]

@Lazarus404 any updates? Installing this manually on EC2 is more pain than I expected.

[Lazarus404]

I've got it running locally in Docker, but haven't cleaned anything up, yet. I can share my Dockerfile if that helps?

[nemani]

Sure, that would be useful. Thanks!

On Wed, Jul 1, 2020, 7:24 PM Jahred Love notifications@github.com wrote:

I've got it running locally in Docker, but haven't cleaned anything up, yet. I can share my Dockerfile if that helps?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/webrtc/KITE/issues/134#issuecomment-652432791, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADX44NW7KIK3HD6HYOJLL2TRZM5ZXANCNFSM4OEPWRPQ .

[Lazarus404]

Dockerfile:

FROM selenium/standalone-chrome as KiteBase

ARG DEBIAN_FRONTEND=noninteractive
ENV DEBIAN_FRONTEND=noninteractive

USER root
RUN apt update
RUN apt install -y default-jre default-jdk openjdk-8-jdk maven git gcc build-essential
RUN apt-get install nano -y
RUN apt-get install -y sudo wget dpkg unzip xvfb nodejs npm fonts-liberation libappindicator3-1 libgbm1 libgtk-3-0 libxss1 xdg-utils libu2f-udev keyboard-configuration nmap

#KITE
RUN git clone https://github.com/webrtc/KITE.git KITE

WORKDIR /KITE/
COPY ./scripts ./scripts/
COPY ./third_party ./third_party/
COPY ./configureLinux.sh .

RUN chmod +x *.sh ./configureLinux.sh

ENV KITE_HOME=/KITE
RUN export KITE_HOME
ENV JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

RUN export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/bin
#RUN export PATH=$PATH:$JAVA_HOME/bin
ENV MAVEN_HOME=/usr/share/maven
RUN export MAVEN_HOME
RUN /bin/bash -c "source ~/.bashrc"
RUN echo 'export PATH=$PATH:/KITE/localGrid/chrome' >> ~/.bashrc

WORKDIR /KITE/KITE-Phone-Test
RUN /KITE/scripts/linux/path/c all
WORKDIR /KITE

RUN /KITE/configureLinux.sh

FROM KiteBase

COPY ./KITE-Framework .
COPY ./KITE-Engine .
COPY ./scripts/linux/setupLocalGrid.sh .
COPY ./scripts/linux/gridConfig.sh .
COPY ./scripts/linux/path/c .
COPY ./scripts/linux/path/r .
COPY ./scripts/linux/path/t .
COPY ./scripts/linux/path/a .
COPY ./scripts/linux/path/kite_init .
COPY ./scripts/linux/createFolderLocalGrid.sh .
COPY ./scripts/linux/installChrome.sh .
COPY ./scripts/linux/installFirefox.sh .
COPY ./scripts/linux/installSelenium.sh .
COPY ./scripts/linux/installMaven.sh .
COPY ./scripts/linux/installDrivers.sh .
COPY ./scripts/linux/interactiveInstallation.sh .
COPY ./scripts/entrypoint.sh .

ADD pom.xml /KITE/pom.xml

USER root

WORKDIR /KITE/KITE-Phone-Test

ENTRYPOINT ["/KITE/entrypoint.sh", "KITE-Phone-Test"]

entrypoint.sh

#! /bin/bash

cd "/KITE/${1}"
rm -rf node_modules
npm i

/KITE/localGrid/startGrid.sh

/KITE/r configs/js.phone.config.json

[Lazarus404]

The difficulty, here, is updating all the scripts so that they don't all ask for y confirmation.

[nemani]

You can pipe yes to the script to send all 'y' to the script

https://man7.org/linux/man-pages/man1/yes.1.html

On Wed, Jul 1, 2020, 7:36 PM Jahred Love notifications@github.com wrote:

The difficulty, here, is updating all the scripts so that they don't all ask for y confirmation.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/webrtc/KITE/issues/134#issuecomment-652439476, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADX44NWPCZTWYJ6QOJWWQSDRZM7GJANCNFSM4OEPWRPQ .