"chrome_driver", Unexpected end of input

[hisapy]

Hello,

I started a headless chrome using this Dockerfile and added the following config for Hound

config :hound,
  driver: "chrome_driver",
  port: 9222,

However as soon as Hound.start_session() is called I get the following error:

 ** (exit) exited in: GenServer.call(Hound.SessionServer, {:change_session, #PID<0.862.0>, :default, [metadata: %{owner: #PID<0.862.0>, repo: Webapp.Repo}]}, 60000)
         ** (EXIT) an exception was raised:
             ** (Poison.SyntaxError) Unexpected end of input
                 (poison) lib/poison/parser.ex:54: Poison.Parser.parse!/2
                 (poison) lib/poison.ex:83: Poison.decode!/2
                 (hound) lib/hound/response_parser.ex:29: Hound.ResponseParser.parse/5
                 (hound) lib/hound/request_utils.ex:48: Hound.RequestUtils.handle_response/3
                 (hound) lib/hound/session_server.ex:78: Hound.SessionServer.handle_call/3
                 (stdlib) gen_server.erl:615: :gen_server.try_handle_call/4
                 (stdlib) gen_server.erl:647: :gen_server.handle_msg/5
                 (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

I'll appreciate some help here because PhantomJS is really unusable.

Thanks in advance

[danhper]

Hi, thanks for reporting.

I managed to reproduce your issue, but it only seems to occur when running Chrome driver inside Docker. For some reason, chromedriver returns a 404 response when running inside Docker. I improved a little the error messages: https://github.com/HashNuke/hound/pull/158, but as everything works when chromedriver is running "normally" (meaning not containerized), I do not think it is really a Hound issue. Did you manage to get some other webdriver clients to work with containerized Chrome?

In the meanwhile, if you just want to avoid phantomjs, you can simply use a non containerized chromedriver, it should work without any issue.

[darksheik]

Generally when I've encountered something like this, it's because the webdriver application did not successfully start and it sent something other than XML back to Hound. Perhaps the ResponseParser just needs to test for valid XML before trying to decode it with Poison.

[darksheik]

Oh look you did something like that 😄 https://github.com/HashNuke/hound/pull/158/files

[hisapy]

Hi,

Actually I think that the issue is that the chrome --headless ... is not a chrome driver. Apparently it uses other protocols under the hood. Moreover, when I tried it manually it only worked to open publicly available sites but didn't work for stuff at localhost.

Anyway, I'm going to update my local chrome as soon as I get back to my MacBook and give it another shot.

Just for the record, I tried it first based on https://github.com/HashNuke/hound/issues/135 ... did you actually try the new chrome headless or just the old chrome driver?

[darksheik]

I tried Chrome headless out myself yesterday and also did not get it to start a session on the port I asked it to (remote_debugging_port). This was the dev channel version 60.

Be interested to know the answer as my tests run really slowly in regular Chrome.

[danhper]

Oh, ok sorry I run a normal chromedriver outside of docker, so it's really not the same thing. Chrome headless seems to run with selenium, and Hound works with Selenium, so I think we should be able to make it work. Has anyone run into troubles with Selenium + Chrome headless?

[danhper]

After googling a little, someone has already done the hard work :smile: See https://github.com/yukinying/chrome-headless-browser-docker

I managed to run Hound with the following container

docker run -it --rm --name chrome --shm-size=1024m --cap-add=SYS_ADMIN  -p=127.0.0.1:4444:4444 yukinying/chrome-headless-browser-selenium

and the following setup:

config :hound,
  driver: "selenium",
  browser: "chrome"

Let me know how it goes for you.

I suppose selenium will have an official image for headless Chrome soon enough in https://github.com/SeleniumHQ/docker-selenium

[hisapy]

Hello again,

Thanks for your comments guys. In my case I tried to upgrade my Chrome to 59 but I'm stuck at 58 and 58 doesn't have the headless mode. See https://developers.google.com/web/updates/2017/04/headless-chrome

Also, I think I wasn't getting localhost to work with the original chrome headless image I posted originally is because there wasn't actually nothing running at localhost inside the container. My app is running at localhost:4001 but the browser inside the container obviously can't access it.

Now I'm running the image mentioned by @tuvistavie but I have the following error:

** (RuntimeError) <unknown>: Failed to read the 'localStorage' property from 'Window': Storage is disabled inside 'data:' URLs.
       (Session info: headless chrome=60.0.3080.5)
       (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 4.9.21-moby x86_64) (WARNING: The server did not provide any stacktrace information)
     Command duration or timeout: 80 milliseconds
     Build info: version: '3.3.1', revision: '5234b32', time: '2017-03-10 09:04:52 -0800'
     System info: host: '0bf2eed9dddd', ip: '172.17.0.3', os.name: 'Linux', os.arch: 'amd64', os.version: '4.9.21-moby', java.version: '1.8.0_121'
     Driver info: org.openqa.selenium.chrome.ChromeDriver
     Capabilities [{applicationCacheEnabled=false, rotatable=false, mobileEmulationEnabled=false, networkConnectionEnabled=false, chrome={chromedriverVersion=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5), userDataDir=/tmp/.org.chromium.Chromium.iEHycS}, takesHeapSnapshot=true, pageLoadStrategy=normal, databaseEnabled=false, handlesAlerts=true, hasTouchScreen=false, version=60.0.3080.5, platform=LINUX, browserConnectionEnabled=false, nativeEvents=true, acceptSslCerts=true, locationContextEnabled=true, webStorageEnabled=true, browserName=chrome, takesScreenshot=true, javascriptEnabled=true, cssSelectorsEnabled=true, unexpectedAlertBehaviour=}]
     Session ID: 69c48df2746ca53c90a07f4bf4141eeb

Obviously, this is something I need to google now.

I'll let you know my findings as soon as I get this working.

Again, thanks!... You can't imagine how much PhantomJS hurt me constantly crashing with Segmentation Fault: 11 or just crashing without any reason. I tried 2.1.1, even trying to install @2.1.3 you only get 2.1.1 and then tried 2.5-beta but couldn't make it work with React even with polyfills.

[hisapy]

I moved my localStorage stuff to on_exit but now the browser can't find my login-form. I have a SPA built with React.

In the container output I see the following when a new session is started

Done: [new session: Capabilities [{rotatable=false, nativeEvents=false, takesScreenshot=true, browserName=chrome, javascriptEnabled=false, chromeOptions={args=[--user-agent=Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36/BeamMetadata (g2gCZAACdjF0AAAAAmQABW93bmVyZ2QADW5vbm9kZUBub2hvc3QAAAHmAAAAAABkAARyZXBvZAASRWxpeGlyLldlYmFwcC5SZXBv)]}, version=, platform=ANY, cssSelectorsEnabled=true}]]

It says javascriptEnabled=false ... Could that be the reason my React app is not working? How can I start a session with javascriptEnabled=true?

[wstucco]

@hisapy

Hound.start_session(driver: %{javascriptEnabled: true})

[hisapy]

Hello again,

Finally, I got this working, but with some tweaks.

1. Make your Phoenix/Elixir app with a host different than localhost.

This is needed when using any Docker based headless browser unless the app is also running inside the container. In other words, if there is nothing running at, for example, http://localhost:3001 inside the container, then you'll only get a blank page.

# file: config/test.exs

config :webapp, Webapp.Endpoint,
  server: System.get_env("TEST_SERVER") == "true",
  http: [
    port: {:system, "TEST_PORT"}
  ],
  url: [
    host: {:system, "TEST_HOST"}
  ]

With this config, you can run your test with something like

MIX_ENV=test TEST_SERVER=true TEST_PORT=3333 TEST_HOST=192.168.0.11 mix test

Here I'm using the IP address of my localhost (outside the container). This IP is in an accessible network from inside the container.

2. Add Xvfb to the original Docker image.

Using just the chrome-browser-headless-selenium created with this Dockerfile won't work if you need to fill form inputs. As soon as you get to some fill_field like the following

        # fill in email
        form
        |> find_within_element(:id, "email")
        |> fill_field(user.email) # this will cause the error explained below

You'll get an error like

org.openqa.selenium.WebDriverException: unknown error: an X display is required for keycode conversions, consider using Xvfb

Notice that Xfvb is not needed if you're running your headless browser in a box with GUI (like your MacBook or Linux Desktop).

So once you start the container with the command in the README of the image but without --rm

docker run -it --name chrome --shm-size=1024m --cap-add=SYS_ADMIN -p=127.0.0.1:4444:4444 yukinying/chrome-headless-browser-selenium

1. Open another terminal and docker exec -u root -it chrome bash
2. Inside the container apt-get update && apt-get install xvfb
3. Exit the container
4. Stop the container (press ctrl+c) in the terminal where you started it.
5. Commit the changes made to the container into a new image docker commit CONTAINER_ID you/chrome-headless-selenium. You can get the CONTAINER_ID with docker ps -a, obviously, the container you're looking for is called chrome.

Now you have a new image you/chrome-headless-selenium with Xvfb installed

3. Start a new container with the new image containing Xvfb

Start the container and get into it with:

docker run --rm -it --entrypoint bash --name chrome-selenium2 --shm-size=1024m --cap-add=SYS_ADMIN -p=127.0.0.1:4444:4444 you/chrome-headless-selenium

And inside the container run:

DISPLAY=:1 xvfb-run java -jar /opt/selenium/selenium-server-standalone.jar

To exit just press ctrl+c and then exit the container. Repeat this last step each time you want to test. Notice that perhaps this could have been started from docker run but at the time of this writing I didn't try it.

4. Start testing with Hound

Put the following in your config/test.exs and start testing

config :hound,
  driver: "selenium",
  browser: "chrome"

Congrats!, Now you can also remove the previous chrome container.

5. Read more info, add this to Hound notes, or write a blog post

For info about running Chrome headless without Docker see:

• https://developers.google.com/web/updates/2017/04/headless-chrome

For info about running Chrome headless with Selenium outside Docker see:

• https://intoli.com/blog/running-selenium-with-headless-chrome/

[darksheik]

I would think running chrome headless should eliminate the need to have Xvfb. Do you have to first launch chrome --headless within the docker container and pick a --remote-debugging-port for selenium to connect to?

[hisapy]

Hi @darksheik

I would think running chrome headless should eliminate the need to have Xvfb

As I mentioned, Xvfb is only needed if you need to fill_field which by the way is something needed in almost every project. I believe that this should be fixed in future releases of chrome.

Do you have to first launch chrome --headless within the docker container and pick a --remote-debugging-port for selenium to connect to?

This is all configured in the image. There is chrome_launcher copied to the image that includes the --headless and --disable-gpu but I don't know yet how does selenium automagically instantiates a chrome driver and browser. I'm looking forward on this because I need to set the window size. I'm on my cell phone now and don't remember why setting window size from Hound didn't worked, but I think it was something related to chrome driver.

What I did notice from python and java examples is how you can pass start up options like window size, --headless, capabilities, etc to the browser from the test suite. In other words the test suite starts a selenium + chrome driver session and when the test ends it destroys the session and shutdown selenium and the browser... is there any way we can pass capabilities to browser from Hound or start selenium and browser similarly?

[wstucco]

is there any way we can pass capabilities to browser from Hound or start selenium and browser similarly?

@hisapy

yes there is: pass the parameters in the chromeOptions key of the capabilities

Hound.start_session(driver: %{
  browserName: "chrome",
  chromeOptions: %{"args" => [
    "--user-agent=Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36",
     "--headless", 
    "--disable-gpu", 
    "--no-first-run"
}})

[hisapy]

Thanks @wstucco

However, I have some more questions

1. What is the exact purpose of --no-first-run flag?

And regarding the Dockerfile

I renamed the file chrome_launcher.sh copied to /opt/google/chrome/google-chrome to google-chrome.OLD. Now, when I run the tests (without passing options from Hound) I can set_window_size and other stuff, but I think it is because the browser is not started as --headless --disable-gpu.

1. Is this possible because I'm using Xvfb?.
2. Is /opt/google/chrome/google-chrome an special script in Selenium+ChromeDriver or Chrome context?

And more, when I run DISPLAY=:1 xvfb-run java -jar /opt/selenium/selenium-server-standalone.jar, a Selenium server is started, and the first time I run the test, I get the following error:

** (exit) exited in: GenServer.call(Hound.SessionServer, {:change_session, #PID<0.478.0>, :default, [metadata: %{owner: #PID<0.478.0>, repo: Webapp.Repo}]}, 60000)
         ** (EXIT) an exception was raised:
             ** (MatchError) no match of right hand side value: {:error, :timeout}
                 (hound) lib/hound/session_server.ex:78: Hound.SessionServer.handle_call/3
                 (stdlib) gen_server.erl:615: :gen_server.try_handle_call/4
                 (stdlib) gen_server.erl:647: :gen_server.handle_msg/5
                 (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

However, from the second time on, the tests start working fine every time.

1. Is there a way to avoid the failure on the first time?

Notice that the error on first time appears both with the script at /opt/google/chrome/google-chrome and without it.

[hisapy]

Just for the record,

When I'm running headless, this is what happens when trying to set the window size

1:00:39.426 INFO - Executing: [get current window handle])
01:00:39.450 INFO - Done: [get current window handle]
01:00:39.545 INFO - Executing: [set window size])
01:00:46.887 INFO - Executing: [get: http://192.168.0.11:3333/login])
01:00:49.642 WARN - Exception thrown
org.openqa.selenium.WebDriverException: unknown error: cannot get automation extension
from unknown error: page could not be found: chrome-extension://aapnijgdinlhnhlmodcfapnahmbfebeb/_generated_background_page.html
  (Session info: headless chrome=60.0.3080.5)
  (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 4.9.21-moby x86_64) (WARNING: The server did not provide any stacktrace information)

Also, there are differences about in the error I get in Hound with and without --headless. For example, without --headless I get a more detailed and helpful errors:

20:59:19.868 [warn] unknown error: Element <input type="text" value="" readonly="" id="company.rpjaDate" class="md-text-field md-text md-text-field--floating-margin md-pointer--hover"> is not clickable at point (889, 497). Other element would receive the click: <section class="md-dialog-content md-dialog-content--padded md-dialog-content--picker">...</section>
  (Session info: chrome=60.0.3080.5)
  (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 4.9.21-moby x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 459 milliseconds
Build info: version: '3.3.1', revision: '5234b32', time: '2017-03-10 09:04:52 -0800'
System info: host: '69d7277713d3', ip: '172.17.0.3', os.name: 'Linux', os.arch: 'amd64', os.version: '4.9.21-moby', java.version: '1.8.0_121'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities [{applicationCacheEnabled=false, rotatable=false, mobileEmulationEnabled=false, networkConnectionEnabled=false, chrome={chromedriverVersion=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5), userDataDir=/tmp/.org.chromium.Chromium.mWMLke}, takesHeapSnapshot=true, pageLoadStrategy=normal, databaseEnabled=false, handlesAlerts=true, hasTouchScreen=false, version=60.0.3080.5, platform=LINUX, browserConnectionEnabled=false, nativeEvents=true, acceptSslCerts=true, locationContextEnabled=true, webStorageEnabled=true, browserName=chrome, takesScreenshot=true, javascriptEnabled=true, cssSelectorsEnabled=true, unexpectedAlertBehaviour=}]
Session ID: 941089993df78888bdd70195afc8fb35
20:59:29.327 [warn] unknown error: Element <input type="text" value="" readonly="" id="company.rpcDate" class="md-text-field md-text md-text-field--floating-margin md-pointer--hover"> is not clickable at point (889, 620). Other element would receive the click: <span data-reactroot="" class="md-dialog-container md-overlay md-pointer--hover md-overlay--active">...</span>
  (Session info: chrome=60.0.3080.5)
  (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 4.9.21-moby x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 434 milliseconds
Build info: version: '3.3.1', revision: '5234b32', time: '2017-03-10 09:04:52 -0800'
System info: host: '69d7277713d3', ip: '172.17.0.3', os.name: 'Linux', os.arch: 'amd64', os.version: '4.9.21-moby', java.version: '1.8.0_121'
Driver info: org.openqa.selenium.chrome.ChromeDriver
Capabilities [{applicationCacheEnabled=false, rotatable=false, mobileEmulationEnabled=false, networkConnectionEnabled=false, chrome={chromedriverVersion=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5), userDataDir=/tmp/.org.chromium.Chromium.mWMLke}, takesHeapSnapshot=true, pageLoadStrategy=normal, databaseEnabled=false, handlesAlerts=true, hasTouchScreen=false, version=60.0.3080.5, platform=LINUX, browserConnectionEnabled=false, nativeEvents=true, acceptSslCerts=true, locationContextEnabled=true, webStorageEnabled=true, browserName=chrome, takesScreenshot=true, javascriptEnabled=true, cssSelectorsEnabled=true, unexpectedAlertBehaviour=}]
Session ID: 941089993df78888bdd70195afc8fb35

... 

     ** (RuntimeError) invalid element state: Element is not currently interactable and may not be manipulated
       (Session info: chrome=60.0.3080.5)
       (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 4.9.21-moby x86_64) (WARNING: The server did not provide any stacktrace information)
     Command duration or timeout: 225 milliseconds
     Build info: version: '3.3.1', revision: '5234b32', time: '2017-03-10 09:04:52 -0800'
     System info: host: '69d7277713d3', ip: '172.17.0.3', os.name: 'Linux', os.arch: 'amd64', os.version: '4.9.21-moby', java.version: '1.8.0_121'
     Driver info: org.openqa.selenium.chrome.ChromeDriver
     Capabilities [{applicationCacheEnabled=false, rotatable=false, mobileEmulationEnabled=false, networkConnectionEnabled=false, chrome={chromedriverVersion=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5), userDataDir=/tmp/.org.chromium.Chromium.mWMLke}, takesHeapSnapshot=true, pageLoadStrategy=normal, databaseEnabled=false, handlesAlerts=true, hasTouchScreen=false, version=60.0.3080.5, platform=LINUX, browserConnectionEnabled=false, nativeEvents=true, acceptSslCerts=true, locationContextEnabled=true, webStorageEnabled=true, browserName=chrome, takesScreenshot=true, javascriptEnabled=true, cssSelectorsEnabled=true, unexpectedAlertBehaviour=}]
     Session ID: 941089993df78888bdd70195afc8fb35

But with --headless I get the error without the warning about the element causing the error:

** (RuntimeError) invalid element state: Element is not currently interactable and may not be manipulated
       (Session info: headless chrome=60.0.3080.5)
       (Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 4.9.21-moby x86_64) (WARNING: The server did not provide any stacktrace information)
     Command duration or timeout: 261 milliseconds
     Build info: version: '3.3.1', revision: '5234b32', time: '2017-03-10 09:04:52 -0800'
     System info: host: '69d7277713d3', ip: '172.17.0.3', os.name: 'Linux', os.arch: 'amd64', os.version: '4.9.21-moby', java.version: '1.8.0_121'
     Driver info: org.openqa.selenium.chrome.ChromeDriver
     Capabilities [{applicationCacheEnabled=false, rotatable=false, mobileEmulationEnabled=false, networkConnectionEnabled=false, chrome={chromedriverVersion=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5), userDataDir=/tmp/.org.chromium.Chromium.A6nem2}, takesHeapSnapshot=true, pageLoadStrategy=normal, databaseEnabled=false, handlesAlerts=true, hasTouchScreen=false, version=60.0.3080.5, platform=LINUX, browserConnectionEnabled=false, nativeEvents=true, acceptSslCerts=true, locationContextEnabled=true, webStorageEnabled=true, browserName=chrome, takesScreenshot=true, javascriptEnabled=true, cssSelectorsEnabled=true, unexpectedAlertBehaviour=}]
     Session ID: f5de068e679ec1e8c2a921bf4b9315a8

[wstucco]

@hisapy

it was an example on how to set chrome arguments, you don' really need no-first-run
you can find here a list of chromium command line switches

BTW you can set the binary that's called from the webdriver by setting the binary option in the capabilities map, like this

capabilities = %{
  browserName: "chrome",
  chromeOptions: [
    "args" => [...]
    "binary" => "{binary_path}", # for example /usr/local/bin/chrome_launcher.sh
  ...
}

[hisapy]

Awesome @wstucco ! ...

We should try to PR all the stuff you've shared so others can find them in the docs

Thank you

EDIT: I read the docs for --no-first-run but I think I didn't understand it :(

[dezmathio]

how would one go about running this without using the Docker images? I don't have an internal app I'm testing either, so the ip/ports for the web app don't really apply in my case.

I tried the simple scenario @tuvistavie gave with the docker cfg and then just selenium+chrome in the config; but I always get connection refused.

Is this possible to just run with a chrome canary executable locally? How do I point to it when passing the chromeOptions?

Appreciate the help offline @hisapy but still unable to get the example working in my scenario... At this point I'd be happy getting it to work even without Docker.

[wstucco]

@dezmathio

Is this possible to just run with a chrome canary executable locally?

As I mentioned here you can set the binary by using the driver's capabilities options.

For Chrome

capabilities = %{
  browserName: "chrome",
  chromeOptions: [
    "args" => [...]
    "binary" => "{binary_path}", # for example /usr/local/bin/chrome_launcher.sh
  ...
}

Hound.start_session(driver: capabilities)

[amokan]

Has anyone gotten chrome working using the chromeOptions: [binary: "<path>"] approach mentioned by @wstucco ? I'm not even seeing code in the library where it would utilize that argument.

[dezmathio]

@amokan I couldn't get it to work with his suggestion

[amokan]

Thanks @dezmathio - hopefully someone has more info on this or if this was removed at some point.

Below is the driver capabilities map I pieced together using the notes above - but like I said, I see no evidence in the source that the binary argument is supported - but i could just be missing something obvious.

%{
  browserName: "chrome",
  chromeOptions: %{
    "args" => ["--headless", "--disable-gpu"],
    "binary" => "/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome"
}}

[wstucco]

@amokan @dezmathio

It's not an Hound capability, but a Chromedriver option.
Hound forwards all the arguments passed as chromeOptions to the underlying driver

You can see it here - just look at the description of the binary param

You can test that it is working by running this snippet in iex

Hound.Session.make_capabilities(%{ 
  browserName: "chrome", 
  chromeOptions: %{ 
    "args" => ["--headless", "--disable-gpu"], 
    "binary" => "/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome" 
  }
})