Linux version check hangs on System.call `/snap/bin/chromium --product-version`

berkes commented 4 years ago

Summary

Since chrome 81.0.4044.92 the linux version check hangs on /snap/bin/chromium --product-version

Debug Info

Please provide the following information for bug reports:

Webdrivers version: 4.3.0 (also reproduced on 4.2.0)
Ruby version: ruby 2.6.5p114
Operating system / CI Environment: Linux xxxx 5.3.0-46-generic #38-Ubuntu SMP Fri Mar 27 17:37:05 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Browser and version: chromium 81.0.4044.92

Expected Behavior

When running Webdrivers::Chromefinder.version it is expected to give back a version string.
When running e.g. rails test:system this happens as well, as somewhere down the stack, the Webdrivers::Chromefinder.version is called.

Actual Behavior

e.g. when running rails test:system. Regardless of the configured browser, the following can be seen:

A simpler reproduction is to run Webdrivers::Chromedriver.update:

The process hangs. Hitting ^C exits. A Process for chromium remains running: bin/sh /snap/chromium/1100/bin/chromium.launcher --product-version

2020-04-16 10:12:20 DEBUG Webdrivers Checking current version
2020-04-16 10:12:20 DEBUG Webdrivers /home/ber/.webdrivers/chromedriver is not already downloaded
2020-04-16 10:12:20 DEBUG Webdrivers making System call: ["/snap/bin/chromium", "--product-version"]
█

At █ it hangs.

Slightly unrelated

This is probably caused by ActionDispatch::SystemTest in Rails: regardless of configuring the type of browser (firefox, firefox_headless, chrome etc), chrome is tested. Somewhere down the line, Webdrivers::Chromedriver.update is called, causing the process to hang.

Described in https://github.com/rails/rails/issues/38856

Speculating on the cause

I did not update webdrivers conciously, but downgrading to webdrivers 4.2.0 did not solve the issue. I did not manually update chromium snap, but I do see an update landed on 2020-04-15, after which the problem started to occur. This is not solid proof, but does hint at a correlation.

So, it could be the snap implementation that spawns children, which do not exit. As can be witnessed by running

/snap/bin/chromium --product-version
ps faux | grep chromium
\_ /bin/sh /snap/chromium/1100/bin/chromium.launcher --product-version

But if so, maybe Webdrivers::System.call could implement a timeout. So that, at least, the process crashes at some point rather than hanging forever. Currently, it remains hanging even if the chromium processes are killed on system level.

kapoorlakshya commented 4 years ago

@berkes Thanks for the bug report. I am able to reproduce this on Ubuntu 18 with Chromium installed via snap.

The odd thing is that I am able to retrieve the version directly from /snap/bin/chromium in the terminal, but not using Webdrivers::Chromedriver.browser_version. This hangs for me as well.

Based on your ps output, I dug deeper into the chromium.launcher and figured out where the actual binary lives (/snap/chromium/current/usr/lib/chromium-browser/chrome). When I set the Selenium::WebDriver::Chrome.path to it, this issue is resolved.

irb(main):001:0> require 'webdrivers'
=> true

irb(main):002:0> Webdrivers.logger.level = :debug

irb(main):003:0> Selenium::WebDriver::Chrome.path = "/snap/chromium/current/usr/lib/chromium-browser/chrome"

irb(main):004:0> Webdrivers::Chromedriver.browser_version
2020-04-16 15:57:11 DEBUG Webdrivers Selenium::WebDriver::Chrome.path: /snap/chromium/current/usr/lib/chromium-browser/chrome
2020-04-16 15:57:11 DEBUG Webdrivers making System call: ["/snap/chromium/current/usr/lib/chromium-browser/chrome", "--product-version"]
2020-04-16 15:57:11 DEBUG Webdrivers System call returned: 81.0.4044.92

2020-04-16 15:57:11 DEBUG Webdrivers Browser version: 81.0.4044.92
=> #<Gem::Version "81.0.4044.92">

Can you give this a try? I'm going to have to spend more time to figure out how the launcher works and why it's not working with webdrivers as it does in the terminal.

berkes commented 4 years ago

@kapoorlakshya Correct!

With Selenium::WebDriver::Chrome.path = "/snap/bin/chromium" (the 'default') it hangs, with Selenium::WebDriver::Chrome.path = "/snap/chromium/current/usr/lib/chromium-browser/chrome" it runs and returns the version.

This makes me think the chrome snap incorrectly forks a process somewhere. This can be confirmed by running it manually:

$ pgrep chromium | wc -l
0
$ /snap/bin/chromium --product-version
81.0.4044.92
$ pgrep chromium | wc -l
1

I've added a note to a (probably) related bug-report upstream at canonical: https://bugs.launchpad.net/ubuntu/+source/chromium-browser/+bug/1859878

berkes commented 4 years ago

Canonical mentions that the bug is handled here: https://bugs.launchpad.net/ubuntu/+source/chromium-browser/+bug/1864901

kapoorlakshya commented 4 years ago

@berkes Thanks for the update. We'll keep this ticket open until the bug is fixed in the snap package.

wielinde commented 4 years ago

Canonical has set the bug to "fix released". After refreshing my snaps to that release I still encounter the same issues. I am not sure what to do next. However, waiting will probably not solve it.

berkes commented 4 years ago

Canonical has set the bug to "fix released". After refreshing my snaps to that release I still encounter the same issues. I am not sure what to do next. However, waiting will probably not solve it.

Numerous comments -including mine- are complaining it is not fixed. So waiting might still be an option.

In the mean time, maybe we can hunt down what binary+options the snap crap (sorry, but really, what a mess!) is trying to run? It runs through at least four layers of indirection, but in the end, should call a binary somewhere. Maybe we can revert to running that binary directly instead of running it through the layers-of-snap, which are broken for us?

nickgrim commented 4 years ago

I've created a PR (#169) which ~is a fix to this problem as-written, in that it~ prevents Webdrivers::Chromedriver.browser_version / Webdrivers::ChromeFinder.version from blocking indefinitely.

It doesn't fix the problem of Actually Running System Tests, in that running Selenium::WebDriver.for(:chrome) still blocks unless you also set Selenium::WebDriver::Chrome.path.

(EDIT: the PR breaks Windows, so I've closed it. The code might work for you, but if it breaks you can keep both pieces etc etc)

foton commented 4 years ago

Just for refference. I have success with just Selenium::WebDriver::Chrome::Service.driver_path = '/snap/bin/chromium.chromedriver' setting. No need to Selenium::WebDriver::Chrome.path = '/snap/bin/chromium'.

kapoorlakshya commented 3 years ago

@berkes Should we still keep this issue open? I'm not sure what we can do to make this work from our end. Looks like the status of the possibly related bug report that you shared earlier has been updated to 'Invalid'.

berkes commented 3 years ago

The workaround of

Selenium::WebDriver::Chrome::Service.driver_path = '/snap/bin/chromium.chromedriver'

Works on my side. Seeing as that the maintainers of the snap are not willing to acknowledge nor fix this issue, I'd say we best close it.

Sorry it too so long, I had reverted from chromium snap to an apt-version of chromium as workaround so testing this took some steps.

kapoorlakshya commented 3 years ago

@berkes Sounds good, and thanks again for reporting this issue!

And just to confirm, is '/snap/bin/chromium.chromedriver' acquired through the chromedriver snap package? As in, is the workaround to use the snap package instead of this gem?

berkes commented 3 years ago

And just to confirm, is '/snap/bin/chromium.chromedriver' acquired through the chromedriver snap package? As in, is the workaround to use the snap package instead of this gem?

No.

My workaround is to remove the snap and then install chromium with apt, using a PPA:

snap remove chromium
sudo add-apt-repository ppa:xalt7x/chromium-deb-vaapi

# Pin the PPA to ensure we install the xalt7x and not the package in the ubuntu store which is a wrapper
# around the snap. See https://launchpad.net/~xalt7x/+archive/ubuntu/chromium-deb-vaapi
cat <<EOF | sudo tee /etc/apt/preferences.d/pin-xalt7x-chromium-deb-vaapi
Package: *
Pin: release o=LP-PPA-xalt7x-chromium-deb-vaapi
Pin-Priority: 1337
EOF

sudo apt update
sudo apt install chromium

This removes snap and installs chromium using apt. The snap booting mechanism is what hangs and breaks, the apt version is just a (large) binary as one would expect.

Consequently, Chrome and therefore the tests and drivers become a lot faster in booting, and somewhat faster in operating. Factor 10+ in my case (lot's of tiny specs all booting a new chromium). Snap version of Chromium is famously slow, for some reason; quite likely the same root cause that makes the snap commands "hang" as reported here: a complex and heavy loading/sandboxing/booting mechanism.

kapoorlakshya commented 3 years ago

@berkes Ah, got it. Thanks for sharing that. I've added this info to the wiki for future reference.

titusfortner / webdrivers