Chrome 72 breaking stubs

[amclelland]

Chrome version: 72.0.3626.81-1

We had our CI set up to install the latest chrome and our test suite suddenly broke today. Reverting the Chrome version to 71 fixed it. We are stubbing:

proxy.stub(/mockstripe/).and_return(body: mock_stripe)

and the error in CI was:

2019-01-29 21:20:24 +0000: Rack app error handling request { GET /javascripts/mockstripe/v2.js }
#<ActionController::RoutingError: No route matches [GET] "/javascripts/mockstripe/v2.js">

Seems like the new version of chrome was requesting something different and got around the proxy? Let me know if I can provide more info.

[thbar]

and got around the proxy

I wonder how related this could be to:

• https://github.com/thbar/repro-bug-proxy
• https://github.com/twalpole/apparition/issues/2
• https://github.com/machinio/cuprite/issues/44

(all in the context of using puffing-billy for exactly the same type of mocking).

EDIT: except that I met the issues above with Chrome 71.0.3578.98 if I'm not mistaken. EDIT 2: not necessarily the same issue ; mine was caused by SSL handling https://github.com/twalpole/apparition/issues/2#issuecomment-458807373

[ndbroadbent]

I think I'm experiencing the same issue, or at least a similar one. My tests are all passing in CI with Chrome 71, but are failing on my local machine with Chrome 72. I haven't changed anything else recently (at least, I don't think so.)

In my case, some requests are just stuck on "Pending" for a long time. I see this when I inspect the network requests in the Chrome developer tools. It seems to get stuck when fetching some things from a third-party service over SSL.

After a few minutes, my tests fail with:

     1.2) Failure/Error: @io.to_io.wait_readable(@read_timeout) or raise Net::ReadTimeout

          Net::ReadTimeout:
            Net::ReadTimeout
          # /Users/ndbroadbent/.rvm/gems/ruby-2.5.3/gems/webmock-3.4.2/lib/webmock/http_lib_adapters/net_http.rb:97:in `block in request'
          # /Users/ndbroadbent/.rvm/gems/ruby-2.5.3/gems/webmock-3.4.2/lib/webmock/http_lib_adapters/net_http.rb:110:in `block in request'
          # /Users/ndbroadbent/.rvm/gems/ruby-2.5.3/gems/webmock-3.4.2/lib/webmock/http_lib_adapters/net_http.rb:109:in `request'
          # /Users/ndbroadbent/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/http/default.rb:121:in `response_for'
          # /Users/ndbroadbent/.rvm/gems/ruby-2.5.3/gems/selenium-webdriver-3.141.0/lib/selenium/webdriver/remote/http/default.rb:76:in `request'

I found a download link for Chrome 71 on this page.

• Chrome 71.0.3578.80 for Mac

(Hopefully that site can be trusted. Unfortunately Google doesn't provide any older versions.)

Then I saw this article, and I tried to disable automatic updates:

defaults write com.google.Keystone.Agent checkInterval 0

I don't think that works now, because when I visited chrome://settings/help, I saw the message: "Updating Google Chrome".

Still, I managed to run my tests before it updated, and they all passed with Chrome 71. So there is definitely an issue with Chrome 72.

[ndbroadbent]

I've figured out how to install two different version of Google Chrome. I can use Chrome 72 for my main browser, and I have a separate installation of Chrome 71 that I use to run my tests (until this issue is sorted out.)

I downloaded Chrome 71 from the link above, and I dragged the "Google Chrome" application into /Applications. When Mac OS asked about the duplicate file, I chose "Keep Both" (which copied it to "Google Chrome 2".) Then I renamed "Google Chrome 2" to "Google Chrome 71".

I have the os gem in my Gemfile (gem 'os'), which I use to detect Mac (development) or Linux (CI and prod).

In my Capybara configuration, I set the binary in the chrome options:

options = Selenium::WebDriver::Chrome::Options.new
# ...
if OS.mac?
  # Capybara / puffing-billing proxy is broken on Chrome 72
  options.binary = '/Applications/Google Chrome 71.app/Contents/MacOS/Google Chrome'
end

This option makes chromedriver use the Google Chrome at /Applications/Google Chrome 71.app.

Here's the complete driver configuration that I'm using with Capybara:

# From: https://github.com/oesmith/puffing-billy/issues/193
Capybara.register_driver :headless_chrome_billy do |app|
  capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
    acceptInsecureCerts: true,
    loggingPrefs: { browser: 'ALL' }
  )
  options = Selenium::WebDriver::Chrome::Options.new
  # options.headless!
  options.add_argument('--disable-gpu')
  # options.add_argument('--disable-web-security')

  if OS.mac?
    # Capybara / puffing-billing proxy is broken on Chrome 72
    options.binary = '/Applications/Google Chrome 71.app/Contents/MacOS/Google Chrome'
  end

  # See: https://developers.google.com/web/updates/2017/04/headless-chrome
  options.add_argument('--no-sandbox')

  options.add_argument('--window-size=1280,1000')
  options.add_argument("--proxy-server=#{Billy.proxy.host}:#{Billy.proxy.port}")
  proxy_bypass_list = [
    '127.0.0.1',
    'localhost',
  ].join(';')
  options.add_argument("--proxy-bypass-list=#{proxy_bypass_list}")

  Capybara::Selenium::Driver.new app,
    browser: :chrome,
    options: options,
    desired_capabilities: capabilities,
    driver_opts: {
      log_path: Rails.root.join('log/chromedriver.log').to_s,
      verbose: true,
    }
end
Capybara.javascript_driver = :headless_chrome_billy

Hope that helps someone, and I look forward to getting this fixed in Chrome 72.

[pouellet]

ndbroadbent try adding the following to your chrome argument list: --tls13-variant=disabled. Chrome 72 now prevent downgrading to TLS 1.2 by default (https://www.chromium.org/Home/tls13).

I did a diff of the Wireshark traffic between Chrome 71 and Chrome 72. Chrome 72 lists the TLS 1.3 ciphers during the SSL handshake and somehow eventmachine responds with a Change Cipher TLS_AES_128_GCM_SHA256 which force a retrigger of the SSL handshake. eventmachine or puffing-billy does not handle this correctly and falls into an infinite loop of aborted SSL handshakes.

I thinks this might explain why this happened as well: https://github.com/oesmith/puffing-billy/issues/193#issuecomment-379249599

[ndbroadbent]

Thanks a lot @pouellet! I just tried adding options.add_argument('--tls13-variant=disabled'), and this works in normal Chrome 72, but I still have the same issue when running in headless mode. Maybe the --tls13-variant is not properly supported when running Chrome this way.

But yes it does sound like an underlying issue in eventmachine or puffing-billy, so it would be great to fix the root cause and support TLS 1.3.

[pouellet]

@ndbroadbent agreed, the --tls13-variant=disabled switch doesn't seem to have any effect in headless mode.

I've done some more tests on this and managed to get something working with headless, but it requires code changes in puffing-billy and pointing to the eventmachine master branch.

eventmachine pushed some changes to deal with TLS 1.3 a few weeks ago, so I've first tried updating it and hoped that everything would work fine, but the Change Cipher TLS_AES_128_GCM_SHA256 still happens and lead to the same infinite loop.

This being said, it is possible for puffing-billy to be more restrictive when it initiates the TLS handshake and explicitly disable TLS 1.3 by passing an extra ssl_version: ['TLSv1_2'] argument to this call https://github.com/oesmith/puffing-billy/blob/088bbc94ec91076df7f28fa9630d4b84ddc761e5/lib/billy/proxy_connection.rb#L56

This unfortunately only works in combination with the eventmachine TLS1.3 changes because unless specifically disabled, OpenSSL will default to having it in its list of supported protocols.

TL; DR; There seems to be two options in fixing this: explicitly disabling TLS1.3 support in puffing-billy once eventmachine gets released or figuring out why the Change Cipher triggers this infinite loop. I unfortunately don't have time at this point to dig deeper in how eventmachine works, but maybe previous contributors to puffing-billy could pitch in.

[ndbroadbent]

Awesome, thanks a lot for looking into this! The eventmachine update and ssl_version: ['TLSv1_2'] argument sounds like a reasonable workaround for now.

[pouellet]

Happy to help! In case you need a monkey patch for this in the meantime:

# frozen_string_literal: true

require 'billy/proxy_connection'
module Billy
  class ProxyConnection

    private

    alias_method :original_certificate_chain, :certificate_chain

    def certificate_chain(url)
      original_certificate_chain(url).merge(ssl_version: %w[TLSv1_2])
    end
  end
end

[ronwsmith]

@amclelland is this still an issue?

[amclelland]

@ronwsmith yes, still getting weird routing errors with latest chrome & billy that I don't get when running Chrome 71

[ronwsmith]

@amclelland if you're still getting issues on Chrome 79, can you add them to this issue?

[ndbroadbent]

Hello, I've just updated from Chrome 76 to 79.0.3945.130, and now I'm starting to see some new SSL errors in the console:

123145383469056:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:ssl/record/rec_layer_s3.c:1544:SSL alert number 46
123145383469056:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:ssl/record/rec_layer_s3.c:1544:SSL alert number 46
123145383469056:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:ssl/record/rec_layer_s3.c:1544:SSL alert number 46
...

I'm trying to proxy a few third-party requests, so here's the error messages in Chrome developer tools:

GET https://fonts.googleapis.com/css?family=Inconsolata... net::ERR_CERT_INVALID
GET https://js.stripe.com/v3/ net::ERR_CERT_INVALID

Tried the Billy::ProxyConnection workaround mentioned above, and also options.add_argument('--tls13-variant=disabled'), but neither of these fix the issue.

I don't know if this is related to the previous issue, or if it's a new issue. But it looks like proxying SSL requests is broken again. Has anyone else run into this?

UPDATE: I forked puffing-billy to relax the eventmachine version, and then used the eventmachine from the master branch on GitHub: 1.3.0.dev.1. Unfortunately this still didn't fix the issue, and I still get the same SSL errors:

123145388773376:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:ssl/record/rec_layer_s3.c:1544:SSL alert number 46

[0xGuybrush]

I'm hitting into the same thing as @ndbroadbent on Chrome 79 (ERR_CERT_INVALID / 123145404719104:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:ssl/record/rec_layer_s3.c:1544:SSL alert number 46).

(Just setting up for the first time with Puffing Billy today so not something that worked on previous version)

[taufek]

To overcome below error

routines:ssl3_read_bytes:sslv3 alert certificate unknown:ssl/record/rec_layer_s3.c:1544:SSL alert number 46

I've added following option

options.add_argument('--ignore-certificate-errors')

[ronwsmith]

Stale issue, closing. Feel free to reopen.