google / tachometer

Statistically rigorous benchmark runner for the web
BSD 3-Clause "New" or "Revised" License
671 stars 22 forks source link

Add `browser.profile` setting and fix Firefox `addArguments` bug #226

Closed aomarks closed 3 years ago

aomarks commented 3 years ago

This PR adds a new browser.profile JSON config setting which can be used like this:

{
  "benchmarks": [
    {
      "url": "mybench.html",
      "browser": {
        "name": "firefox",
        "profile": "/Users/<username>/Library/Application Support/Firefox/Profiles/<profile-name>"
      }
    }
  ]
}

It also fixes a bug where addArguments was not being applied to Firefox, even though it was documented that it was supported.

For Chrome, it was previously supported and documented to use "addArguments": ["user-data-dir=<path>"] to achieve this same effect, but I found that the equivalent for Firefox ("-profile=<path>") caused Selenium to timeout trying to connect to the process. I wasn't able to figure out exactly why this was happening, but I did notice a dedicated setProfile method just for Firefox, which does work. So by adding the browser.profile setting, we now have a way to call this special API, plus the user doesn't need to remember the user-data-dir flag in Chrome.

Fixes https://github.com/Polymer/tachometer/issues/222

cc @guybedford

guybedford commented 3 years ago

I just tested this out here and it works perfectly for me. Thanks so much for filling in the gaps on this workflow @aomarks it's a huge help!

AndrewJakubowicz commented 3 years ago

Oops! I was completely fooled by that code folding. Thanks for answering my questions.

aomarks commented 3 years ago

Also threw in a fix to tests relating to chromedriver having updated to Chrome 94 while GitHub Actions is still on Chrome 93.