search

facebook / react

The library for web and native user interfaces.
https://react.dev
MIT License
228.19k stars 46.67k forks source link

Improving Browser QA #11079

Closed nhunzaker closed 4 years ago

nhunzaker commented 7 years ago

We manually test a lot of behavior that can not be captured by JSDOM. Change events, number input quirks, event pooling, error handling, etc.

Despite this test suite, comprehensive unit tests, alpha, beta, and release candidates, a number of bugs have slipped through. Particularly around change events.

I want to explore what it would look like to reduce the burden of manual browser QA when possible. Particularly around things like change events, which hook into JavaScript DOM APIs and should be automatable.

I've filed this issue for myself to explore the following areas (though I'd be happy for others to pitch in 😄):

I'm probably missing something, but this is where I'm going to start. I'll post follow-up comments as I learn stuff.

nhunzaker commented 7 years ago

cc @aweary and @jquense

aweary commented 7 years ago

Deploy a nightly or edge build of React with an accompanying JSFiddle to make it easier to determine if a fix works on master without needing to know how to setup React locally

@zpao used to host a site that deployed builds from master, I'll try to find it when I get home.

sophiebits commented 7 years ago

http://react.zpao.com/ (still builds from our CI)

gaearon commented 7 years ago

I think @sophiebits also suggested having contractors run the QA tests in different browsers. Not sure how hard it would be.

jquense commented 7 years ago

I think that having a few, focused selenium tests via sauces labs or the like for particularly fraught areas is a a reasonable idea. Yes they can be flakey but if they can at least provide some canary-in-the-coalmine early warnings that can inform where to spend time doing some manual QA. The unit-test nature of these tests too should reduce the flakey-ness, plus we get some insight when we break peoples selenium workflows.

CircleCI's v2 workflows feature also makes it easier to set up flows that aren't required or need manual approval.

jquense commented 7 years ago

if it helps also i have some experience with saucelabs/browserstack I've found SL to be less flakey across browsers and done a bunch of e2e testing with protractor and react with better than expected results. glad to help out here :)

aweary commented 7 years ago

I also want to call back to https://github.com/facebook/react/issues/9301. If we have a definitive list of the browsers we need to test in, we could make sure that we're not wasting time testing in unsupported browsers.

nhunzaker commented 6 years ago

I've started to work with NightwatchJS and BrowserStack Automate. It is very easy to run a test against all of the browsers we cover.

This flow would involve:

  1. Automated deployment to an integration URL whenever a pull request is sent out
  2. A hook to expose that URL to a pull request
  3. Run the automated tests against the URL, providing it as environment variable

Here's some code: https://github.com/nhunzaker/nightwatch-test

I'm able to run an automated test in Safari, Chrome, Firefox, IE, and Edge. It's pretty neat. BrowserStack even records the test output.

screen shot 2017-11-30 at 8 49 02 am

This isn't cheap though. If we went this route, we would need an Automate or Automate Mobile subscription. This is $1000-1500 for a yearly subscription.

Still. I think this a promising way to go. We could automate testing against ~30 browsers.

jquense commented 6 years ago

@nhunzaker that's awesome I was just thinking today i wanted to make some time to work on this. maybe can chat a bit on discord (or whatever)? I fairly recently hooked up selenium and e2e tests for a work app and have some thoughts and experiences that might be useful here, about which stack to use and browserstack vs saucelabs

klamping commented 6 years ago

I'd like to try helping out this cause. I still need to figure out what part I'd like to work on first, but I figure I'd start by saying hi here.

FYI, Sauce Labs offers a free plan for open-source projects, so you wouldn't need to worry about a monthly/yearly license. All you have to do is reach out to their customer support. [Edit: BrowserStack also has free plans for open-source projects]

On that note, Netlify is a static-site server (similar to github pages) and may be useful for hosting the fixtures pages. They also have a free open-source plan: https://www.netlify.com/open-source/

/cc @giltayar

klamping commented 6 years ago

@nhunzaker fyi, I copied your NightwatchJS example and modified it to use WebdriverIO. You can check it out here: https://github.com/klamping/wdio-react-example

Benefits of WebdriverIO over NightwatchJS are that WebdriverIO provide built-in services to connect to tools like Browserstack or Chromedriver, and it also provides more flexibility in how you write your tests.

Would love to get your thoughts on this direction, as I have some time to keep plugging away at writing some browser automation for the DOM fixtures.

klamping commented 6 years ago

So I've been doing some work on that test repo and wanted to provide a quick update.

First off, I changed the repo name so that it now lives here: https://github.com/klamping/react-webdriverio-tests

I've also added a few more tests and plan on adding many more to come.

Using CircleCI and Zapier, it's "integrated" with the React PRs to run the tests I do have on the PR code and print out a report for each PR.

I have a temporary issues page which stores the links to the PR, report and CircleCI job. You can check it out here: https://github.com/klamping/react-webdriverio-tests/issues/1

Right now, the tests are running in Firefox and Chrome on the CircleCI box. I'd love to get it integrated with BrowserStack or Sauce Labs, but don't have an account.

Any feedback or suggested direction would be much appreciated. I'm hoping this can help provide a good proof-of-concept (or even some real value) in to how automated browser testing works.

christian-bromann commented 6 years ago

@nhunzaker I am happy to power up your open source account at Sauce Labs for such beautiful open source projects like React 😉

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contribution.

stale[bot] commented 4 years ago

Closing this issue after a prolonged period of inactivity. If this issue is still present in the latest release, please create a new issue with up-to-date information. Thank you!