search

thoughtbot / capybara-webkit

A Capybara driver for headless WebKit to test JavaScript web apps
https://thoughtbot.com/open-source
MIT License
1.97k stars 428 forks source link

WIP: Make running the selenium compatibility specs a separate task #1034

Closed twalpole closed 6 years ago

twalpole commented 6 years ago

Looking at PR# 833 I went ahead and added a test for multiple selects and set up the selenium compatibility specs to be run as a separate task. As can be seen from the failures capybara-webkit does not match selenium with either firefox or chrome for selects in any case. That being said, selenium with firefox doesn't match selenium with chrome either. What, if anything, would you want to do about this?

twalpole commented 6 years ago

@jferris @mhoran What, if anything, would you want to do about this?

jferris commented 6 years ago

The general idea behind these tests was to use Selenium/Firefox as the reference implementation. People would open bug reports because filling out forms didn't behave as they would expect, and frequently they'd add the line, "it works with Selenium." Comparing to Selenium was the easiest way I could think of to make sure the WebKit driver worked as expected.

If we're not going to use this as part of the regular build and run it on CI, I don't think it's worth having. It did catch a few regressions, however. What it means to fill out of a text field is unfortunately complex, and doing a brute force comparison to what Selenium did was pretty reliable, compared to trying to think of every reasonable event we would emit and test by hand. We may have enough general coverage in our test suite that this is no longer necessary, though.

twalpole commented 6 years ago

@jferris It is setup to run on CI, just as its own run (that's currently allowed to fail) because it requires installing Chrome, Firefox, geckodriver, and chromedriver. Interestingly failures are also platform dependent, since the text field filling out test fails against selenium with Firefox on MacOS due to focus and blur events, but doesn't fail on travis.

twalpole commented 6 years ago

Oh, and of course none of them match what you actually get if you manually select in the browser.

twalpole commented 6 years ago

@jferris @mhoran Another option would be to have the select behavior match the web driver spec behavior for option elements - https://w3c.github.io/webdriver/webdriver-spec.html#element-click step 7 - and ignore what selenium is currently outputting since it should eventually match the webdriver spec

jferris commented 6 years ago

Given how much it seems to vary now from backend to backend, I think it makes sense to delete this test entirely. I think it would be useful to try and follow the webdriver spec, but I'd be fine removing this test without introducing something new.

mhoran commented 6 years ago

I agree with @jferris; most of the tests for key events have been covered in other suites as well. I used the compatibility tests to help drive out the implementation, but given things behave differently now it seems like this is more trouble than it's worth.