Closed twalpole closed 6 years ago
@jferris @mhoran What, if anything, would you want to do about this?
The general idea behind these tests was to use Selenium/Firefox as the reference implementation. People would open bug reports because filling out forms didn't behave as they would expect, and frequently they'd add the line, "it works with Selenium." Comparing to Selenium was the easiest way I could think of to make sure the WebKit driver worked as expected.
If we're not going to use this as part of the regular build and run it on CI, I don't think it's worth having. It did catch a few regressions, however. What it means to fill out of a text field is unfortunately complex, and doing a brute force comparison to what Selenium did was pretty reliable, compared to trying to think of every reasonable event we would emit and test by hand. We may have enough general coverage in our test suite that this is no longer necessary, though.
@jferris It is setup to run on CI, just as its own run (that's currently allowed to fail) because it requires installing Chrome, Firefox, geckodriver, and chromedriver. Interestingly failures are also platform dependent, since the text field filling out test fails against selenium with Firefox on MacOS due to focus and blur events, but doesn't fail on travis.
Oh, and of course none of them match what you actually get if you manually select in the browser.
@jferris @mhoran Another option would be to have the select behavior match the web driver spec behavior for option elements - https://w3c.github.io/webdriver/webdriver-spec.html#element-click step 7 - and ignore what selenium is currently outputting since it should eventually match the webdriver spec
Given how much it seems to vary now from backend to backend, I think it makes sense to delete this test entirely. I think it would be useful to try and follow the webdriver spec, but I'd be fine removing this test without introducing something new.
I agree with @jferris; most of the tests for key events have been covered in other suites as well. I used the compatibility tests to help drive out the implementation, but given things behave differently now it seems like this is more trouble than it's worth.
Looking at PR# 833 I went ahead and added a test for multiple selects and set up the selenium compatibility specs to be run as a separate task. As can be seen from the failures capybara-webkit does not match selenium with either firefox or chrome for selects in any case. That being said, selenium with firefox doesn't match selenium with chrome either. What, if anything, would you want to do about this?