Selenium-style UI tests

[dbrgn]

(This is more of a brainstorming issue than a concrete feature request.)

wasm-bindgen provides a great way to do headless browser tests using #[wasm_bindgen_test]. However, it only offers access on the DOM level. Some UI interactions like "click on this content editable div" are hard to replicate using only DOM Events.

My first approach would be simply setting up browser tests with selenium, in parallel to the wasm-bindgen tests. But maybe we can do better?

wasm-bindgen already uses the webdriver protocol to launch the tests, right? Could we build on top of that to enable selenium-style browser tests? I saw that someone started implementing the webdriver spec here: https://github.com/saresend/selenium-rs (ping @saresend)

Here's what such an API might look like (with dependency injection, could also be done as a plain library call):

#[wasm_bindgen_test]
#[webdriver_test]
fn test_handle_clicks(webdriver: &WebDriver) {
    my_wasm_lib::prepare_element(WRAPPER_ID);
    let document = web_sys::window().unwrap().document().unwrap();
    let button = document.get_element_by_id(INPUT_FIELD_ID).unwrap();
    webdriver.click(button).unwrap();
}

[saresend]

This is something I was considering for the library alone, providing a sort of #[webdriver_test] macro that would hook into usual rust test harnesses, and enable test configuration and potentially data injection into an application being tested. I'm curious if you think this would be enough, or should there be some additional work to do specifically for using selenium in a wasm context?

[corbinu]

@dbrgn Might want to take a look at https://github.com/atroche/rust-headless-chrome

Firefox has been playing around with the puppeteer API and with Edge moving to Chromium so it will be supported also.

[dbrgn]

@corbinu looks interesting too. But my point is that wasm-bindgen already includes browser automation (it launches a headless browser instance and runs tests in it). So maybe that infrastructure could be reused for selenium style tests.

[corbinu]

@dbrgn My understanding is that it runs unit tests inside the browser these are very different from selenium and puppeteer which control the browser from the outside interacting with the page for E2E testing. The puppeteer API is specifically designed to replace webdriver/selenium

[alexcrichton]

I don't personally know too too much about the headless browser testing space, but I'm all for improving the integration with wasm! I'd personally defer to others more knowledgeable to help see how something like this would be designed/implemented

[dbrgn]

Firefox has been playing around with the puppeteer API

Are there any concrete steps / comittments in that direction? From the Puppeteer Docs it looks like it's only focussed on Chromium. It actually bundles Chromium.

[corbinu]

@dbrgn It looks like actually google was the one working on it. A Mozilla dev was the one who pointed me to the project a while back though: https://github.com/GoogleChrome/puppeteer/tree/master/experimental/puppeteer-firefox/

[chinedufn]

@dbrgn is there a use case that better highlights the value here?

Because from what I can gather from your example it seems like you'd have a function that was running in the browser but also communicating with the webdriver process (over websocket or something?) all in order to more simply dispatch an event. Which could be handled by some third party library that made dispatching events feel more ergonomic, but without all of the complexity.

For example, in your example above you'd need to somehow send some information about that button to the webdriver process so that it could come back and click it. But.. you'd also have to remember to append it into the DOM first.

Not saying that a use case doesn't exist! Just having trouble understanding the goal here and would love to better understand the use case(s)!

[dbrgn]

I'll try to explain my use case:

I work on a "compose area" project, a content editable div that allows to type text, to insert emoji, to autocomplete usernames, and things like that. https://github.com/threema-ch/compose-area

It has its own internal state datastructure: https://github.com/threema-ch/compose-area/blob/40f6ddbc4f7a026f1ef81fd6c3a4150628cc877c/src/state.rs#L5-L17 Every input event in the compose area triggers an event that is handled by the WASM backend code and modifies the state. The changes in the state are then diffed (using percy's vdom implementation) and written to the DOM.

I have some unit tests for the DOM independent logic. I also have some browser tests for things like "if I press the keys "a" and "b" followed by "backspace" then the text field should only contain "a".

Things get tricky for integration tests though. For example, I have to deal with the caret position (through the selection API) a lot. There is a difference between a page that does an initial load (no selection or caret position on the document) vs the state when a user clicks on the empty input field. I could try to approximate the clicking by manually creating a new selection on that DOM element, but then I only tested my expectation of what happens when the user clicks on the input field. I don't test what actually happens when the user clicks on that input field. This is something that can differ between different browsers, so I need a framework that will allow me to do cross-browser tests (at least for Firefox, Chromium and Safari). (Trust me, content editable elements are by far the most utterly broken things I have experienced in web browsers so far...)

Unfortunately DOM manipulation will not allow us to simulate things like clicks. There are also a few other user-browser interactions that can't be done through the DOM / JS. Therefore we need some kind of automation framework.

I set up integration tests for now using NodeJS / selenium-webdriver. The test does exactly what I need but I had to write the code in JS/TS and create a custom test runner which is not integrated with Cargo and the rest of the codebase. Note that I do not start any Java Selenium server, it seems that this is not required anymore and that automation is supported through the Webdriver protocol (in the case of Firefox implemented in Geckodriver).

And that's the reason why I wanted to ask about integration of integration tests (ha-ha) in wasm-bindgen, since the browser tests already use webdriver in some form (at least I'm assuming this because Geckodriver is a requirement). I'm not sure how and to what extend this integration already exists, and I'm not too familiar with how the webdriver automation actually works. But purely from a developer use case point of view it would be fantastic if we could do unit tests, dom tests and headless integration tests all in the same codebase.

[chinedufn]

Cool! Thanks for explaining!

Right now wasm-bindgen-test takes your test compiles it to Wasm and then runs it in a browser or Node.js and print the relevant info back to your terminal.

So the only way to then access a webdriver from that test would be to have wasm-bindgen-test run a server and effectively give your test something that could talk to that server in order to make that server use the webdriver protocol to talk to the page that your test was running in.

But... you're still in the browser with a single thread that you can't pause so you'd need to make sure that in the meantime you weren't running any more code in your test until theserver responded back to you "Hey, I'm done clicking! You can continue with your test now!"

And since there's no await syntax you'd have to have a new callback every time you did anything with your web driver. Which would be no fun syntactically IMO.

---

That sounds like a lot of complexity vs. just using rust-headless-chrome.

So I don't think that you'd want to combine #[wasm_bindgen_test] with any webdriver test stuff.

But ... I do have an idea that builds on top of your cool idea ...

---

Could it be useful to be able to do something like this..?

// Each setup has a corresponding test
#[wasm_bindgen_integration_setup]
fn my_setup () {
  let content_editable = create_cool_component();
  body.append_child(&content_editable);
}

// Each test has a corresponding setup
#[wasm_bindgen_integration_test]
fn my_test(webdriver: Tab) -> Result<(), failure:Error> {
  webdriver.wait_for_element("#your-div")?.click()?;
  assert_eq!(2, 2);
}

wasm-bindgen-test would start a server that served the wasm for each of your tests... Something like rust-headless-chrome would be used to open a headless browser. One thread per test. Each thread opens a new browser tab and hits the server at the endpoint that is serving your test.

After that it passes the tab to your wasm_bindgen_integration_test so that you can run your test.

The value proposition here would be that it becomes dead simple to browser UI test the smallest or largest pieces of your application by simply annotating functions, vs. today just about everyone's only feasible option with browser UI testing (not just in Rust but in all languages) is serving up their entire app and then testing the pieces that they care about...

Just make a new setup and integration_test pair of functions and you're off to the races.

This works nicely for an all Rust application ... but if you have a bunch of JS that needs to be in the picture that's another story.. Maybe a v2 of something like this would be being able to tell wasm-bindgen-test to inject some JS into the page or something..........

---

I of course haven't thought through all of the logistics.. corner cases.. or finer details to know if this is even feasible / truly useful.. just brainstorming to see what y'all think!

Your thoughts @dbrgn @corbinu ?

[dbrgn]

This works nicely for an all Rust application ... but if you have a bunch of JS that needs to be in the picture that's another story.. Maybe a v2 of something like this would be being able to tell wasm-bindgen-test to inject some JS into the page or something..........

That's already supported by WebDriver / Selenium. And WebDriver is a W3C recommendation supported across multiple browsers while rust-headless-chrome / puppeteer currently only works with Chrome and is primarily developed by Google. (I'm sure Puppeteer has a lot of exciting use cases, but cross-browser testing isn't one that's currently one of them.)

Your idea with the setup and test sounds interesting, although tabs wouldn't work since it makes a difference whether a tab is focussed or not.

I think since there seem to be a lot of different ideas on how testing should work, I think it would be best to work with separate crates for now and to explore the integration testing design space separately. It seems that there are already a few crates related to webdriver: https://crates.io/search?q=webdriver Maybe a cool solution could be built on top of them.

And maybe the webdriver remote instrumentation functionality can be integrated into wasm-bindgen at a later point in time.

[chinedufn]

Makes sense!!

[jquesada2016]

My personal two cents on this is that although Selenium is the most popular approach, it is far from being the best. Especially since this space is new for Rust, supporting newer and more flexible technologies, is more important. There are very good projects, such as Cypress and many others which are cross browser and far more intuitive, in my opinion, to Selenium.