jonathanolson
commented
9 years ago It sounds great to put the sims into an iframe with a fuzzer. We'd presumably want to run with a try-catch around events (Scenery integration?) and the animation frame for this, but we don't want that behavior for normal sim runs. This would help a TON for Scenery development. ?eall might be a better option, unless that slows things down too much.
Additionally if possible, I'd love a tool like Selenium for this. See one of the website tests:
public void testSignIn() throws Exception {
// checks validation
String signInFailed = SeleniumUtils.getString( "signIn.validation.failed" );
String signInTitle = SeleniumUtils.getString( "signIn.title" );
selenium.open( "/" );
selenium.click( LogInOutPanel.SIGN_IN_ID );
loadWithoutError();
selenium.type( "username", "bogus-email" );
selenium.type( "password", "bogus-password" );
selenium.click( "submit" );
loadWithoutError();
assert ( selenium.getBodyText().contains( signInFailed ) );
assertEquals( selenium.getTitle(), signInTitle );
selenium.type( "username", "test@phet.colorado.edu" );
selenium.click( "submit" );
loadWithoutError();
assert ( selenium.getBodyText().contains( signInFailed ) );
assertEquals( selenium.getTitle(), signInTitle );
selenium.type( "password", "test-password" );
selenium.click( "submit" );
loadWithoutError();
assert ( !selenium.getBodyText().contains( signInFailed ) );
assertNotEquals( selenium.getTitle(), signInTitle );
}I'd really love to be able to do a functional test on a sim, record the input events, model changes, and screengrabs. Then the input events could be played back at a later time, verifying the model changes and optionally screenshot comparisons (style changes that affect all sims would throw a wrench into that).
samreid
pixelzoom
aaronsamuel137
I'd like our team to have an automated test harness that launches each simulation, runs a mouse/touch fuzzer (or some other way of exercising some of the sim functionality) on it and summarizes the results. It is getting very difficult to know whether common library changes will cause breaking changes in usage sites, and it is time consuming to manually test. Even knowing if the simulation launches without crashing would, in many cases, be very helpful.
Perhaps this can be done through the iframe API? A parent frame that iterates through simulations: for each one, it launches the sim in an iframe and listens for messages that indicate the progress--if messages stall out, then it means the sim encountered a bug. The sims should be run with ?ea
This should be designed so it could be run on any developer machine or a continuous integration machine (the latter may require PhantomJS?).
A way to make this even sweeter would be to add temporary instrumentation into the changed library code so that the parent frame can collect statistics about how many times the modified code was actually hit in each simulation so we can get assurance that it has been tested. Note: the tests described here are mainly to make sure things don't error out and crash--more manual tests would be required to confirm the behavior is correct.
Not sure what repo such a thing should live in (if it should live at all).