search

expfactory / expfactory-games

MIT License
3 stars 4 forks source link

robot testing of games #6

Open vsoch opened 8 years ago

vsoch commented 8 years ago

@boomsbloom this isn't priority, but when we have some games established let's talk about if there is a way to automatically test them. I built an experiment robot to go through the jspsych experiments, and this is pretty easy to do because you can extract the acceptable responses (keyboard, click, etc) from the jspsych object. Success for the test means making it to the end of the experiment without any javascript errors, 404, etc. Games are likely different, but there might be a way.

vsoch commented 8 years ago

To start, I'll just validate the config.json files for the continuous integration.

cbattista commented 8 years ago

this would definitely be a great feature - would be really excellent for our screener as well and i'd love to hear more about your robot

On Fri, Apr 15, 2016 at 10:59 AM, Vanessa Sochat notifications@github.com wrote:

To start, I'll just validate the config.json files for the continuous integration.

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/expfactory/expfactory-games/issues/6#issuecomment-210565765

vsoch commented 8 years ago

If you cd into an experiment folder, you can just type

  expfactory --test

and the robot will run through it, and you can watch. It's the same robot that runs on circle, for example, if you poke around circle:

https://circleci.com/gh/expfactory/expfactory-experiments

The continuous integration typically has a few tests. First to validate the config.json files, again can be done on command line in a folder with:

  expfactory --validate

then to check that the circle.yml includes all experiments/surveys etc that should be tested, and then it clones the current master, compares to the PR being done, and tests the experiments that were changed using selenium. It's pretty cool, definitely useful to find errors you don't anticipate.

jonathanicholas commented 8 years ago

This should be a fairly straightforward port from what you have written for jspsych since I think the only difference will be taking a click from each sprite rather than from the jspsych object, right?

vsoch commented 8 years ago

Right - if you just tell me the object in the console to look at to get the current state (and calculate what to click / do next) I can probably get a robot up and running for it!