vaadin-bot
commented
15 years ago Originally by @hannuvaadin
What are we exactly testing here? How approachable Vaadin is? How easy it is to use Vaadin? How developers actually use Vaadin? Are we gathering quantitative or qualitative data? I presume qualitative.
First, I propose we extend the test's scope to include downloading and installing Vaadin. This way we get a glimpse of real UX and might catch any issues related to it - that is, installing and setting up the IDE.
Secondly, I'm afraid we can't do a five hour session - it'll get tedious after a few hours. We need to plan the test so that it'll take no more than 2-3 hours. If there are major issues, those will show their ugly heads in shorter time as well.
I'm thinking we could design (and verify the design by coding) a working Vaadin application. Then we'll hand over the blueprints to the developer, and ask him to implement it. UML, a wireframe or a screen shot will probably do. This way the developer can get to the essential part - writing code - sooner, and he'll probably use less time fixing design mistakes. A confident developer is also more relaxed, and we need him as relaxed and at home as possible in a situation where someone's lurking over his shoulder, asking "stupid" questions on purpose..
And while we're talking about relaxation: This test would probably work best if there were as few people present as possible: only the developer and the researcher. The less the researcher knows (or pretends to know) about web app development with Vaadin, the better. We need to build a friendly conversation, a classic master-apprentice setting for the test to get as much output from the developer as possible while he's doing his thing. Therefore, I don't think it's a good idea to bring a member of the dev. team into the test situation. His presence will probably affect the test developer negatively (increases stress, need to prove skills, fear of judgement from experts).
I understand the dev. team needs to see the raw output from the test to make their expert conclusions, so it might be useful to capture the test somehow. Maybe we should invest in an HD camcorder and record the whole test with it? (This is actually a long time plan for me anyhow; will give you an elevator pitch on this later on).
So, to sum things up: Yeah, sure, let's do it. But let's plan it properly in advance so we get the most out of it.
Originally by @jojule
Test process should be:
Please give comments on process
Imported from https://dev.vaadin.com/ issue #3036