JamesPHoughton
commented
2 years ago Lab group meeting Sept 23 feedback:
Should we use the exact wording?
- depends on if its a survey that another researcher has already validated
- Some things (such as "if I made a mistake") are not particularly clear in the context of deliberation
- Others are not things you would actually do in the context of deliberation ("I can ask for help")
- There's a limit to how much surveys can be general anyways, ie, non-group experiments can't use surveys about groups. But it might make sense to write it so that any group experiment could use it. Maybe put in qualifications about when you can use a particular survey? When its the survey valid to use?
- team performance survey could be reworded to think about "groups" instead of "teams" (does the wording matter very much?) - don't lose much when using group instead of team. Team has to be a group, but a group doesn't have to be a team.
- Either make the questions more specific to deliberation, or vagueify them so they are inclusive of all group processes. Thinking more about internal consistency with themselves and reusability (which is possible) rather than adherence to previous studies (which are all over the place).
- If there was a study that we were trying to replicate, there is an argument that we should use the same measure.
- But as none of these surveys are specifically designed for this context anyways, it doesn't probably matter.
- Also internal consistency improves the panel data, and makes it easier for us to reuse across our studies
- Todo: do we want a standard for questions that we use for the panel going forward?
Should we use different numbers of points in the scale?
- Make sure the scales are aligned on direction, scale them to 0-1. This may be something we don't want, so keep the raw data.
- If we have the scoring function, we can have mixed scale lengths and still compare across scales in our survey set.
- There are times when its advantageous to not have an odd number, to force someone to make a choice of some sort. Mark says there is some evidence that people do give more honest answer when they can't pick a middle value, but there is also evidence that you lose resolution when you have fewer points.
- Doesn't seem like there's a compelling reason to make one choice over the other.
- The literature is probably not consistent either, and we just picked a scale from the literature - that's an argument for internal consistency, why try and be consistent with something that isn't consistent with itself.
Title in the survey?
- Probably don't want the title in the psychological survey question, at least, and as a general rule unless otherwise helpful.
- Dan Goldstein is a good person to talk to about these things.
"Do you have any other comments"
- onerous if people feel that they need to do it
- one strategy is to explicitly label them as "Optional"
- not having it on every page minimizes turker work, and so probably improves the data quality
- the original surveys don't have an open response item
- just have one open response item at the end of a series of these surveys so that people can say what they need to say
markwhiting
The following surveys are implemented in our surveys repo already:
Our goal is that:
Gut check
These aren't the final surveys we're going to use in the deliberation experiment, so don't want to assess them yet for content, but I do want to check the presentation before we spend a lot of time implementing more surveys in the same format. Some questions that come to mind: