General Guidelines for the 2019 Community Survey

[mirandapedro]

Here are the guidelines that I believe should orient the design and conduction of the 2019 WCA Community Survey.

Here's a book I found that could be useful. It's called "Designing and Conducting Survey Reseach: A comprehensive search", by authors Louis M. Rea and Richard A. Parker. It thoroughly addresses the whole process of surveying, from designing questions to processing data.

Here's a website that succinctly tackles the same topic.

And here's the Google Doc where the writing and editing of the first draft of the survey will take place. Anyone with a @worldcubeassociation.org email can edit it (contact me if you can't). Anyone with a link to the document can make comments and suggestions. Please feel welcome to suggest any new question you might find relevant, as well as any improvments to existing ones.

General Guidelines

Here are the guidelines we already have on the first draft Lucas presented:

Each survey will be tied to a WCA website account. < We can do this using custom variables with SurveyMonkey using their paid plan. This is $32 for one month (or $23 if we demonstrate non-profit status), which I (Lucas) am happy to contribute personally. < This means responses from competitors with WCA IDs can be associated with their competition results. This allows us to break down results by region, or by how recently someone competed. Results will be available to any WCA staff member who asks the WCT. < This survey is designed so that we don't collect any sensitive information. We'll probably want to run bucketed or anonymized stats based on competitor ages, but this should be restricted to a very limited number of software team members. Every answer except the first one is optional.

Right from the get-go, the goals of conducting the survey should be clear. I believe these are the main goals of this process:

• Measuring the community's opinion on the competition experience and how it can be improved;

• Understanding the nature of its interaction with the WCA as a whole and gathering the data that allows us to improve the quality of it.

• Asking how it feels about a few particular issues;

With these goals in mind, a few things should be clear:

• The survey is not a referendum, particularly when it comes to questions relating to the Regulations. The results we get are not binding and serve only to subsidize WRC's and the WCA's activities in general.

• The data gathered that is not going to be publicly released and is for WCA Staff use only (Although I think it's worth it to put together a public report with some of the less sensitive data).

• Designing and conducting the survey is only half the process. We'll need to carefully process the data gathered after surveying is done.

With these things in mind, we can start talking specifics.

Web-based Surveys

There are some pros and cons of conducting web-based surveys that I think we should keep in mind while conducting this process (taken directly from pages 12 and 13 of the book I linked):

Advantages:

• Convenience: This technique represents a convenient and efficient way of reaching potential respondents. They are able to receive the questionnaire and complete it in the privacy of their home or office. This advantage is becoming particularly significant as the availability of computers becomes increasingly widespread.
• Rapid data collection: Information, especially information that must be timely (e.g., a political public opinion poll related to an upcoming election), can be collected and processed within days.
• Cost-effectiveness: This technique is more cost-effective than the traditional mail-out survey because there is no need for postage or paper supplies. It is also more cost-effective than the telephone and in-person surveys because it is not at all labor intensive.
• Ample time: The respondent is not pressed for time in responding to the web-based survey and has the opportunity to consult records in answering the questions. There is time to consider response choices and respond to open-ended questions in the form of text.
• Ease of follow-up: Potential respondents can be reminded to respond to the survey through follow-up e-mail messages.
• Confidentiality and security: Personal or sensitive information supplied by the respondents can be protected on a secure server through the efforts of the research team.
• Specialized populations: The survey is particularly useful in reaching specialized or well- identified populations whose e-mail addresses are readily available. For example, we have successfully used this technique to conduct surveys of satisfaction among employees and stakeholders of large public organizations.
• Complexity and visual aids: As with mail-out surveys, web-based surveys can use visual images and more complex questions.

Disadvantages

• Limited respondent bases: A major disadvantage of this technique is that it is limited to populations that have access to e-mail and a computer. Furthermore, the technique assumes a certain minimal level of computer literacy that is necessary for the completion and submission of the questionnaire. Such literacy is improving rapidly within the general population.
• Self-selection: As in the traditional mail-out, there is a self-selection bias that leads to lower response rates. Those who do not use e-mail or are not comfortable with web-based technology exclude themselves from the sample. Also, individuals with reading or language issues tend not to respond to web-based surveys. Some researchers send the survey by e-mail in multiple languages in an effort to obviate this problem.
• Lack of interviewer involvement: Since there is no interviewer involvement in the web-based survey, unclear questions cannot be explained, and respondents may not follow instructions. These problems can seriously compromise the scientific reliability of the survey even though telephone contacts are provided to the respondents in the event that they need help.

I believe that if we are mindful of these characteristics when designing, conducting and interpreting the survey, we can reach a better, more useful result.

Deeper discussion

Here are the four axes of work on the logistical side of the survey that should be discussed on an issue of their own:

• Guidelines for the Questions: standardizing the phrasing of questions so they are as clear and concise as possible (e.g. the balance between open and close-ended questions, the length of the survey, etc.). The book I linked dives deeply into this topic and should be very useful!

• Community outreach strategy: what are we going to do to get as many people to participate in the survey as possible (e.g. translation; public campaign in conjunction with the WCT; possible email notification of all registered speedcubers about the email; contact with local leaders and regional organizations, etc)

• Technical aspect of the Survey: issues relating to setting up the technical aspect of surveying (e.g. linking survey monkey with WCA IDs; beta testing the survey; setting up translations on the survey monkey website; etc)

• Survey Report: processing and analyzing the data gathered and compiling into a document (the document should also contain a short introduction of the original goal of the survey, the methods using for collecting the data and the questions asked).

If we could, I think having a WRC member leading each of these topics would get things to run smoothly.

Schedule

Moved to #9

[c-goodyear]

possible email notification of all registered speedcubers about the email

Aside from this being circular logic, this is not something that any registered speedcubers have accepted to be involved with during an account creation.

As for translations, please be aware that while it will increase the amount of people who can respond adequately, answers do not mean the same thing in different languages. The most obvious example of this would be asking someone if they are a new or an experienced competitior, such classifications can have largely different implications in two languages, so you can no longer classify the entirety of the survey results with those categories.

[Laura-O]

I have commented on some issues in the Google Doc, but as as general rule of thumb I would recommend you to check if all response options are actually valid answers to the questions. There are several question-answer combinations which sound a bit odd and which might make it more difficult to understand the answers.

For example, if you ask "Do you have concerns...?", all answers should start with "Yes" or "No" (or maybe also "I am not sure" or "I don't know").

[mirandapedro]

That's very good advice.

We'll definitly make sure to improving the wording of the questions and the responses before we send out a first draft for pretesting.