Methodology

[shhaine]

@atopal mentioned in the last meeting that we've got no frame to use for sampling means we cannot do probability sampling.

As a solution, we can redefine our population by limiting it to some group in which we can get a complete list of members so we can do random sampling and generalize findings to that group. This takes more time and effort given the situation. I'm also guessing it could be hard to get that list and to initially redefine the population given the survey goals.

Suggestions

Non-probability sampling techniques such as what was applied in the past years can be used. Results from this, however, should be interpreted with caution. Data from non-probability sampling techniques cannot be used to generalize results to the entire population. The good news is, we can still do descriptive analysis and doing this will still be aligned to the survey's goal and data usage as indicated here (particularly the italicized ones as shown below).

What is the survey's goal? The survey's goal is to track the evolution of upcoming features and libraries, and help developers decide which new technologies to focus on.

How will this data be used? All data collected will be released openly for any developer or company to consult. Browser vendors also use this data to prioritize focus areas and inform their roadmaps.

The question now is: How can we can collect the data that we need to allow us to perform the analysis method that is assumed to address the survey goals, given the target launch date?

We can do the following:

1. get as many target respondents as possible by making sure we make the survey available to them, esp. to the subgroups we are interested about
   • anybody who writes CSS, whether regularly or occasionally, as part of their job, as a student, or just for fun as indicated here
   • respondents who have hopefully, 100% completion rate so that we can
2. obtain data that would allow us to describe the patterns in the results in context (data on demographics will be valuable on this). This means, we get results that are not generalizable to the population of people who write CSS but we are capable of describing the diversity of choices within the group of respondents (descriptive analysis)
   • through clustering or
   • simple correlations in plots made possible by demographic data

To add, given the past results we need to make sure we make the questionnaire available to communities of female CSS coders, and other subgroups of interest from which we got low number of respondents. @SachaG, @atopal let me know your thoughts!

Originally posted by @shhaine in https://github.com/Devographics/surveys/issues/70#issuecomment-1579074566

[SachaG]

I think that's a great overview of the issues we are currently dealing with. I think the next steps would probably be collecting practical suggestions of what we can do for this upcoming survey round?

[shhaine]

@SachaG, an important aim is for us to increase the completion rate to 100%. As mentioned, this will be helpful in the analysis and reports. Below is a result from State of CSS 2022

Image above shows that 25% of the respondents have completion rate less than or equal to 50%. This means that we used 3268 less than half-accomplished response data out of 12986 for our reports last year. Either these response data were left unfinished for some reason unknown to us or it is just that many items were intentionally left blank since we allowed everything to be optional.

A survey questionnaire which is barely accomplished does not tell us much. An abstain means we lose a data point from a respondent. Hence, the more abstains that we have from multiple respondents in many items, the less we will be able to describe profiles and address the survey's goals. In fact, we should have a threshold of completion by which we should say that this response data is qualified to be analyzed.

Suggestion 1

We ensure that we are asking the right questions to the right respondents. That is, each question that the respondent will be asked should be applicable to him or her. We take away from our respondents the need to determine if a question applies to them or not. We can attain this using filter questions. Whenever applicable, ask yes or no question first and filter the next questions to be asked based on that initial response.

Here is an example. In the below image, if I am not experiencing any disability, then I have to figure out that this question does not apply to me so I have to skip it. This lowers my completion rate.

As an improvement, we can first ask if the respondent is experiencing any disability. This question will then allow me to say No. I didn't have to think if this applies to me. I did not have to skip it. It did not have to lower my completion rate. If on the other hand the answer is yes, then that's the only time we give the question in the image. We can then make it required.

@Gio-Q @michaelquiapos

[shhaine]

There are also other candidates for filter questions. The list below are examples of 'leading questions'. They all assume that the respondents read blogs/magazines/etc., consult sites/courses/etc., listen to programming-related podcasts, want to highlight people they read, follow, or just want to highlight and answer other surveys.

1. Which blogs/magazines/etc. do you read?
2. Which sites/courses/etc. do you consult?
3. Which programming-related podcasts do you listen to?
4. People you read, follow, or just want to highlight.
5. Which other developer surveys do you participate in?

If I do not do any of them, then I have to decide all of them are not applicable to me so I have to skip them all. Again, this lowers my completion rate.

Another solution would be to add a 'None of the above' option. However, due to the many options that we are presenting in all of these questions, it might be overwhelming for none-doers of the action to be presented with these. Best option for me is to use filter questions.

[SachaG]

Great feedback!

In fact, we should have a threshold of completion by which we should say that this response data is qualified to be analyzed.

Up to now I've considered that even if someone only answers a single question of the entire survey, we can still use their data for that one question. Are you suggesting that we establish a threshold, and that any response that doesn't reach that threshold would not be included in the overall dataset? If so what should that threshold be in terms of completion percentage?

Best option for me is to use filter questions.

My issue with filter questions is that they add one more click for people who do want to answer the question (click "yes" first, then answer question vs just answer question currently).

I'm also worried that people might click "no" because they think it doesn't apply to them, but if they actually see the question options they'd find some that do apply.

So I think the "none of the above" option seems like the most practical. We could make it stand out visually so that people who want to skip a question can easily locate and click it?

[shhaine]

I am also curious about this question:

It is possible for me to check the ff: Student, Hospitality. Do we expect this kind of result? Or should this question only be asked to respondents who are already 'working' (as the question assumes)? If we only wish to look at which industries CSS is being used among the working respondents, we can turn this into a question that only gets asked if a respondent is already working. Being clear about our intention in this question will make analysis later lighter :)

The same is true for the following questions. A suggestion would be to ask whichever applies. Student / Working. If the respondent is working, we present all the questions below. If the respondent only checked student, he or she no longer needs to see those questions. Respondent could also be allowed to choose both. This, again, lessens the need for respondents to judge if these questions are applicable to him or her.

[shhaine]

Up to now I've considered that even if someone only answers a single question of the entire survey, we can still use their data for that one question. Are you suggesting that we establish a threshold, and that any response that doesn't reach that threshold would not be included in the overall dataset? If so what should that threshold be in terms of completion percentage?

I was leaning more on making the survey experience something that will make the respondents finish everything so that we won't have to deal with incomplete response.

However, in line with the threshold, we can rely on time spent to finish the survey data. Suppose that we expect a respondent to finish answering the survey for about ten minutes and some respondent X submitted the answered questionnaire after 10 seconds. We will expect that respondent X only answered very few questions. Respondent Y who completed 100% of the questions within a minute should also be disqualified. I read that this is how they do it in StackOverflow. That should apply to us, too.

This requires us to add a feature that records how long it took the respondents to complete the survey.

[shhaine]

My issue with filter questions is that they add one more click for people who do want to answer the question (click "yes" first, then answer question vs just answer question currently).

A possible alternative to that is what @Gio-Q mentioned about adding programming logic to make it easier for the respondents. That is, if say I choose I do not have a disability. Then the other choices should not be available for checking anymore. The same is true if I select a particular disability - the I do not have a disability option should not be available for me to check, too. Not sure how this could be applied on the Student / Working user flow case. @michaelquiapos might have an idea.

I'm also worried that people might click "no" because they think it doesn't apply to them, but if they actually see the question options they'd find some that do apply.

Hmm, I guess it depends on the clarity of the questions.

So I think the "none of the above" option seems like the most practical. We could make it stand out visually so that people who want to skip a question can easily locate and click it?

None of the above works but making it stand out visually might make it seem like we are leading our respondents to select that option, which is again, a no-no.

[shhaine]

Suggestion 2

We want to avoid double-barreled questions. Suppose that I were a respondent, I want to identify 5 items in which I have difficulties using because of lack of support AND another 5 items because of differences between browsers. How should I respond to the question below?

Browser Incompatibilities Are there any existing CSS features that you have difficulties using (or avoid altogether) because of lack of support, or differences between browsers? Pick up to 5 items.

This question is double-barreled since it presents two different ideas: 1. lack of support 2. difference between browsers. This is confusing among respondents. Some could be leaning more on lack of support, others on difference between browsers, and then the remaining could be combining the two. (The main heading though makes it feel like this is more of a browser compatibility than lack of support issue.)

If these two are not meant to pertain to the same problem, we can separate into two questions. One that focuses on lack of support and another on difference between browsers. If these two are the same, then we eliminate the other and only retain one to avoid confusion

Analyzing this later on will make us rethink - so what are we talking again about here? browser compatibility or lack of support?

@foolip, I have to tag you here since you showed this question during the meeting. Also adding here that it is okay to indicate common responses based from the previous recent result and still have the 'Others:' accepting free-form answers. This is because it is naturally difficult to answer open-ended questions and adding common responses will entail less effort from the respondents.

---

The same is true for this question (which we suggested to be considered under filter question suggestion)

How long have you been working or studying in this field, in years?

[shhaine]

Suggestion 3

We can avoid confusing options by not listing down overlapping ones that will lead to difficulty in the analysis stage. Below are some examples.

Not Listed and Others: overlap. The only difference is that, Others: allows users to specify an unlisted option. We can eliminate 'Not Listed' to make this easier to understand. This has an impact in the user experience.

Suppose that I am Hispanic, East Asian, and White, then I can check all of these OR simply choose multiracial. They overlap.

Suppose I am East Asian but I also checked Not Listed. Does that mean that I have another race that is not included in the list? We an simply remove Not Listed if the options are exhaustive or use filter question/ program logic as well to restrict options that can be selected together.

[SachaG]

I was leaning more on making the survey experience something that will make the respondents finish everything so that we won't have to deal with incomplete response.

I don't really think that's an option, as I'm afraid with such long surveys making every question required or discarding any non-complete response would drastically lower the response rate.

However, in line with the threshold, we can rely on time spent to finish the survey data. Suppose that we expect a respondent to finish answering the survey for about ten minutes and some respondent X submitted the answered questionnaire after 10 seconds. We will expect that respondent X only answered very few questions. Respondent Y who completed 100% of the questions within a minute should also be disqualified. I read that this is how they do it in StackOverflow. That should apply to us, too.

This is a good idea, but I don't know if we actually have people "cheating" by completing the whole survey in a suspiciously short time. It's something I'll have to look into.

This question is double-barreled since it presents two different ideas: 1. lack of support 2. difference between browsers.

I think in this specific case it's basically the same thing? Or at least two very close ideas. So combining them in a single question doesn't seem too weird to me.

How long have you been working or studying in this field, in years?

This is a clearer example of a badly worded question though, I agree.

Not Listed and Others: overlap. The only difference is that, Others: allows users to specify an unlisted option. We can eliminate 'Not Listed' to make this easier to understand. This has an impact in the user experience.

Good point! "Not Listed" is a carry-over from before we had the "other…" option, we can remove it now.

[SachaG]

Regarding the questions about company size, work experience, etc. in the JS survey we had this question that we can use to help identify if someone is a student or not. I'll add it to the CSS survey as well, this way we can at least use this data to know if e.g. the experience question applies to work experience or student experience.

[SachaG]

OK, I've added more "none" options throughout the survey :)

[shhaine]

OK, I've added more "none" options throughout the survey :)

will check on the items too.