Idea: Allow respondent to priority rank questions such as Library Evaluation

[djaddison]

Hi, writing surveys are tough and often thankless work. So, firstly, thanks for putting together such an amazing survey! The JS and CSS surveys are interesting and important.

Since I'm an unknown to many of you and not involved in your respective communities, hopefully you'll go easy on me if my suggestion is completely out of place. I read through the other open issues, but didn't see anything that captured the following idea.

A stacked list would likely allow for a more accurate expression of the respondent's priorities. The current UI introduces bias by presenting a tournament bracket style UI element. For example, how would a respondent indicate that their top priorities are user experience, documentation, inclusivity?

Tournament brackets also has the implication of a victor, head-to-head competition, and confrontation. In a passionate space such as front-end, this implication moves the narrative further away from inclusivity.

Anyways, nice to meet y'all, Kind regards

[SachaG]

By a stacked list do you mean some kind of drag-and-drop widget where you can rank items from 1 to 8? The reason I didn't go with that is that I felt it would stop people in their tracks as they spend time deciding if e.g. item A should be ranked 4th or 5th place. The tournament bracket felt more fun and quicker to get through (just 7 clicks), although of course we'd have to do actual testing to be sure.

[djaddison]

100% agree. This would have to be tested

A stacked list was what I was initially thinking, but you bring up a good point. I would speculate that at some point in the ranking a respondent would be less opinionated about a topic (option) and this lack of opinion could stop people from moving forward. Or, at least provide unnecessary friction.

What if the questions were framed as "top 3 priorities"? It could provide limits on the question to surface the highest priorities, limit to 3 clicks or drags, allow respondents the full range of options, and reduce the risk of friction by avoiding lower priority rankings.

The interactivity would have to be worked out, but a quick mock to highlight the idea:

[LeaVerou]

YES! I remember having the same issue. With the tournament-style question, at some point early on in the tree I was asked to choose between my single 2 most wanted CSS features. I didn't like that, I felt a lot of the data was lost by picking one. Especially since I cared much less about literally all other branches in the tree.

I really like the idea of choosing the top 3. Do note that this is also gentler on people's time, as with the tournament-style question they have to go over 2n options (n + n/2 + n/4 + ...), whereas with this one they only go through n options max (possibly even less if they are sure about their top options before they reach the end)

[SachaG]

We could easily add a "shuffle" option to the tournament bracket that gives people a better starting line-up. This would probably give us more accurate results since it would eliminate the situations where people have to discard one of their important concerns right off the bat (although in aggregate I think it all works out anyway).

The "pick the top 3" idea is interesting too, and we could just use checkboxes (but limited to 3), no need to develop a new drag-and-drop component I think. But to me it feels less fun than the bracket, I think it's good to have something that makes us stand out from other surveys a bit…

• [ ] Add Shuffle button

or, if not

• [ ] Consider "Top 3" ranking instead

[LeaVerou]

Checkboxes don't do ordering though and the ordering is important IMO. No need for a drag & drop component, you can just click on items to add them to the "top 3" in order, and have controls to delete from there. It would be nice to be able to drag & drop to reorder the 3 choices, but not a dealbreaker, you could do it with buttons since it's only 3 items.

I honestly didn't find the tournament style question fun at all. I found its UX highly annoying, and as I explained above, it forces the user to go over twice as many choices. Not sure I'm in the minority on this, but I suspect there is a sizeable percentage of users that don't find this fun at all. In general, it's not usually fun to ask users a question but prevent them from being able to select the answer they want until they have jumped through hoops (log(N) hoops in this case!).

[LeaVerou]

FYI I asked on Twitter for some outside perspectives: https://twitter.com/LeaVerou/status/1563114943343656962

[djaddison]

I agree fun factor and having something stand out is an important aspect of any experience like this.

The tournament style widget is interesting and does stand out. Out of curiosity, are only the top two priorities (ranks) used in the results? Are the 3rd to 8th ranking used? If so, how is that calculated? The reason I ask is that in a single elimination style tournament bracket, the ranking becomes ambiguous around the 3rd place ranking.

https://strongvon.com/help/Brackets-2.htm

However, I would also speculate that most people (respondents or consumers) would prioritize accuracy higher than fun.

From my personal experience, without the additional visual cues of which priority was ranked 1st, 2nd, 3rd, and on, I find the tournament bracket requires too much thought. I find myself having to trace the graph backwards once I've made all my selection while trying to reason about what is actually 1st, 2nd, 3d... This is in addition to the issue I mentioned above.

The shuffle option sounds interesting. It solves one aspect of bias introduced by the starting line up, but adds to the complexity and doesn't resolve the bias introduced by the head-to-head selection. Assuming the shuffle button is random, it could be imagined that a respondent would have to click the shuffle button many times to give them a starting line up that allows them to pick their desired priorities. All while having to mentally evaluate the graph to see if the shuffled values result in their desired selection for each click of the shuffle button.

The "top 3" suggestion is one idea, but maybe stepping back and opening up the question: "What would an ideal widget be?"

• Accurate and aims to reduce bias
• Allows respondents to accurately reflect priorities
• Has a fun factor and something that stands out when compared to other surveys
• At a quick glance, respondents should be able to figure out what is the order of priorities
• Should prioritize the time it takes for respondents to answer
  • Less time is better
  • Has a low number of clicks to provide ranking
  • Low friction and doesn't stop people in their tracks

[djaddison]

I had a quick skim of the twitter replies. There are some interesting perspectives there worth reading. Aside from the sentiment for or against a widget, these comments were interesting:

• "yes they are only good for finding an absolute winner" -- @magnemg. << I keep coming to this conclusion the more I think about tournament ranking given the ambiguous nature of rank 3+. If the top two ranks are the only reliable data out of this style of widget, then could this be one or two clicks in a list? Click one is rank 1, click two is rank 2, cycle the clicks?
• "tournament style is fun, but ranking is more practical (need to be able to rank all, not just top 3). Instant runoff style selection is possible with ranking all." -- @TigerC10 << I wonder if there is an existing widget for instant runoff style that would be applicable

[SachaG]

A quick note that what is counted for the final results is the number of matchups won for any item (the idea being that items that make it to the top of the bracket more often accumulate more wins in the process). So yes in a single bracket there would be ambiguity between ranks 3-8, but when you aggregate thousands of brackets there is no such ambiguity as far as I know.

From skimming the Twitter replies it seems like what people really don't like is being forced to choose between two equally important items. But what matters is the aggregate of all brackets, not your own individual bracket. In other words if both A and B seem very important to you, you can pick either one with a coin flip and rest assured that they'll both rank high in aggregate because they'll tend to win their other matchups when faced with a different "opponent".

But of course, I realize most people won't see it like that and will only focus on the inconvenience of having to make a bad choice.