Increasing amount and quality of participation in code review / conversations

[kytrinyx]

For the engines of Exercism to really work, people need to see comments and code based on their solutions.

There are two overarching ways to tackle this, and we should investigate/pursue both.

The first is machine learning / AI to use all of the existing data (both code and comments) to show people something useful. We have an open issue about this here: https://github.com/exercism/discussions/issues/65

The second thing is to get people to contribute to the conversations more, and to do so in ways that are more valuable.

People might resist giving feedback for the a number of reasons:

• They don’t realize the benefit. You learn more from giving feedback than getting it. It forces you to:
  • Read code. Like actually, really read it.
  • Recognize code smells.
  • Articulate your thoughts and opinions about readability, simplicity, performance, design, idioms, conventions. This pushes you to a much deeper understanding of concepts, and uncovers those areas where you don’t actually understand something (which can lead you to go learn about it).
• They are intimidated.
  • They might be wrong. People might be smarter than them. They might get laughed at or ridiculed. People might say mean things.
  • For code newbies especially, they are like “who am I to tell someone else what to do?” which is a totally valid objection, but which is kind of irrelevant. A code newbie could look at some code and ask questions: How does this work? Why did you do it this way? I don’t understand this bit here.
• They simply don’t know what to say. They look at code and just see… code.

Other relevant discussions:

• https://github.com/exercism/exercism.io/issues/1956
• https://github.com/exercism/exercism.io/issues/1927
• https://github.com/exercism/exercism.io/issues/1957
• https://github.com/exercism/exercism.io/issues/2016
• https://github.com/exercism/exercism.io/issues/2711

[GustavoCaso]

@kytrinyx Totally agree with you on everything.

Maybe we could include some basic guides on how to comment code. A rating system for comments, so the user could see how his or her code reviews skills improve by the more he or she review code.

What about after you submit a exercise, you are assign a exercise to review, from another user but same exercise of course that way you have clear idea of what the exercises was about. This will help to get more comments as the same time, you will create more comments.

[GustavoCaso]

Anyone?

[dvberkel]

I don't really like the assigned exercise to review idea. For me it feels like an obligation that I am maybe not willing to commit to. I would rather want to opt-in to such a mechanism and therefore have a choice.

Having said that, I do acknowledge the value of feedback. Both in receiving good feedback and providing good feedback. But the problems and resistance people feel against receiving and giving feedback is as much a cultural thing.

I would love for Exercism.io to instigate a change, but I am not sure if "forcing" people to review others is the best way to start

[nilbus]

There's a nice middle-ground between forcing and what we do now (which is to say, “See related solutions and get involved here: [url]” (source)).

Instead, let's spoon-feed them the the actions that we'd like to motivate them to take, and sell the benefit. For example:

Now that you've completed this exercise, let's see how your solution compares to others: [url] To benefit the most from this exercise, find 3 or more submissions that you can learn something from, have questions about, or have suggestions for. Post your thoughts and questions in the comments, and start a discussion. Revise your solution to incorporate what you learned.

After that's done, you can grab the next exercise: exercism fetch \<track>

You could probably tweak the wording all day long. Some things I like about this:

• It plays on the innate sense of competition: “let's see how yours compares”
• It describes reviewing code as the natural next step before moving on to the next exercise
• It reminds you that you're really here to learn, and that these are important steps toward that goal
• It suggests review reactions appropriate for people at all levels: learn, ask, incorporate, suggest

Thoughts?

[GustavoCaso]

This feels more natural.

Help avoid the concept of obligation and plant the idea of the importance of code review, as @nilbus the wording can be changed all the day long but the core concepts is clear.

I really like it.

[kytrinyx]

I've added the "design-research" label as I think this is a really interesting and meaty discussion for us to explore as we think about the user experience on the site.

I appreciate all the thought and detail here.

To benefit the most from this exercise, find 3 or more submissions that you can learn something from, have questions about, or have suggestions for. Post your thoughts and questions in the comments, and start a discussion. Revise your solution to incorporate what you learned.

I love this. I have found that a really powerful usage pattern is when people do exactly this: and thank people when they find something that they learn from.

[tleen]

We could always go down the reward incentivization route: add a review leaderboard somewhere, perhaps split into daily/weekly/..., this is also the type of thing you would see with badges, trophies, etc. The downside is that it could discourage quality over quantity, the latter easy to calculate the former not so much.

Along the lines of @nilbus comment, perhaps having the site actually select a few (recent, not commented) entries and link explicitly to those. So instead of

find 3 or more submissions

it would be more like:

It looks like @GustavoCaso, @tleen, and @kytrinyx have also completed this exercise and could use some feedback.

Where we could link directly to their solutions. This would save the user a step and make sure that recent uncommented solutions would get first billing. Its a way to avoid the sort of Bystander effect users may feel. While not an explicit assignment, it would help in perhaps cutting down the sheer (overwhelming?) mass of solutions from which to choose.

[GustavoCaso]

So the idea is to facilitate the reviewer, the solutions to comment, is not like assignment but suggestions, but following the idea of gamification in my opinion I think it would help the user to comment more, maybe bring less quality because they want to see there badges grow or something.

I know for me the badges thing works 😄, but maybe for improving the quality the reviewed has to upvote or mark the comment as useful so the quality do not go down that much

[tleen]

Unfortunately I think meta-reviews may discourage exercise reviewing: people may not want to be judged on the "quality" of their commentary. That is why it may be tough to un-game any sort of encouragement-by-quantity in exercise reviews.

[Polsonby]

We run a similar peer review system in our learning platform. I think one way to go might be to have a "helpful" button for each review, that anyone can press other than the author. Then the number of "helpful" reviews is what you score people on.

[kytrinyx]

@Polsonby Thanks! I think something like that could work, so long as it's not a number that we use to rank you against others (competitive) but rather some way to encourage you to compare to yourself. Or something.

I wonder if it would be different if we let people opt in to get feedback on their feedback.

[nilbus]

@Polsonby Curious, what is your learning platform?

[nilbus]

I haven't contributed on these discussions before now, so I'll introduce myself. :wave: I'm Edward. As part of completing my computer science Masters program at Georgia Tech, I decided to research and study the question posed here—how can we apply game design to Exercism to increase the amount and quality of participation in code review / conversations, without harming the intrinsic motivation people have to do code review.

@kytrinyx has been rightly wary of introducing point systems or other extrinsic rewards, because research has shown that external motivators such as these actually diminish the internal motivation one has to do the task being motivated. In rewarding peer review with points, we could actually be harming people's intrinsic motivation to do code review in other contexts.

Gamification / game design can however be applied in ways that enhance intrinsic motivation, according to more recent research. Some key elements include keeping feedback informational, not controlling, to increase the sense of autonomy, and aligning the goals presented by us with the goals and motivations of our users.

I have gone into much more detail in #123, including references to backing research, proposed ideas for applying game design to code review in Exercism, and the outline of an exploratory study that I'm performing with a fork of Exercism to test these ideas. Eventually, you can expect to see pull requests for the ideas that work. With the pending UX reboot (#113) that I just learned of, the primary goal will be learning what works well, which can inform the next version of the site. This will be a primary area of my focus for the next 2 months. I welcome your feedback and help with this!

[kytrinyx]

@nilbus this is very exciting! I will pull in the other reboot folks into #123 to discuss further there.

[robphoenix]

A lot of what @nilbus mentions here is also true for parenting :smile:

[nilbus]

A little update… 3 of 7 proposed ideas are implemented so far. I may get one or two more done today before starting the experiment on Monday.

I like @tleen's suggestion:

It looks like @GustavoCaso, @tleen, and @kytrinyx have also completed this exercise and could use some feedback.

we could link directly to their solutions. This would save the user a step and make sure that recent uncommented solutions would get first billing. It's a way to avoid the sort of Bystander effect users may feel.

Right now the call to action just suggests to choose 3. With the CTA code being in the exercism.io repository however, it's totally feasible to suggest and link to some specifics. I imagine something like selecting the first 3 submissions from a TrackStream that have no comments (from real users, not automated). Would anyone be up for implementing that? I don't see a problem with that change going out later than Monday.

[huettenhain]

Hello there, I am just a user, and frankly I have not read everything that has been said here. However, I had my own thoughts about the website and simply wanted to share. Feel free to ignore it. I would suggest to make the review process two-fold:

1. (preview) You can (anonymously) vote on a submitted solution with either "requires review" or "this code is fine". Users have a preview queue where they can vote on submissions with few votes.
2. (review) In actual review mode, submissions are sorted by the number of votes in favor of "requires review".

For me as a user, I think this would make it far more appealing to participate in the review process. If I am not in the mood to write something, I can still scan through solutions and give my thumbs up or down. If I want to write a review however, I know exactly where to go, namely to the top of the "to be reviewed" list.

I also think it would be nice to have tags for reviews, helping me to filter this list further so I can find a submission in need of review where I also feel comfortable participating in the discussion.

[kytrinyx]

Thanks for adding your perspective to this @huettenhain!

/cc @iHiD for some ideas about mentorship and code review.

[kytrinyx]

We're tracking this separately as one of the core important questions for the user experience / redesign that we're doing.

I'm going to go ahead and close this, as I think we've covered the range of points and topics—but rest assured, I'll be back with a new discussion once we've done some legwork on this for "nextercism" (as we're calling the new prototype)