TL;DR:
This PR reorders the core exercises; see the end for the new ordering.
Please read the full thing before replying!
I wrote a script that takes some data points into account to move certain core exercises to be optional side exercises, and which reorders the remaining exercises.
This is not meant to be the "be all end all" of perfection :-)
This is an attempt to make some simple changes in the short term. In the longer term, over the next 12 months, the product team will be doing a bunch of work to dig into what makes great Exercism exercises, and how to structure tracks to provide a better experience for both learners and mentors.
Once this goes live we will continue to monitor the continuation rate, the median wait time, and a few other stats as well, to help decide whether or not we need to take any other immediate steps. That said, if you see anything weird or worrying once the change is out, please ping us on Slack (or here) so we can discuss a fix posthaste!
The most helpful thing you can do in reviewing this is to look at the results and tell me whether anything is obviously terrible or suspiciously weird. I know very little about the Scala track (or the language, for that matter), so I could easily have missed something. One thing to note: there might now be a pretty big gap in difficulty between the ETL exercise and Robot Simulator. We might want to see if there are any side exercises that should be promoted to core.
I'd also appreciate it if you posted this to the mentors as a heads up, and also because mentors tend to have a great gut sense about the core exercises.
Bottleneck detection and fixes
Removal from core:
The following exercises were moved out of the core track:
sum-of-multiples
accumulate
perfect-numbers
matrix
This is a list of exercises that typically are unappealing to students and not very interesting to mentor. They tend to be math problems or implementations of CS algorithms and the like.
Worth considering: Are any of these exercises particularly valuable in the Scala track? If so, we should keep it. Let me know and I can regenerate the PR.
Reordering:
The data that I based the reordering on was taken from the production database, and is based on the last 6 months.
I reordered by taking the original order, and then giving "penalty points" for two different things
low continuation rates (percentage)
long wait times in the mentor queue (median wait time in minutes)
"Continuation" is when someone completes an exercise, but then does not submit the following exercise. We have other interesting numbers that we may use to adjust things in the future, but this one seemed like our best bet for now for the bottlenecks. I somewhat arbitrarily chose 75% as the cutoff for whether or not to give penalty points, i.e. continuation rate < 75% gets penalized. The only exercise I did not penalize based on this rate was the exercise directly following hello-world, since hello-world is basically just a taste test, and it is natural to assume that many people will decide that Exercism is not really for them.
Long wait times in the mentor queue is an indication that mentors don't enjoy mentoring the exercise, or put it off. This could be for a number of reasons, which we have not explicitly identified, but we've observed that sometimes the exercise is simply too difficult in the current position, which means that mentors have to go back and forth a whole bunch with students to get something right, and the discussions can be quite frustrating. Also sometimes the exercise is just not very interesting to mentor. Or if the exercise is too early in the track sometimes people will submit lots of really complicated solutions rather than a small handful of mostly reasonable solutions, also making it harder to mentor.
So this pushes exercises with long wait times further back in the core track. We may decide that we need to remove some exercises from core altogether, if wait times continue to be bad.
Note: the numbers indicate the position in the core track, as defined by the index of the array. Minus means that it moved earlier, plus means that it moved later.
ricemery
commented
5 years ago
TL;DR: This PR reorders the core exercises; see the end for the new ordering. Please read the full thing before replying!
I wrote a script that takes some data points into account to move certain core exercises to be optional side exercises, and which reorders the remaining exercises.
This is not meant to be the "be all end all" of perfection :-) This is an attempt to make some simple changes in the short term. In the longer term, over the next 12 months, the product team will be doing a bunch of work to dig into what makes great Exercism exercises, and how to structure tracks to provide a better experience for both learners and mentors.
Once this goes live we will continue to monitor the continuation rate, the median wait time, and a few other stats as well, to help decide whether or not we need to take any other immediate steps. That said, if you see anything weird or worrying once the change is out, please ping us on Slack (or here) so we can discuss a fix posthaste!
The most helpful thing you can do in reviewing this is to look at the results and tell me whether anything is obviously terrible or suspiciously weird. I know very little about the Scala track (or the language, for that matter), so I could easily have missed something. One thing to note: there might now be a pretty big gap in difficulty between the ETL exercise and Robot Simulator. We might want to see if there are any side exercises that should be promoted to core.
I'd also appreciate it if you posted this to the mentors as a heads up, and also because mentors tend to have a great gut sense about the core exercises.
Bottleneck detection and fixes
Removal from core:
The following exercises were moved out of the core track:
This is a list of exercises that typically are unappealing to students and not very interesting to mentor. They tend to be math problems or implementations of CS algorithms and the like.
Worth considering: Are any of these exercises particularly valuable in the Scala track? If so, we should keep it. Let me know and I can regenerate the PR.
Reordering:
The data that I based the reordering on was taken from the production database, and is based on the last 6 months.
I reordered by taking the original order, and then giving "penalty points" for two different things
"Continuation" is when someone completes an exercise, but then does not submit the following exercise. We have other interesting numbers that we may use to adjust things in the future, but this one seemed like our best bet for now for the bottlenecks. I somewhat arbitrarily chose 75% as the cutoff for whether or not to give penalty points, i.e. continuation rate < 75% gets penalized. The only exercise I did not penalize based on this rate was the exercise directly following
hello-world, sincehello-worldis basically just a taste test, and it is natural to assume that many people will decide that Exercism is not really for them.Continuation rates
``` - hello-world (100%) - two-fer (33%) - leap (100%) - bob (74%) - space-age (89%) - sum-of-multiples (100%) - grade-school (81%) - accumulate (100%) - hamming (95%) - etl (99%) - perfect-numbers (99%) - secret-handshake (92%) - robot-simulator (88%) - matrix (100%) ```Long wait times in the mentor queue is an indication that mentors don't enjoy mentoring the exercise, or put it off. This could be for a number of reasons, which we have not explicitly identified, but we've observed that sometimes the exercise is simply too difficult in the current position, which means that mentors have to go back and forth a whole bunch with students to get something right, and the discussions can be quite frustrating. Also sometimes the exercise is just not very interesting to mentor. Or if the exercise is too early in the track sometimes people will submit lots of really complicated solutions rather than a small handful of mostly reasonable solutions, also making it harder to mentor.
So this pushes exercises with long wait times further back in the core track. We may decide that we need to remove some exercises from core altogether, if wait times continue to be bad.
Wait times
``` - hello-world (0 min) - two-fer (1013 min) - leap (402 min) - bob (515 min) - space-age (840 min) - sum-of-multiples (3721 min) - grade-school (557 min) - accumulate (906 min) - hamming (588 min) - etl (856 min) - perfect-numbers (450 min) - secret-handshake (2162 min) - robot-simulator (965 min) - matrix (369 min) ```Outcome
Before
After
Note: the numbers indicate the position in the core track, as defined by the index of the array. Minus means that it moved earlier, plus means that it moved later.