jhoffner
commented
6 years ago I hear you and I don't argue that the system needs to be improved, however we need to go based off of data, determine what the key issues are, and provide solutions.
So, what is the problem, defined concisely and backed up with data, and what is a proposed solution? That would be the ideal format to begin this discussion.
For now I will try to cover what I can from what you wrote.
it is set up as 0 points for 100% satisfied
It is currently 0% for 0 votes. If the first vote was a not-satisfied, it would be 0% of 1, if somewhat it would be 50% of 1, if very satisfied it would be 100% of 1.
while every somewhat satisfied and not satisfied deducts 1 and 2 points
There are no points and nothing is being deducted. We count the number of positive votes and divide them by the total number of votes. If 4 people voted "very satisfied" and one voted "somewhat", it would be 4.5 / 5 = 90%. If 4 people voted "very satisfied" and one voted "very unsatisfied" it would be 4 / 5 = 80%.
And I (and maybe some other people too) have lots of problems about it because while nobody wants poor quality katas to get approved, we still want katas to be approved.
I just did a check, out of ~2k kata that are currently in the "needs testing" state, only 9% of them currently have reached the minimum vote count and have no open issues, meaning the only thing left holding them back is their low satisfaction score. I would love to see examples, actual kata, that have been held back by the satisfaction score benchmark so that we can start to break down the issues of what happened. As the data stands now, 9% doesn't scream complete bottleneck and of that 9%, it still has to be determined how many of them are actually good kata that got grouped into the poor ones.
Of course that 9% is just those only held back. There may be other kata that have open issues but also stand little chance of ever making it back to redemption due to a low satisfaction score.
If you look at every single kata that's most completed, they've all converged to ~85%. (And the entry exam kata, Multiply, has just 75%.) By the dumb 90% requirement they wouldn't even have been approved at all.
There are a number of factors going on here which makes these kata completely irrelevant for comparison.
- Most are very old, which means they inherited the data from the previous voting system and are not purely reflective of the updated system
- They have been approved, which means they are no longer being voted only by a subset of the community which cares to vet beta kata. The voting behavior will change once approved, that behavior is not relevant to the 90% benchmark which is only relevant pre-approval. Something can certainly be said/argued that since voting behavior changes there should be something involved to deal with that and reflect it within the algorithm that determines the satisfaction score - which currently isn't. However it terms of the issue being presented, how satisfaction scores converge post-approval are not relevant to the main issue being presented here - which is that beta kata cannot be approved due to low satisfaction scores.
"Multiply" is even worse of an example, as it is only meant for signing up. If it was a kata being introduced today, it should hopefully be much lower than a 75% as it isn't an actual good kata for once you are passed signing up. By normal CW standards, it's steaming hot crap (but works well for the landing page format).
In the end, we probably should separate "katas that are hot garbage" from "katas that are not hot garbage but not stellar".
I think lowering the bar a little bit makes sense. 90% may be too high for white kata. It is pretty hard to get back to 90% once you get 1 negative vote and its a very small margin of error.
A list of some issues with the current system
😢 Once a kata has a very poor initial reception, it's basically impossible to get its rating back up.
Movie ratings make sense because once the movie is release its done. Kata can be iterated on and changed. I think the fix here is to simply restart the beta process for the kata. If it has a poor rating, then it probably should be rethought and a v2.0 should be released. How far the system goes to facilitate this process vs just having someone unpublish and copy their challenge to a new one is debatable, but doing it manually is a workaround that works for now.
😢 The rating system does not take into account bad actors
Bad actors being those who tend to downvote the same user, who are new to the site, never complete a kata they downvote, etc.
😢 Rating characteristics changes post-approval, due to a larger audience now rating the kata.
The system may need to start distinguishing pre/post approval ratings and tracking this information. This combined with other characteristics (such as rank level, power user, etc), the rating algorithm could be updated to consider these factors.
😢 The margin of error for reaching the required satisfaction score, especially for white kata, is too great.
I would like to run some numbers first, maybe look at some kata that are good but didn't make the cut, to try to find a good range, instead of just arbitrarily selected one (like done initially due to lack of data). With that said, moving it down to at least 85%, maybe 80% probably makes sense.
Side note, it would be very interesting to see everyone with an opinion on this to create their own kata with their own requirements & expected algorithm for calculating the score - as an experiment in collaboration.
docgunthrop
Voileexperiments
anter69
ghost
Blind4Basics
dolamroth
This is a response to #1165, and a detailed tangent (maybe?) to #997 that I feel like would benefit to stand on its own.
I've tried to talk about this many times before, but I don't think jhoffner has got my idea yet:
Setting to expectation to be "everyone will vote
very satisfiedunless otherwise" is not a realistic measure at all. By how the current beta process and satisfaction metric are set up, it is set up as 0 points for 100% satisfied, while everysomewhat satisfiedandnot satisfieddeducts 1 and 2 points. This is not how ratings system work literally everywhere else on the internet:From the distributions of 5-star ratings I've seen on various places, when you have something good the distribution doesn't go like everyone's voting a 5: usually it's an exponential curve that begins at 5 and tails at 1. (Of course, there's another such exponential curve pointing at the other direction. For good stuff this part is negligible).
So even if a kata is, as jhoffner usually emphasizes, "high quality", it's not gonna get a pure 100%, i.e
5/5; it'd be more like4.2/5. And this will get even lower as you throw more people into the rating system (I'm talking about at least a thousand votes, so blue/purple katas are out of the question). Just look at how many white/yellow katas that get approved and then converge to something like80%. If you look at every single kata that's most completed, they've all converged to~85%. (And the entry exam kata, Multiply, has just75%.) By the dumb 90% requirement they wouldn't even have been approved at all.And I (and maybe some other people too) have lots of problems about it because while nobody wants poor quality katas to get approved, we still want katas to be approved. What is
high qualityis subjective, but what isnot hot garbageis mostly objective. We don't want a bunch of non-hot garbage beta katas staying beta just because they're like4.3/5, and the dumb approval requirement says90%satisfaction rating is required (i.e4.6/5).(This also doesn't mention another elephant in the room: beta process is usually participated first by the power users, so the initial statistics kinda matches what jhoffner expects, but not for the expected reason. It doesn't work in general. Basically, the beta process somehow works by some amount of luck.)
In the end, we probably should separate "katas that are hot garbage" from "katas that are not hot garbage but not stellar". We can all agree that we should not approving hot garbage katas and we should approve stellar katas, but still we have tons of katas inbetween (mostly in the white/yellow range), and the various discussions in the past doesn't address these katas at all. And the satisfaction system doesn't help at all either, instead it just puts these katas dangling between the boundary of approving and not approving (which is arguably worse, now it depends on whether a
somewhat satisfiedwill come out randomly from somewhere, or nowhere). We need a system that focuses and addresses these katas currently standing on the verge.┆Issue is synchronized with this Clickup by Unito