search

codewars / codewars.com

Issue tracker for Codewars
https://www.codewars.com
BSD 2-Clause "Simplified" License
2.08k stars 219 forks source link

The entire kata curation process is outright broken #1626

Open Voileexperiments opened 5 years ago

Voileexperiments commented 5 years ago

This is the culmination of my lines of thoughts in gitter chat during these few days.

In short, I'm very unsatisfied with the current beta process and I have lots of reasons why it should be completely redone.

Supposedly, CW has a beta period for katas where user-generated content is rated by others before they get to approval. The purpose is to get the katas in a good shape before they become official CW content.

The problem is, it has never worked since the first place. And it's really going downhill since the last year.

I think at this point, CW problem list's status is so unsalvagable the only course of action is to purge the existing, approved katas, at least in a half. And there are no one but all of us to blame for that.

What do you mean? Beta has always been fine as it is!

No, beta have never worked as intended in the first place.

At the beginning, the katas were overranked because the overall skill level of even the power users are quite low to begin with, and the problem set are just, well, lackluster. Then CW attracted more people used to competitive programming sites.

The result? Creativity, you thought? Nah! Creativity almost always come from only a handful of skilled person. Instead most of what we get are these instead:

In fact, I'm surprised nobody has done a copyright claim on CW user content yet, given how many katas are straight out plagiarized. It could really, really, really have gone very bad. In some sense, CW is really lucky here.

And nowadays, since everything is approvble after 3 votes, pretty much everything that is not outright abyssmal (extremely poor kata idea/test setup or straight out duplicate, of course, only if someone points it out during beta process) get approved. Most katas that are approved aren't even in a good shape and has issues lurking wide open.

In other words, the average quality of the katas are very appalling. Emphasis on average because there are always great, original katas, but they only come from a tiny portion of users, and when you have pearls buried by a pool of sand, talking about the existence of pearls alone is pointless. They just get drowned by the sand, and you'll never be able to find them from the sand pool.

But we're already rewarding users for participating in beta process! And they can raise issues! Surely the users are doing their job properly, right?

Haha, no. That's a cute and idealistic thought, and has proven to never have worked in the beginning. Why? Because most people are just going to give a very satisfied no matter what. The modus operandi of the majority of users are:

Even the old power users do this all the time. And they don't raise issues for things that are actually issues (good luck of they even raise it as a suggestion). Raising issues for missing random tests and other stuff were not pushed and advocated until like last year.

This should give everyone an idea how unreliable beta process performed by users are. In other words, it straight out doesn't work. After all, everyone's treating beta process as in filling the questionnaire survey after eating at a restaurant, not reviewers/judges doing peer-review and giving serious feedback.

But hey, I used to be a newbie and I grinded on these katas to get better. I think they certainly have a value, and so they should stay!

One of the main counter-arguments I have encountered so far.

For one part, "beginners need repeated practice to hone their foundations" is legit, and I agree with that. The problem is, you're not supposed to let user-generated content fulfill that purpose. They belong to the class of procedurally-generated problems, e.g Wolfram Problem Generator, which is a completely different category. It's just like you don't mix campaign mode and endless mode together in a video game.

Also, if most of the user-generated content you're featuring are low-quality with the average quality of, at best, only marginally better as procedurally-generated problems, that goes on to tell a lot about the quality of your user-base. Basically, your katas are not even katas, and your users are really just shitposting katas.

But if we're actually removing approved katas, everyone will lose honor! This will discourage user!

Also a counter-argument I see oftenly.

Every change will break someone's workflow, so that's a moot argument. Moreover, the counter-argument is missing a key point: what are you trying to encourage/discourage at the first place?

I always have a belief like this: if the reason one's doing a kata is to get some imaginary internet points, if they're so low of self-esteem they have to be validated by these imaginary internet points for the struggle, then said person also have blistering ego and they should probably reflect upon themselves, rethink their life and why are they even learning, or coming to CW in the first place. The ultimate purpose of someone training on CW is learning; honor is just part of the gamification which, ultimately, is also exploitationware, to give users excuses to build up motivation to train themselves.

Along this line of thought, excessive inflation of honor points reduce the sentiment of the award you're giving to the user (after all, if everything is 1kyu, there would no satisfaction finishing a 1kyu kata), Being so keen on honor whore (akin to "rep whore" on StackOverflow) also drives user to cheat, which we already have a big list on this.

And guess what? The more ironic part of this is, you don't get the most points for doing all the katas. You get the most points by shitposting very easy 7-6kyu katas, which will guaranteed to gain lots of tractions once approved, providing your with a continuous source of honor since every upvote gives you 1 point. This is self-evident from the leaderboard. So if by "discourage users" you mean "not allowing users to be honor whores and shitpost lots of easy katas for really easy points", then sure. (Also, by the way, this system is completely imbalanced and should also be revamped. Yeah.)

But do we really need every kata to be under such scrutiny? They're just entry-level katas, for entry-level users. Why not be more friendly to entry-level users?

Also a typical counter-argument I've seen.

Apparently they're okay with the natural, disastrous implication that "my users being lower in skill level means I can feed them hotter garbage to them, they won't complain anyway since they're not as picky as skilled coders". Funnily enough, my observations on various competitive programming sites is indeed following this trend: the average problem set quality on a particular site is correlated to the average skill of users on there. So, the rest is kinda obvious, and is left as an exercise to the reader.

Besides, if anything, you want to teach entry-level coders well and lead them in the right path. With like a million forums, subreddits and discords out there where lots of people who don't know what they're talking about most of the time get treated as "teachers" simply for sounding intellectual, the last thing you want to do is to add fuel to this majestic fire.

Why are you posting issues all over the katas? Why are you being so un-cooperative? I just want to have my katas be in peace. You're nothing but an annoyance that blocks everyone's progress.

Oh yeah, right, the fun part where kata authors are living in their ivory tower, being full defensive and don't even try to communicate with others, or just straight out denying that their katas have problems/their code are sloppy.

Sure, I mean, in where else would you see creators blocking lots of users to avoid issues being posted on their creations, and outright admitting it? Probably nowhere else but CW. So that's a record I guess?

While it's extremely easy for one to just shove the entire problem of beta process under the rug as "some users are not being friendly", anyone with a brain should've already realized that "being nice" is not going to solve the issue at all. The issue is too many users are shitposting too many katas and dragging the (quality of the) entire site down. Being nice will not help the issue. Blocking low-quality katas from approval or removing existing low-quality katas, however, do. (Of course, barely anyone actually do this. It's why the beta process is crap in the first place.)

In fact, if anything, everyone should thank the power users for making the beta process not in the worst shape possible. If there weren't anyone who'd dive head into new beta katas and downvote the bad stuff, while also posting issues to block katas approval, the worst stuff would've been approved already. Don't forget: most users don't even care if the katas has lots of issues. They just fumble their way out, give an upvote, and never appear again.

If I can solve a newly approve kata without picking out at least 1 issues per kata on average, then perhaps the beta process has finally improved. However, as it currently stands, it's still like that. It's a pile of hot garbage.

And after all, as a certain unihedron has said, CW being such a mess is, ultimately, not my problem. I'm only giving my damn about the state of it because I still care. In fact, most users don't care if the entire site dives into oblivion.

Why are you raising this? You know this would just drive away potential users, right? That's not helpful to CW at all.

Okay, so let us get the really, really ugly part straight away, shall we?

Given the current situation and conflict around kata creation, you can choose CW to aim for either one of these two things:

The elephant in the room here is this: if CW dives into the first option, it's out of the competition to literally any of the competitors. Every other entry-level competitive programming site is much, much better than CW: they are more actively maintained, they have more resources, they have much better community and social engagement, like actually having a dedicated forum (the only way to communicate is by CW comments and gitter chat, which is simply horrible), they follow the OJ format and use stdio for interaction and so is much more practical. Heck, even their sites are at least no as bug-ridden as CW do.

There are really no reasons anyone doing entry-level programming would have chosen CW over literally every other site. No kidding, the only reason I can use to maybe try to convert someone else into doing CW kata is, there are some interesting problems, and that's it.

(Besides, CW is a side-product of qualified.io, which uses a closed, curated list of programming problems, and I doubt most of the katas being approved nowadays can even go into this list.)

It is in this sense that I've always been advocating for striving for the second option. It's the only reason CW's existence as CW can even be justified in the first place. Also you can guess who'll be more likely to subscribe for CW Red: entry-level users, or recurring, long-staying power users?

Incidentally, codingame has made code contests (aka clash of code) user-submitted for a while, and people agree that in general, the quality of them are kinda bad with ambiguous description and dubious tests, aggravated by the site moderator's inability to review and curate them before making them public for everyone to attempt.

CW is basically in the same boat: you can't rely on the users themselves to maintain high quality content, unless your community is closed and has a slow influx of new members, because users do not curate content seriously by default, and they always have the tendency to create low-quality content because it takes little effort. It's always an uphill struggle.

Or in short, there should be some kind of dedicated resources dedicated to curating beta katas and to weed out the low-quality existing katas. (Don't forget that #1315 is still a thing.) And it should be done as soon as possible, since it should've been done since a long time ago in the first place. Getting into a position like CW currently is basically means the entire process has utterly failed.

┆Issue is synchronized with this Clickup by Unito

kazk commented 5 years ago

Thanks for taking the time to write this.

Just one thing, I appreciate that you're concerned, but all the negativity doesn't help. It's not about just being nice to the people who made something you use, it'll lead to maintainer(s) to burn out. It's already extremely stressful to work with undocumented and poorly maintained legacy code with thousands of bugs that's affecting users everyday.

Your assumption that we believe the current system is working is wrong. We know Codewars is full of bugs (much much more than reported here) and the quality of the content isn't high. Now we have more than 8000 kata and good ones are buried. I don't know about Jake, but I've been concerned about this specific problem for a while.

For the next version of Codewars, which I haven't started since I'm still learning the current code base and fixing issues as I see, I'd like to introduce tiers of kata:

I'm also thinking of using Discourse for forum for better ways to communicate.

Note that these are not official yet.

I don't think it's realistic to add features to current Codewars because of the way it's written and maintained.

hobovsky commented 5 years ago

I have some thoughts on he topic, I have some insights, I just think that for the time being, it's of no use to rant, complain or present any great ideas how to tackle the problem. I might, if someone asks and wants to listen, but I think that it's not a time for that.

What I would like to say, however, is that I think that it is possible to 'temporarily fix' things with some functionalities and/or workarounds available now. Good first step would be to increase the amount of required honor for creating a kata and creating a translation.

Even better solution would be to make content creation (or content approval) process completely unrelated to honor points, but I am just not sure if it's possible with some minimal changes/patching up existing CodeWars. Just tell some users (let's say for starters that everyone >= 1 dan implicitly volounteers :) ) they are moderators from now on, and only they can approve content. No more crappy katas out of beta anymore! But I am aware that this approach might be 'not in scope' of current activities.

To balance Voile's negative attitude and make maintainer(s) feel better ( ;) ) I'd just like to say that I personally feel that CW is a great collection of problems and exercises for both newbies and experienced coders, and I am looking forward CodeWars 2.0.

Voileexperiments commented 5 years ago

To be fair I wasn't this negative and cynical about the kata process in the first place. But then once upon a time, I took the time to go through (almost) all the existing approved katas, and the experience was, well, boring but still okay. However, then I took the time go through most of the existing, dangling beta katas, and the experience was... an insurmountable amount of frustration. I think that was the time I just don't care about it anymore, because apparently if such content as a major part of the beta katas can exist, what's the point of holding back anyway?

(And it's not like they've stopped. The only thing that helped somewhat is instant retirement of beta katas. Which doesn't address the existing katas at all: the best we got was the ability to fix existing katas, which is still a massive work, and still doesn't stop the new ones coming. Also, duplicates and overranked katas.)

I wish I don't have to point out all the obvious things and write such a long post, but the point is, again, that we shouldn't have had reached such a state in the first place. It was never handled for the entirety of CW's life, which means everyone is responsible, and has contributed to this (or otherwise it'd already have been pointed out and extensively discussed). The fact that it managed to have snowballed to such a scale without control and still growing is disheartening. The frustration from the pile of existing katas easily build up, and nobody who has gone through the majority of them has anything good to say about these katas. So don't expect anything not negative to come out from discussions about kata process. The massive negativity is kind of the only possible course of response when it's making the experience truly miserable for anyone who bothers to actually go curate the content.

kazk commented 5 years ago

@hobovsky Thanks :) I agree that honor is not appropriate for this. I also agree that it's currently set too low, but some users with really high honor points make low quality translations too (giving them lots of points again).

@Voileexperiments

So don't expect anything not negative to come out from discussions about kata process.

I'm talking about the tone. We should be able to discuss without taking the frustration out in it. All I'm asking is to keep it constructive. Your comment have some good points, but it's wrapped in unnecessarily negative tone, so it takes significantly more energy to read through them.

you can't rely on the users themselves to maintain high quality content, unless your community is closed and has a slow influx of new members, because users do not curate content seriously by default, and they always have the tendency to create low-quality content because it takes little effort.

I believe this is the root of the problem. Codewars grew and the strategy was no longer effective, but no one took the action to change it.

Another factor that contributed is that the original developers aren't active users anymore. A maintainer/developer needs to be a user so they have the right context for decisions, understand the users' needs, and stay passionate.

I don't have short term solution for this at the moment. I wouldn't want to retire old beta kata systematically, but I'm also hesitant to let community do it. Codewars is full of N+1 query issues and some actions are super expensive even it doesn't seem like it from outside.

CliffStamp commented 5 years ago

I have said most of what I have had to say with Voile on Gitter, I would just summarize it very condensed here by saying this :

The current honor system rewards mass posting of low quality easy to solve katas and has no reward for curation by users - hence it is no surprise there is mass posting of low quality easy to solve katas and little curation by users.

Ask yourself what you want users to do and adjust the honor/ranking system to reward that, this is basic human psychology, reward what you want to promote, ignore what you don't.

Unihedro commented 5 years ago

Just because the results don't turn out well doesn't mean the current system is the worst possible system out of the entire universe. Trying to redevelop a system is difficult; everyone can look at what the results are and say "well, we wish this and this was true", but systems simply aren't as simple as cranking a feature and having the right results come out magically. Designing from the ground up is hard. It's a lot more practical (read: easier to do) to look at where we got ourselves into and look at how to make things better, instead of taking a fatalistic world view and assume that nothing will improve. Here are my points of interest:

  1. The model of codewars favours user contributed content and distributes them as a curation platform, not as a publication platform.† There are incentives for the model to remain this way, since it will stay easier for both current users and future users, and it's unlikely that changing this will actually be easy considering the speed at which new problems are written and it requires the work of dedicated reviewers - likely volunteering - which will lead to another bucket of problems. Using the same object to study from OP:

Incidentally, codingame has made code contests (aka clash of code) user-submitted for a while, and people agree that in general, the quality of them are kinda bad with ambiguous description and dubious tests, aggravated by the site moderator's inability to review and curate them before making them public for everyone to attempt.

For a long time, clash of code problems have been user submitted, for a time frame long enough to be considered "through the very start". However the ability to approve / reject problem proposals was a privilege - limited to power users at Lv.20 or above (quite high). Not a lot of power users actually care about reviewing problems, and proposals can sit for 6 months to 1 year until someone makes a comment on it or rejects it. (A lot of the problem authors would have already stopped using the site by then and wouldn't be able to respond / improve their problem, so it just stays there, and no one wants to touch it either). A reward system was added - If you vote (either approve or reject, but approving requires you to check a list of items) and your vote ends up in the overruling majority, you get rewarded experience points. This led to a lot of users farming reviews on clash of code proposals even when they are low quality without bothering to improve the problems, since it was easy experience points.

This happened before they changed it up so anyone who has played for 50+ clash of code rounds (a very low bar that qualifies most users), and now the flood of garbage clashes in the pool is simply unstoppable, but it was not the cause, rather a symptom. The main cause is that there was a reward of experience points in the first place, as a solution to make users review more. Well, they succeeded.

Since changing this system is unlikely to be ineffective, it's better to simply add a second layer - "featured" problems or "official recommendations" as a select class for people who need a better direction than having to navigate the rest of the low quality katas. This is obviously not a solution and will not be a replacement to changing how the current rewards are, but it will help leading users into the good problems without actively disencouraging bad problems - which will likely be just as effective for the energy that goes into it, if not more.

†. More specifically, a system that takes in content and presents it without filtering would not be a publication platform, such as youtube where there are no restrictions; Platforms such as codeforces where all content are specifically reviewed before being shown are considered to be publishing.)

  1. The current features which are supposed to be vehicles to facilitate using the site simply don't do the job. I don't want to rant so this part will shorter.

Collections are supposed to let you manage favourites / types of problems that lets others search for your collection, or for you to go through them to practice. It's very hard to use. The /collections always takes a long time to load, especially if you scroll down and force it to paginate. The tools in this feature are poor, you literally only have two options to filter by: "Created by me" or "Any", without even an option to filter by friends. Most collections are also named poorly and obviously aren't meant to be public, such as names like "aaa" or "TODO", collections should be able to be made unlisted or profile only.

The email newsletter often recommends stuff I've already done. It's also not really that helpful. But I think you already know that.

The leaderboards are a joke.

The wiki is split between the on-site wiki, the wiki of this repository and a wiki in some other repository (which is linked to from the kata writing pane). (They're not even on the site)

No searching by tags. You can show by one (not two) specific tags by clicking it but it relies on finding a problem that already has the tag. Also that means you can't list by AND of two tags. And the site doesn't tell you when you're set to showing a tag so when you search the tag filter remains (and can only be seen through the URL).

Studying your past work is difficult. More specifically the pagination for profile items are difficult to use. If I'm on Stack Overflow and I want to link someone to my past answer, I can search for it or click on the right tab and navigate the pages comfortably which is not the same experience as that of trying to find past code in CW.

I personally do believe that the points on article 2 are just as important if not more important than the kata model and I wish you all the best on trying to figure out the next version of codewars.

Voileexperiments commented 5 years ago

So, there are some additional points that I should add to this thread:

(Warning: the tone has not changed at all. You'll see why at the last point in this thread.)

Why can't we just reward the users more for the right things? Surely we can just solve everything with this method, right?

This sounds good in theory, just like all those "if we do this and this, we will get world peace and everyone is happy!" speeches. They never hold in practice:

Besides, I don't think mangling "maintaining site content" with honor points (which are supposed to represent the entire gamification) is even a good idea in the first place. The former is an action that sits outside the game, and does not belong to the main game. A reasonable thing to be done instead (note that I didn't say this is a good idea, it could be bad) is to mark the users who actively maintain lots of site content and is proven to be actually helpful in this as "contributor/maintainer", which will properly recognize their status and effort, and also not mangle these effort as the main game.

But, we can resolve this in a positive way, right? Why do you have to be so negative?

This is a direct response to @kazk:

I might believe in this in last year, but if CW has taught me anything, it's that if you don't push for something hard and continuously, no discussions will be made and nothing gets improved.

CW is incredibly fragile in community management, which are not related to site function but should've been done already since a long time ago. I don't even know why nothing has been done for an entire 5 years. As far as we all know, there are zero community management in place from kata creation to beta period to kata practice to kata trainer suggestions to comments to literally everything, which I've already raised issues about lots of them:

So far the only things that are actually done are #1010, #1065, #1174 (not able to vote until solved the kata, auto-retire katas and relieve the approval criteria), #1301 (but note that the follow-up tasks are mostly still in-progress). However note that they were not implemented until lots of pushing has been done. Re-ranking katas is now still forbidden (#1472), and I will push for that at some point.

Yes, I know CW is doing a rewrite at the moment, but these issues have existed since a long time ago, even before the rewrite. Why aren't they done back then? The site would be pointless even if it had all the functionality, if the user base were not performing up to par. Community management should've been one of the top concerns, especially since CW is powered by it.

This issue is the culmination of all the previous massive issues about site community management: those issues are just the symptoms. The real issue is the lack of management, and it is still there. It is depressing and negative, and we feel bad when it's pointed out bluntly, but we deserved it in the first place; we let it snowball to this point where it drags down all the power users. So please deal with it and actually do something instead of complaining about the negativity; it's not like I want to bring up all these things together in a massive thread like this either (it's time-consuming for me too!), but as these issues gradually build up without resolve I can fairly I hold very little faith about our ability to tackle these problems. So I have to push harder, signal how problematic these things are, and urge discussions and resolve.

kazk commented 5 years ago

@Voileexperiments

This issue is the culmination of all the previous massive issues about site community management: those issues are just the symptoms. The real issue is the lack of management, and it is still there. It is depressing and negative, and we feel bad when it's pointed out bluntly, but we deserved it in the first place; we let it snowball to this point where it drags down all the power users. So please deal with it and actually do something instead of complaining about the negativity;

I think you're misunderstanding what I meant by negative tone. Pointing out the problem is necessary and that's totally fine. I appreciate the feedback even if it's a negative one. I've never said to be positive and I don't even know what "resolve this in a positive way" means. All I wanted was constructive discussion without unnecessary emotional overhead (at least try to keep it minimum).

I didn't think asking that would be a problem and I don't understand why you're focusing on it because that wasn't my main response. So I'm going to give up on you. I've already tried multiple times and I don't think I'm making myself understood. It's clearly a waste of time and ruins the whole point of minimizing the overhead.

By the way, I have been dealing with it and I also wrote about my plan in the first reply. The problem is that there're so many obstacles in order to get there.

You're also misunderstanding the situation and/or underestimating how bad the current state is and/or never worked with legacy code.

Yes, I know CW is doing a rewrite at the moment

It hasn't even started and I might not be able to if I get assigned to other tasks with higher priority. If that happens, I'm still going to try to do it on my own time though.

I've been fixing issues and updating ancient but core dependencies so that at least Codewars won't be forced to shut down when a service discontinues the version it requires.

I'll give you an example so hopefully you understand my pain. I'm currently trying to update to something that was released in 2013. Something from 2013 is an update. Finding relevant information is already difficult. And I'm fighting with 6 years old code base with minimum documentation and it's written in Ruby so anything is possible. Before this, I was updating things in order to eliminate dependencies with vulnerabilities reported.

This should improve the performance a bit. It's necessary before considering to let users have more control because many of the actions to manage contents are expensive. It'll affect overall site performance that's already bad. And I can't even estimate the impact because everything is tangled up and implicit.

but these issues have existed since a long time ago, even before the rewrite. Why aren't they done back then?

This is very simple. Because no one was working on Codewars. The active development basically stopped around the end of 2014 according to the repository history. Not much changed since then. Then Codewars Red was quickly added in 2016 because it was necessary. So it's been just occasional fixes when developers had the time.

Community management should've been one of the top concerns, especially since CW is powered by it.

I completely agree, but it wasn't and I can't change it at the moment.

You don't know how much I want to fix this mess and how much I've been working towards it. Pushing harder won't speed up the progress for sure. I've used more time than I wanted to for this reply and I'm exhausted.

g964 commented 5 years ago

@kazk: Don't feel bad. You are not blamed. Everybody knows that you are working great and CW is not that bad!

hobovsky commented 5 years ago

While I mostly agree with Voile and I am really impressed by his skills to dig through the piles of pretty much anything and collect the results into pretty nice conclusions, I also think that his approach of "this and that is broken, whys isn't this fixed yet, we need to have this changed" is not appropriate in the time being, because I guess CW staff/maintainers are so overwhelmed with current tasks that it's not possible to address the issues and bitching about them (while somewhat justified) simply does not help in any way.

I thought that CW could use another approach, which I'd call A little less conversation, a little more action - since quality drop was noticed by more and more users, some of them (more or less independently one from each other) thought they might make some difference: put more effort into beta reviews, do not insta-approve translations and actually review them, point out issues in old and new katas, or use power user status and fix issues right in the edit panel (to add random tests, for example). I do realize it's not a remedy for problems with the platform, and I do know it has limited possibilities to help with existing, approved content - but hey, it's something, it's possible with currently available means, and it might help with flood of low-quality content (although not 100% bulletproof, because still someone might not adhere to community-defined 'process' or other good practices). Great initiative, I think, but... as every community driven initiative, it suffers from community problems, and one of such problems are ambiguities and lack of guidelines for various aspects.

And this is the place where staff/admins/owners could come in and help a bit. Hint others how to do things so they conform to overall CW's vision. Am I allowed to rip kata apart in edit panel, and put it back together, improved, with issues resolved, consistent between language versions, while invalidating tons of existing solutions? Are katas "language-first", and translations are just "convenient possibility", or maybe rather a task is primary drive of a kata and it should be kept as language-neutral as possible? Am I allowed to change "return matrix or -1 if not possible" to "return matrix or null/raise error/whatever" in a Python kata to make it easier to translate? Am I allowed to add a "do not mutate input" requirement and tests to existing language version, if other versions already have it? Can community count for some kind of support from people who can do what users can't? I personally would be willing to search for all polynomials katas, or math expression parser katas, and deduplicate them in some agreed way, but is it wort an effort if it's not possible to retire or delete existing ones?

The matter is, there are users willing to help. There are also some users who wanted to help some time ago, but they got bitter over the time and they lost their will ;) But it's not as easy as rushing into edit frenzy mode and fix issues in a way which is perceived best by one user, because it might not be what is considered a good way by others. There needs to be some medium to resolve ambiguities and drive "potentially helpful" efforts in some direction. And users do not know if there's anyone who even cares about that. CW is really underrepresented in community, and other than kazk who works hard on solving technical issues, users see no other person behind it. God is dead. Does CW even acknowledge the problem of content quality? Well, now @kazk have said that, and we know that they do. Do they want to take any steps to make things better? Yes they do, there are plans, there is work in such direction which we do not see directly, but they are understaffed and too overwhelmed by technical issues to take care of community, and I totally understand this. But, does CW want to be helped by users? Is there any way we could help? Is there any possibility to hint us or give some directions what could users do to take at least some burden from CW's back? Or maybe let's just leave things as they are, let's solve katas, and let's not bother staff with problems nothing can be done about right now anyway?

There's still lots of purples and blues for me to do, so I think I will stick to CW for a while. But I prefer action to conversation, so if there's any sightliest possibility, I am willing to help. Just let me know how.

kazk commented 5 years ago

@g964 Thanks for your support :) I'll continue doing my best to make Codewars better.

@hobovsky Thanks. I'll definitely think about what the community can help.

In the past, many users have helped fixing the kata affected by the new runner and that was extremely helpful. And I think that format worked well. So I might create similar pages that users can collect information and help with fixing. I haven't decided how to handle duplicates and low quality kata yet, but the collected information will be useful later. #1315 will be a good start. It was really difficult to manage the information there, but wiki pages should be much easier for everyone. We should probably come up with some standardized format to make digesting the information easy. If that doesn't scale well, I'll create a new repository just to track content issues that's difficult for the users to resolve.

I completely agree about the lack of communication and guidelines. I've been trying to improve the situation whenever I can, but I often don't have the answer either. This will need more time without actions.

Voileexperiments commented 5 years ago

@kazk I understand that CW code base is a mess and it's a pain to deal with. I've never had problems or questions about that.

However note that I haven't talked much about CW being full of bugs, except at one point that given its state it's not gonna be very competitive to other entry level programming sites.

That point was only there to drive my main point, which is this:

Lots of CW power users are uncooperative and cause more hindrances than it's worth in lots of times; which makes actually maintaining CW content, curating katas and fixing existing issues incredibly difficult and frustrating. They're actually undoing what has been done to keep the content sanity from not plummeting below zero, which should explain why my sanity has kinda dropped below zero in some sense.

For some reasons some users keep saying that CW community is good and okay. Well, let's lay out the entire can of worms that inevitably pops up when one (aka me back in a year ago) tries to clean up all the mess around and improving kata quality:

(As a bonus, I've been restraining to calling names here, but anybody experienced enough should know who I'm referring to for most of the points above.)

If the users are straight out uncooperative and harmful, and the system is doing nothing to address this, what do you expect me to do? Of course my blame will go to your system.

Or if you can think of a better way, please advise, because I'm sick of this and I don't want to spend my effort on this lost cause anymore. I've already spent a year cleaning up lots of the ugly parts, plowing through them with ruthless force, but if it's gonna revert back to the mess before all of this is done, you're basically telling me I wasted an entire year of life on CW that I should've used on other more prosperous things.

kazk commented 5 years ago

@Voileexperiments

If the users are straight out uncooperative and harmful, and the system is doing nothing to address this, what do you expect me to do? Of course my blame will go to your system.

What are you expecting me to do? You can blame the system and I agree it has lots of issues, I never disagreed. I'm glad you understand the difficulties and the pain. But now I'm confused because you still sound like you believe there's something I can do at the moment. If you have anything specific, please let me know so we can discuss further.

  • Since there are no clause about what to do with low-quality beta katas, as long as the tests are acceptable and it's not an exact duplicate, it's gonna be approved

  • Issues. What is an issue, and what distinguishes an issue from a suggestion?

  • ... If a kata is lacking something, it shouldn't be approved.

  • Approving good katas dangling in beta is a good thing. Consistently approving problematic beta katas is bad thing.

  • Some kata authors become really butthurt when issues are raised, and start yelling intelligible nonsense.

  • And some kata authors just go straight out defensive and claim every issue around that is not breaking as "not an issue".

  • Ranking.

This is all because users have different standards. I don't think they're actively trying to harm the community. We can start defining official guidelines, but we don't have ways to enforce it at the moment. And I don't think it'll improve much if we continue to rely on the community the current way. I've been thinking about adding some tools to help improve the content (like some automated checks like linting), but that's not going to happen soon.

  • 1365. It actually freaking happened.

This is basically the same problem. He has his own standards, he wants to keep it that way and got sick of someone else telling him what to do. I agree that blocking shouldn't be used that way and it's not implemented well. But the core is the same.

  • Then you have users who keeps making translations that often have issues.

I'm aware of this and I've reported issues multiple times in the past. His translations are often sloppy and unreadable. But what can I do about it? Bad translation should be just rejected. But the process of approving translation is not implemented well (usually only one user is responsible) and discovering pending translation is hard. I can't change this anytime soon because of the technical debts.

  • ... some other users come along and blame you for blocking katas from approval by raising dangling issues and not just fixing the issue right away too! So you're blamed for doing the right thing!

You didn't do anything wrong and you shouldn't be blamed for.

Codewars evolved extremely organically. Users think of Codewars differently and have different standards. This is largely because Codewars lacked authority and leadership. Undoing this will be very difficult if not impossible. And I don't think we should go that route. That's why I'm proposing something that adds on top instead.

CliffStamp commented 5 years ago

Users think of Codewars differently and have different standards. This is largely because Codewars lacked authority and leadership.

It is largely because different people have different standards, it isn't a Codewars specific issue. You could take a piece of code to a hundred "expert" coders and get very different perspectives on issues, from the mundane and trivial all the way up. In fact you could likely pick any issue and find some professional / respected individual who had pretty much any perspective you could name. In JS, recently I have watched a bunch of Kyle Simpson's video's where he attacks the perspective of various respected people on all kinds of issues, and he himself is a much in demand JS educator.

Once you extend it to thinking about how CW works as a teaching tool, well the standards in pedagogy are much looser even than in code and experts disagree about everything essentially, and many things are still in conflict as if they are beneficial or not as a learning tool. When educators have to decide on policy, or what is kept/removed in regards to content, there is always conflict because almost no two people would make the same identical course. There also is no standard you could even dream to apply to such a thing either (due to lack of consistent empirical support), hence the IRL solution is for the people to work together, and those that can't - well they don't.

Right now you have a system which rewards mass production of low quality kata which are easy to solve, and ignores curation and content quality, and actually punishes people who write high quality hard to solve kata. If you don't want that, well change the reward system. This is likely the most uncontroversial topic in all of human behavior, if there is one thing no one disagrees with then it is that. If you set up a system which rewards what you don't want and then put a punitive system in place to prevent it - well that is about the exact worst thing you could do. A completely random reward system would even be better as that would have a overall neutral effect.

kazk commented 5 years ago

@CliffStamp

I'm just mentioning why it's difficult for users to have shared standards at this point and why I think adding another layer is more effective. Also, I'm only talking about standards that can be applied to any kata.

You could take a piece of code to a hundred "expert" coders and get very different perspectives on issues, from the mundane and trivial all the way up. In fact you could likely pick any issue and find some professional / respected individual who had pretty much any perspective you could name.

I'd assume they're consistent within their projects that they maintain. Each project usually have sets of rules that must be respected. In your JS example, a project might have ESLint rules configured for it and any contributions not respecting it will be rejected. Doing this as an afterthought is difficult and very time consuming because contributors have to come to an agreement and then fix/remove offending code.

It's also very difficult to do this at the community level (e.g., think all JS projects). But it's possible if it's introduced early, it's official or officially endorsed, and the users are convinced. For example, almost all projects written in Go have consistent style because of gofmt's success even though developers have different opinions about its style.

If Codewars started with better official guidelines (or developed one with the community) and had better ways to manage contents then I believe it was possible to have some level of agreements between users. At least it should've been much better than the current state that @Voileexperiments described.

Right now you have a system which rewards mass production of low quality kata which are easy to solve, and ignores curation and content quality, and actually punishes people who write high quality hard to solve kata.

Can you explain how you would change it more specifically? Note that quality is not well defined at the moment.

actually punishes people who write high quality hard to solve kata.

Are you talking about a hard kata being stuck in beta for a long time?

CliffStamp commented 5 years ago

I'd assume they're consistent within their projects that they maintain.

Think about how this is done, and specifically how it is enforced. I don't think CW could adopt that kind of policy trivially. Yes it would resolve one problem, but it likely would make worse ones.

As just one example, Crockford has a very specific JS linter and imagine if that was the way JS was to be coded on CW. The community as a whole (not CW, just the larger JS community) is very fragmented on linting and can be hyper polarized on some of Crockfords opinions. What would happen if CW actually mandated that all JS coders follow the same linting? Would all of that opinionated/fragmenting just fall away without conflict?

The general policies that I think CW should have are much less stringent, for example when posting an issue make sure it is actionable. This means the complaint has to be specific, the resolution has to be obvious (or stated explicitly) and enough detail is provided to actually resolve the action. Guidelines like this are likely simply beneficial because there is almost no conflict in the literature on goal setting (SMART goals, etc. ). But if you try to set some kind of quality goals, there is no consensus in the literature on that, and every content expert will give you something different if you were to ask them for a "quality problem" to solve.

How do you solve it then? If you want quality, you have to reward quality. Think about what CW wants, describe it (show exemplars) and reward it. It could be done simply be having featured kata which are ideal examples of how to write them (at any kata level and being picked for that list again is a reward, and is a statistic for the author #of exemplar kata, etc.).

If Codewars started with better official guidelines (or developed one with the community) and had better ways to manage contents then I believe it was possible to have some level of agreements between users.

Yeah sure, but the thing to keep in mind is that you have to be realistic. It isn't going to be perfect at the start, it is going to be broken. And it is always going to be broken, you can just make it less broken by paying attention to what is broken and attempt to change it, ideally with small changes which are well audited and necessary adjustments made. If you look at what CW is attempting to do (manage a HUGE and diverse community) creating a pedagogical tool it is obvious it is going to be in a state of always being broken because there are no known solutions to that general problem (community and content management) and there is no consensus at all in pedagogy on almost anything content wise (look at the constant stream of criticism of Math's curriculum) .

The way to move forward is not to fight that chaos, but embrace that is the way is it and move forward anyway. How to do this is simple - you look at the behavior / results you are seeing, if they are not what you want then you change the reward system and evaluate. Do this over and over, paying attention to the results, using informed opinion when you can which is empirically supported and when you don't have that, well just make the changes based on any kind of inference you can and again watch and see what happens. This is how it is done in the real world by professionals who are trying to create quality educational content - they do things, look at the results, talk to the users and implement change (ideally getting expert feedback when possible).

Can you explain how you would change it more specifically? Note that quality is not well defined at the moment.

To be clear, very clear, designing a reward system for behavior is about as non-trivial a problem as you can attempt because people are kind of complicated. What you can do is just look at the behavior you want to see, and try to reward that when it happens. For example imagine when a kata was published that in addition to difficulty there was a quality ranking

If someone solves a kata they get rewarded based on the difficulty level, but when you write it, you get rewarded based on both difficulty and quality.

actually punishes people who write high quality hard to solve kata.

Are you talking about a hard kata being stuck in beta for a long time?

If you write a very easy to solve Kata, you get a constant reward by people who solve it, this decreases as the kata difficulty increases (due to frequency of solving going down) which is likely rewarding the exact opposite of the behavior you want to see.

kazk commented 5 years ago

@CliffStamp did you read my first reply? I think we're mostly in agreement? What do you think of that?

If someone solves a kata they get rewarded based on the difficulty level, but when you write it, you get rewarded based on both difficulty and quality.

Are you thinking of letting the community evaluate quality rankings? Also what happens when those change over time?

If you write a very easy to solve Kata, you get a constant reward by people who solve it, this decreases as the kata difficulty increases (due to frequency of solving going down) which is likely rewarding the exact opposite of the behavior you want to see.

Good point. There's definitely some bias against less exposed kata. You do get more points when authoring more difficult kata, but the current difference is probably not enough.

CliffStamp commented 5 years ago

@CliffStamp did you read my [first reply]

Yes, I think it is excellent.

Are you thinking of letting the community evaluate quality rankings? Also what happens when those change over time?

They evaluate difficulty, so yes. In the new system you are proposing whoever is curating them would.

If there were exemplars to reference it is likely that would not be as chaotic as you might expect, though again when you ask any population to judge the quality of something you can expect deviations. Ask a bunch of English teachers to grade an essay, you are not going to consistently get 83.5% as the result. You can't even get that in Math grading, so when you look to things like originality, creativity, etc. well expect some swings. But you have to keep in mind you are looking at the total/average content, not focusing on one particular kata which is off ranked.

The difficulty rankings are somewhat chaotic now, but there are not that many current outliers and the disagreements are usually just over one level. Everyone now and again someone will set some difficulty level crazy off, but it is rare from what I have seen, it isn't like there are a bunch of 1 kyu's which are actually 8 kyu's. There are a bunch of people who seem to be polling the content stream because kata's tend to get approved as soon as they are able (the old ones, which are not in the stream sit in the land of lost and forgotten).

In any system, there will be changes over time, and any time you have changes you will get resistance from the people who have to meet the new standard complaining about the ones that had the old. It goes away once people realize that new standards happen, and complaining about them isn't useful. What it takes to be a professional in something pretty much is in constant flux.

hobovsky commented 5 years ago

If you write a very easy to solve Kata, you get a constant reward by people who solve it, this decreases as the kata difficulty increases (due to frequency of solving going down) which is likely rewarding the exact opposite of the behavior you want to see.

Good point. There's definitely some bias against less exposed kata. You do get more points when authoring more difficult kata, but the current difference is probably not enough.

IMO any authoring efforts, like creating a kata/translation, having your kata solved, etc. should not be rewarded with honor points at all. It would stop noobs from creating easy katas, and authors of difficult katas are mature enough to just do not care - they earn honor by solving katas, and not creating them. If it was up to me, I would limit a reward to a Recognized Kata Author badge or anything just as decorative, and no Honor reward. It would probably cause that katas would be created by people whose intention is to actually enrich the site and its content, and not just have the egos inflated by imaginary ranking.

Ideally, CW could use Completed Kata Honor/ranking everywhere (privilege evaluation, displayed rank, etc.). It might increase amount of cheaters, but it's easier to protect against them than clean up all the poorly authored katas.

And I also like the kazk's idea of having "officially recognized" problems, where author's kata could be promoted to - this possibility itself could be rewarding enough to serve as an incentive for creating well crafted problems.

kazk commented 5 years ago

I'd like to improve gamification in general. The current honor points and kyu/dan ranks are too minimal. Privileges based on them are awkward and are causing problems.

Badges are obvious addition. Badges are not only for bragging rights, it'll be more accurate ways to measure users' contributions and their ability. And badges allow granular control based on metrics that are actually based on some facts (specific achievements/behaviors), e.g. allow X if the user have all {A,B,C}.

But this will be much later.

CliffStamp commented 5 years ago

IMO any authoring efforts, like creating a kata/translation, having your kata solved, etc. should not be rewarded with honor points at all.

Writing kata and translating them are among the highest activities for learning, that is one of the least controversial points in the literature. New coders should be actively encouraged and rewarded for those efforts by an active curation user base who are also rewarded for positive interactions.

Again, assuming you

Then you reward activities which do both.

CliffStamp commented 5 years ago

I'd like to improve gamification in general. The current honor points and kyu/dan ranks are too minimal. Privileges based on them are awkward and are causing problems.

The problem is that what is rewarded is either one of two things

It is obvious that neither one of those is who you want with some of privileges. Coding is known for being full of very high level coders who are anti-social / hostile to the point of being memes so it is kind of obvious that isn't a great way to select curators for example, if anything they are often the exact opposite of what you want from a learning perspective.

There are also a number of people who are very high level curators but are not on the high level (points wise) likely simply because of lack to time to invest (hobovsky). By high level curators I don't simply mean fix kata's, but in general have positive interactions with the user base, especially new users are who are likely to actually increase learning through their interactions.

Johan is another one, much of what I know about Haskell (which isn't much) is due to interactions with him and dramforever. Both of them have not only reviewed things I have written but actually forked translations/solutions and rewritten them to clarify coding issues, show BP, simplify concepts, etc. . Unnamed as well is crazy detailed in curation / code reviews.

Ideally you would want a curation status, and you want curation of major events (like code rewrites, translation approval, etc.) to be in the hands of people like that.