[RFC] Introduce a kata bounty system

[jhoffner]

There are a ton of kata authors on Codewars creating some great content, however we would like to expand the breadth of the type of challenges we support. This would result in Codewars becoming an even better resource for developers to train on different programming concepts.

The idea is to introduce a bounty system that would reward authors who create content that meet certain requirements. Each time a kata is approved, the author would get somewhere between maybe 20 to 500 additional honor, depending on the specific bounty and the approved rank level.

Requirements could be very specific or broad. Some ideas for types of content that would be targeted is:

• Machine Learning
• AI Bots
• React/Angular/Django/etc based challenge
• Different components to a hypothetical billing system
• Utilize MongoDB as part of the challenge
• Specific Libraries/Packages, such as numpy, scikit, TensorFlow, etc
• Async Programming
• Java Streams
• Education: Teaching OOP, immutability in JavaScript, etc.

Each bounty would have its own page. An author would start by creating a kumite off of the bounty page, which would associate the kumite to the bounty. This would allow ideas to be experimented on first, either privately or in the open. The kumite could then be converted into a kata, retaining its relation to the bounty. Once the kata is approved, the author would get the bulk of the bounty, but contributors who helped along the way (via ancestor kumite) would also share a smaller portion.

The bounty page would track all kumite and kata related to the bounty. It would also contain its own discourse page. Discussion could be used by the community to help figure out the best approach for supporting the requirements.

It would be expected that the bounty system would also fuel open source contributions to the CLI project, so that as additional libraries and adjustments are needed in order to support requirements, that the community would feel encouraged to add them themselves. It would be encouraged to reference pull requests on Github from within the discourse page.

Phase 1 of the bounty system would have bounties created and approved by Codewars staff, and perhaps some designated moderators. Phase 2 would expand on the UI to support suggestions by all Codewars members which could automatically be voted on and approved by the community. The bounty amount would possibly be voted on by the community, or use some sort of supply and demand mechanism to calculating it.

┆Issue is synchronized with this Clickup by Unito

[kazk]

It would be expected that the bounty system would also fuel open source contributions to the CLI project, so that as additional libraries and adjustments are needed in order to support requirements, that the community would feel encouraged to add them themselves.

This is great, but don't we need to improve the CLI project to be more flexible first? Language versions and package/library versions are becoming unmanageable. For example, Python 3.6 with numpy doesn't work because it's installed for 3.4.3 (Codewars/codewars-runner-cli#367). The same problem can happen on NodeJS any time (uses one npm for both versions). Docker image sizes will grow and becomes hard to maintain. Also, what if one user's request conflicts with another?

I'm worried that if we start this unprepared, we'll be forced to apply adhoc hacks to meet users' demands.

---

Just to be clear, I think this is a great idea and I'm super excited :)

[jhoffner]

@kazk that's a really great point. We could possibly just try to limit the amount of libraries/packages to only the most common/popular ones. We already support many of these packages, but since no one has made much use of them yet the issues you mentioned haven't been felt for the most part. My goal is to start working on a prototype soon for a new design for the CLI and CodeRunner server, to actually support persistent connections to the same machine throughout the entire lifecycle of taking the challenge - and to support customizing images. This would allow us to install things as needed instead of maintaining everything up-front. So there are some ideas for that as a long term solution but in the meantime if we want to move forward with this we may just have to deal with some maintenance pains.

Some things I'm still not so certain about:

• What is a reasonable bounty amount? Is 500 too much or not enough as the high end?
• What other types of challenges/requirements can you guys think of?
• Is a moderator approval good enough or do we need some sort of "meets requirements" approval vote?

[kazk]

We already support many of these packages, but since no one has made much use of them yet the issues you mentioned haven't been felt for the most part.

I don't think authors know about @config or how to use them.

Can we add packages only if it's tested to work by having minimum working example? Otherwise, uninstall them? For example, Django appears to be supported because it's explicitly installed (python.docker#L45), but it's not tested to work and we can't give directions to potential authors (Codewars/codewars-runner-cli#344).

---

• What is a reasonable bounty amount? Is 500 too much or not enough as the high end?
• Is a moderator approval good enough or do we need some sort of "meets requirements" approval vote?

I think these are hard to answer until we see how things go.

[kazk]

• What other types of challenges/requirements can you guys think of?

I grouped the ones you've listed and added few.

Computer Science

• Theoretical Computer Science
  • Algorithms
  • Data Structures
  • Cryptography
• Applied Computer Science
  • Artificial Intelligence
  • Machine Learning
  • Image Processing
  • Scientific Computing (numpy, scikit, TensorFlow)

Programming

• Paradigms
  • Functional
  • Object-Oriented
• Language Features
• Design Patterns
• Legacy Code
• Implementation
  • Web Apps
  • Frameworks
  • Front-end
  • Back-end
    • database
  • Full-stack
  • Games (I remember you mentioned this during Lua PR)

[dinglemouse2250]

Just to play devil's advocate for a moment...

Some users more equal than others?

Currently all users are equal. Authors proficient in different languages don't really have any advantage over each other - I can create a 5Kyu Java Kata for any topic and be rewarded same as another user who does same in the language of their choice. A bounty introduces an inequity because my knowledge may not include (or my languages may not support) the kinds of things the bounty is on offer for. Advantage for gaining points becomes skewed towards a few. 500 points is too much. It is going to annoy me every time I see somebody got a bounty for a topic I have no knowledge of and so had no chance to participate in.

Some bounties not achievable?

I'm not sure how some of these bounties would be controlled. Let's say there is a bounty for Java Streams. Using streams or not seems more like a choice for a user implementing a Solution versus something that can be a restriction imposed by the Kata task - how worj that work... maybe the bounty should be on the solution? ;-)

Where did the fun go?

Where is the bounty for "Fun Kata"? The is a risk it's all starting to sound more like work than play. If I want to write serious work code then I go to work to do that - and it's not much fun. So I come to CW to write the fun code and enjoy myself.

--

Please don't take this the wrong way. Bounties may turn out to be terrific ... The above counter-arguments are only offered as some food before everybody leaps onto the band-wagon ;-)

[DonaldKellett]

Hmmm ... food for thought ... I'm afraid I'll have to agree with @dinglemouse2250 here. Having a bounty system is fine but the rewards should be limited - for example, if a certain bounty is satisfied upon Kata approval, maybe the user should be able to gain at most 10% or 20% more Honor than what he/she would normally gain for an approved Kata of that Kyu level.

[jhoffner]

@dinglemouse2250 You make great points and this is why I wanted to get some feedback, for issues like these. We definitely don't want to make the system unfair or corrupt it in any way.

To plays devil's advocate to your devil's advocate:

Some users more equal than others?

Great point that needs to be sure it is incorporated into how bounties would be setup. However the point of the system is to learn and there is no reason why someone couldn't learn the skill and then present a kata for it (even if it's an easy one).

Some bounties not achievable? Java Streams...

Yeah I just added Java streams because someone was talking about it the other day and it was fresh in mind. I would see a kata like that based on the idea of incorporating streams, so as long as the problem is not just solvable but well suited for Java streams, then that would suffice. Or it could be teaching about streams, or the underlying concepts (not specific to just the Java Stream implementation) that could then be implemented using Java Streams (or something else in another language). Again, a good point and the type of point that should be considered when deciding what the criteria for a bounty is.

Where did the fun go?

For some developers, they are here to learn not necessarily just have fun with puzzle style kata, but regardless there should be no reason why making a kata fun can't be involved. It's all about creativity. If we can make cryptography fun (which CW has), then we can make anything fun right? In fact I would say that making the kata fun is if not a requirement, at least always a best practice. Otherwise no one will do them and they will stay in beta forever.

[jhoffner]

@DonaldKellett I like the idea of percentages. I would say a minimum would be required, like say 10 - 20 honor, to make it at least enticing to all levels, and then it would be a 10 - 20% bonus on top of whatever rank it ends up getting set as.

[jhoffner]

To kind of follow up some more with the idea of being fun. I think one of the main goals of this proposal is to try to encourage more creativity on the site. Having a fun story around yet another Palindrome kata doesn't usually change the fact that it's a repeated concept. Don't get me wrong, Palindromes are cool and some great kata have been made around them - but maybe we can start to unearth new concepts to start to build challenges around instead of recycling the same core CS ones. Real world concepts that make learning a new technology, package, concept, etc more fun for developers, who could then go and apply that to their career/job.

For example, the best kata I've written (probably the only one that I actually take much pride in) is builder of things. When I first set out to build that kata, I had it in mind to focus on metaprogramming as the core requirement to what I was going to build. Having that requirement to achieve I think forced me to be creative about how I was going to satisfy it in a way that I don't know I would have come up with if I didn't have those limitations to work within.

So anyway thats kind of my hope with this, that bounties inspire a new focus around creativity and pushing the limits on what we can do, as opposed to restricting creativity (which it may seem like because there are specific criteria that must be met).

[smiks]

I like a bounty system idea. I just want to mention one thing about data-mining/machine-learning category. 12 seconds is not enough for more advanced data mining methods and katas. Less time means smaller datasets which can sometimes result in bad model which gives bad predictions. I had this problem with my own kata. I modified testing which slightly avoids this problem but still. Would increasing it to 20-30 seconds be too much? I tried making a data-mining kata (https://www.codewars.com/kata/data-mining-number-1/).

PS: I also noticed that lot of katas are very similar and that codewars lacks katas similar to ACM-ICPC tasks. I've been competing in ACM competitions for 4 years now and I've seen a lot of tasks. I would gladly create katas with such tasks but what if it is too hard and I will not get enough feedbacks? I hope I'm not too annoying :)

[Voileexperiments]

Throwing in more devil's advocate stuff:

Some users more equal than others?

To be fair, I don't think the honor system had been very equal at the first place. There is a massive early bird bonus to like everything, because you can get lots of points from having things that exist (think of depositing some money in the bank that receives an interest); meanwhile there really lacks high-level katas for honor from solving katas. Completing every single kata (including the betas) probably gives you ~40-50k honor, but that involves solving at least 3k katas and it's... not something that's highly enjoyable. Meanwhile, you can easily get the same amount of points by publishing lots of easy 7kyu katas and leave them there for a few months or so. Solution/comment upvote gains for early users are pretty unbalanced as well; they can well reach 5k+, especially since early katas have lots of such votes while newer users struggle to even get 100 (e.g me).

So, if we're going to adopt the bounty system (the above also applies to the dan system), we need to make sure things are still fair and early bird bonus don't dominate everything. Think of the late joiners who have a significant disadvantage because all the bounties were claimed by earlier users! While honor definitely shouldn't be taken too seriously, it should still be taken seriously (and in fact most of us do in some extent :P), so things like this should be accounted for.

As a side note, for the dan system it definitely should be decoupled with the honor itself. I wouldn't want to solve 1k 4kyu+ just to get to 1dan. However, that also means that dan and kyu are becoming orthogonal and separate to each other. In fact, publishing katas is also orthogonal to this system: you can publish like 100 1kyu katas and I don't think count towards your kyu/dan progress (except for the part that you've solved the kata yourself). So there are already like 3 systems that behave very orthogonal to each other, and adding an equally so 4th dimension would aggravate the matter.

Or, maybe we need to revise the whole "honor as a single number" approach. It doesn't seem to survive the more thorough reflections.