search

code-423n4 / org

Code4rena Governance and Discussion
70 stars 17 forks source link

Discussion about Hunter/Gatherer Role #158

Open bytes032 opened 7 months ago

bytes032 commented 7 months ago

Since there will be plenty of questions/discussion, I am frontrunning these by opening an issue here to have all the comms in a single place.

https://code4rena.com/blog/code4rena-spring-update-2024

MiloTruck commented 7 months ago

And as further incentive, we’re introducing two new bonuses within the HM pool:

  • Hunter bonus: 10% of the HM pool will be awarded to the warden or team who identities the greatest number of unique HMs.
  • Gatherer bonus: 10% of the HM pool will be awarded to the warden or team who identifies the greatest number of valid HMs.

Both bonuses weigh Highs more heavily than Mediums, similarly to C4’s typical awarding mechanism.

Some clarifications I had that would be good to make public:

  1. What if two wardens tie for 1st place?
  2. Is there a cap on the bonus? I can't imagine 20% of the pool going to individual wardens in a 1M contest.
  3. What's the motivation behind having a hunter bonus? The current awarding formula already heavily rewards unique HMs, there doesn't seem to be a need to incentivize them further.
  4. Any reason why the bonus is only for the top warden, instead of the top X wardens? (eg. top 3 wardens split the 10% bonus)
dontonka commented 7 months ago

What problem are we trying to solve with those roles? In the end, it's always few top auditors that win the entire pot, either because they find a lot of findings and/or because they are unique findings. Essentially, those 2 roles will magnify this situation even further.

If the objective we are trying to solve is to create retention of top auditors on C4 (similar to what Sherlock do with their Lead Watson which get a fixed pay), that is definitely working into that direction.

CloudEllie commented 7 months ago

@MiloTruck

  • What if two wardens tie for 1st place?

They would split the awards for 1st and 2nd place, evenly. (This is how the existing award curve works.)

  • Is there a cap on the bonus? I can't imagine 20% of the pool going to individual wardens in a 1M contest.

Good question; we haven't discussed this, but we'll consider it.

  • What's the motivation behind having a hunter bonus? The current awarding formula already heavily rewards unique HMs, there doesn't seem to be a need to incentivize them further.
  • Any reason why the bonus is only for the top warden, instead of the top X wardens? (eg. top 3 wardens split the 10% bonus)

Both of these questions are a bit philosophical in nature -- but the north star at C4 is always to secure projects. The objective here is to add further incentives for wardens to provide projects with as much coverage as possible, and to hunt for the hardest-to-find vulnerabilities.

Top performers at C4 will continue to be awarded well. And it's worth underlining that the HM pool is increasing as a percentage of overall awards.

0xA5DF commented 7 months ago

the greatest number of unique HMs.

How do we measure that? Do we use the same formula as the awarding formula (meaning the warden with the biggest HMs payout would also get this bonus)?

the greatest number of valid HMs.

How do we measure that? Do highs and meds have equal points for this, or do we have a 10/3 ratio for H/M like in the awarding formula? (Edit: the post says it does use the 10/3 ratio like in the awarding formula)

CloudEllie commented 7 months ago

How do we measure that? Do we use the same formula as the awarding formula (meaning the warden with the biggest HMs payout would also get this bonus)?

@0xA5DF We'll get the awarding formula added to our docs ASAP.

0xA5DF commented 7 months ago

I'd like to suggest an optional improvement for this: 2nd & 3rd place would get part of the bonus, but only if they get close enough to the 1st place (e.g. 80-90% of the points of 1st place). This seems fair - if you won by a big margin you get the entire bonus, if only by a small margin you share it with 2nd & 3rd. This also sets the right motivations - the motivates the first place not just to win but to win by a big margin.

kamensec commented 5 months ago

If theres only 1 solo finding thats a medium and 3 highs all duplicates, how does that 10% bonus for solos get distributed?

Seems like this is just going to make judging more complicated, which is still the bottle neck of any audit. Wouldn't this just be incentivising people to argue that certain issues are duplicates or find basically identical edge cases in related issues and submit them separately hoping one can be argued as a unique solo?

We have gone from just incentivising people to find issues of certain severity, which already has unanswerable issues where people try to game the system by inflating or deflating severity, to now a situation where we will have people arguing over how 'solo' their issue is.