search

code-423n4 / org

Code4rena Governance and Discussion
70 stars 17 forks source link

Disclose who is the judge before the contest #124

Open JeffCX opened 11 months ago

JeffCX commented 11 months ago

relevant discussion

https://discord.com/channels/810916927919620096/976603323450941440/1159524993332416622

I don't see disclose who is the judge before the contest can bring bias

GalloDaSballo commented 11 months ago

I don't have an opinion on disclosing the Judge before or after

xuwinnie commented 11 months ago

I agree

Rassska commented 11 months ago

I agree, support it strongly!

CloudEllie commented 11 months ago

Two considerations from a staff perspective:

  1. If the inclination is "This judge is too tough; therefore, I won't bother submitting at all" -- that's not a good outcome for projects' security or the quality of c4 competitions.
  2. If the discrepancies in judging are significant enough to sway participation rates, that seems like a problem that could be addressed via a combination of rule standardization and some kind of judge feedback / rating system, which we've discussed.

Overall this feels like an opportunity to use the "5 Whys" technique and get to the underlying concerns.

bytes032 commented 11 months ago

Two considerations from a staff perspective:

  1. If the inclination is "This judge is too tough; therefore, I won't bother submitting at all" -- that's not a good outcome for projects' security or the quality of c4 competitions.
  2. If the discrepancies in judging are significant enough to sway participation rates, that seems like a problem that could be addressed via a combination of rule standardization and some kind of judge feedback / rating system, which we've discussed.

Overall this feels like an opportunity to use the "5 Whys" technique and get to the underlying concerns.

Simply put, people are requesting this because they'd rather avoid contests that are judged by a specific judge.

CloudEllie commented 11 months ago

Simply put, people are requesting this because they'd rather avoid contests that are judged by a specific judge.

In that case I expect that the underlying need could be better met by instating a judge feedback / rating mechanism, which could benefit everyone IMO.

JeffCX commented 11 months ago

Simply put, people are requesting this because they'd rather avoid contests that are judged by a specific judge.

that is true

In that case I expect that the underlying need could be better met by instating a judge feedback / rating mechanism, which could benefit everyone IMO.

what is the plan to implement that and when

a few proposal has been pending for very long time

https://github.com/code-423n4/org/issues/73

that's not a good outcome for projects' security or the quality of c4 competitions.

who is the judge is only one factor,

the total amount of reward pot is another factor, code complexity, warden's availability and skill level is another factor

difficult to verify whether disclose who is the judge can harm the outcome for projects' security or the quality of c4 competitions.

I'd purpose let backstage warden vote on this proposal

bytes032 commented 11 months ago

Simply put, people are requesting this because they'd rather avoid contests that are judged by a specific judge.

In that case I expect that the underlying need could be better met by instating a judge feedback / rating mechanism, which could benefit everyone IMO.

I couldn't agree more with you.

Rassska commented 11 months ago

This might sound a little bit controversial, but judging inconsistencies are also the big factor here and you can't do anything about it.

gjaldon commented 11 months ago

In that case I expect that the underlying need could be better met by instating a judge feedback / rating mechanism, which could benefit everyone IMO.

@CloudEllie who would be reviewing the judge feedback/rating? I strongly agree with having this implemented. I still think the original proposal by @JeffCX makes sense though. Wardens would not want to participate in contests where judges are unfair and/or overly strict. That is a sign to C4 and to the judge that they need to change their behavior.

I think both are necessary since we currently lack ways to keep judges accountable.

CloudEllie commented 11 months ago

who would be reviewing the judge feedback/rating?

Judges are appointed by a working group; it seems reasonable that they would be the ones to review the data and make recommendations/decisions.