topcoderinc / cab

9 stars 3 forks source link

Including handle in the submission #46

Open lstkz opened 7 years ago

lstkz commented 7 years ago

The review process is anonymous by default, and members shouldn't reveal their identity anywhere. However, members can be recognized in an easy way by:

birdofpreyru commented 7 years ago

It makes no sense. Because if a competitor and a reviewer want to hack the rules, the competitor can just privately email the ID of his submission to the reviewer. Thus, you may close any other ways for a reviewer to figure out the identity of competitors, but you'll never be able to prevent this simple way to hack the system.

The only way to deal with this is: if a review is not fair, other competitors should bring it up to the co-pilots/managers/community, the issue should be considered in public, and all involved in cheating and/or unfair judgement should be strictly punished.

ThomasKranitsas commented 7 years ago

Totally agree with @birdofpreyru !

lstkz commented 7 years ago

It's not related to cheating because we can't prevent it.

See last post https://apps.topcoder.com/forums/?module=Thread&threadID=623369&start=0&mc=8#1021109

But for instance if I knew I was reviewing a design by Pops I would be expecting it to score very highly; if my review score was less than 95 I'd be worried I had made a mistake. And if I knew the design was by a guy with a rating of 700, I would be expecting it to be poor.

birdofpreyru commented 7 years ago

Anyway, I don't like this because:

  1. Demands extra efforts from a competitor to hide his identity (e.g. I use Linux, and in all my demo/verification videos you can see my Terminal Console with my username in the path, do you want me to hide it somehow? I have no idea, how to do it, it sure demands extra efforts). Similar thing applies to usernames in various services, etc.

  2. using TC API submissions are not hidden on the challenge page

These demands some changes to the codebase, i.e. $$ to be paid and work to be done.

Thus, needs efforts from all parts, while the impact is not obvious for me, I believe in 95% of cases, any initial bias does not change the outcome of reviews, while the rest 5% are probably closer to cheating, and won't be prevented by these efforts.

lstkz commented 7 years ago

discuss here https://apps.topcoder.com/forums/?module=Thread&threadID=897631&start=0&mc=1#2178578

hokienick commented 7 years ago

We need to discuss a way for us to create a process/rules/policy to keep anonymity