search

Cloud-CV / EvalAI

:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI
https://eval.ai
Other
1.77k stars 787 forks source link

Add feature to disable private submission #4419

Open Harshit28j opened 2 months ago

Harshit28j commented 2 months ago

image

codecov-commenter commented 2 months ago

:warning: Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 15.38462% with 22 lines in your changes missing coverage. Please review.

Project coverage is 68.91%. Comparing base (96968d6) to head (9d8b8fc). Report is 1112 commits behind head on master.

Files with missing lines Patch % Lines
frontend/src/js/controllers/challengeCtrl.js 15.38% 22 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #4419 +/- ## ========================================== - Coverage 72.93% 68.91% -4.02% ========================================== Files 83 20 -63 Lines 5368 3600 -1768 ========================================== - Hits 3915 2481 -1434 + Misses 1453 1119 -334 ``` | [Files with missing lines](https://app.codecov.io/gh/Cloud-CV/EvalAI/pull/4419?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) | Coverage Δ | | |---|---|---| | [frontend/src/js/controllers/challengeCtrl.js](https://app.codecov.io/gh/Cloud-CV/EvalAI/pull/4419?src=pr&el=tree&filepath=frontend%2Fsrc%2Fjs%2Fcontrollers%2FchallengeCtrl.js&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-ZnJvbnRlbmQvc3JjL2pzL2NvbnRyb2xsZXJzL2NoYWxsZW5nZUN0cmwuanM=) | `60.57% <15.38%> (-13.13%)` | :arrow_down: | ... and [63 files with indirect coverage changes](https://app.codecov.io/gh/Cloud-CV/EvalAI/pull/4419/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) | [Files with missing lines](https://app.codecov.io/gh/Cloud-CV/EvalAI/pull/4419?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) | Coverage Δ | | |---|---|---| | [frontend/src/js/controllers/challengeCtrl.js](https://app.codecov.io/gh/Cloud-CV/EvalAI/pull/4419?src=pr&el=tree&filepath=frontend%2Fsrc%2Fjs%2Fcontrollers%2FchallengeCtrl.js&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-ZnJvbnRlbmQvc3JjL2pzL2NvbnRyb2xsZXJzL2NoYWxsZW5nZUN0cmwuanM=) | `60.57% <15.38%> (-13.13%)` | :arrow_down: | ... and [63 files with indirect coverage changes](https://app.codecov.io/gh/Cloud-CV/EvalAI/pull/4419/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) ------ [Continue to review full report in Codecov by Sentry](https://app.codecov.io/gh/Cloud-CV/EvalAI/pull/4419?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://app.codecov.io/gh/Cloud-CV/EvalAI/pull/4419?dropdown=coverage&src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None). Last update [11ea23e...9d8b8fc](https://app.codecov.io/gh/Cloud-CV/EvalAI/pull/4419?dropdown=coverage&src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None).
Harshit28j commented 2 months ago

image image

Harshit28j commented 2 months ago

@gchhablani I have also tested this by hosting a challenge from EvalAI-Starters using challenge_config.yaml, adding the disable_private_submission attribute to a challenge, and also tested by upgrading its value.

gchhablani commented 2 months ago

@Harshit28j Can you share screenshot of the my submissions page too?

Harshit28j commented 2 months ago

@gchhablani here:

  1. When the value of disable_private_submission = true image

  2. When the value of disable_private_submission = false image

SSHurtado commented 1 month ago

Hello all,

We’re hosting a challenge at the end of this month, and this feature would make a big difference. In past editions, everyone kept their submissions private, which left the leaderboard outdated and discouraged competition.

Disabling private submissions will keep the leaderboard active and make the event more engaging. Could you help speed up the review of this pull request? It’s exactly what we need for this challenge!

Thanks a lot!