Streamlining challenge creation on EvalAI

[Sanji515]

Project Title: Streamlining challenge creation on EvalAI

Description:

EvalAI is a platform to host and participate in AI challenges around the globe. To host a challenge, challenge creation is one of the core features which is utilized by challenge hosts to create AI challenges. The idea is to use private GitHub repositories to host the challenge files instead of the zip file. The next step is to build and integrate a continuous deployment pipeline with GitHub so that for every new commit in the challenge repository, the changes are automatically reflected on the UI. We will also build support for tests so that new commits are fully tested before they are pushed to the live challenge hosted on EvalAI. The goal is to enhance the challenge creation experience for challenge hosts and set up a challenge on EvalAI by the challenge hosts involving minimal human effort from the EvalAI team.

Deliverable:

• [ ] Setup a pipeline to integrate a continuous deployment system for creating a challenge on EvalAI.
• [ ] Write tests to validate the challenge configuration.
• [ ] Integrate the tests with the continuous deployment system to run them whenever a challenge host commits a change to github.
• [ ] Setup a mechanism to authenticate evalai-user with private github repository.
• [ ] Add APIs for updating details on the backend database.
• [ ] Once the tests pass, configure autobots on github which will push the changes using APIs on EvalAI database.
• [ ] Add feature to display submission worker logs on the UI so that debugging is simple for challenge hosts.
• [ ] Write robust test cases for the code.
• [ ] Add the documentation so that it is easy for challenge hosts to set up.

Mentor: Ram @Ram81 , Rishabh Jain @RishabhJain2018

Skills: Python, Django, Django Rest Framework, Knowledge of any CI/CD tool.

Skill Level: Ambitious

Get started: Try to fix some issues in EvalAI (note that there are some issues labeled with GSOC-2020)

Tutorials:

• https://www.django-rest-framework.org/tutorial/quickstart/
• https://docs.djangoproject.com/en/3.0/intro/tutorial01/
• https://docs.travis-ci.com/user/tutorial/

Important Links:

• EvalAI Website: evalai.cloudcv.org
• EvalAI Github repository: Cloud-CV/EvalAI
• EvalAI Docs: http://evalai.readthedocs.io/en/latest
• GSoC Proposal Template: Cloud-CV/GSoC-Ideas/wiki/GSOC-2020-Proposal-Template
• Gitter Channel: gitter.im/Cloud-CV
• Mailing list: groups.google.com/forum/#!forum/cloudcv

[spectre1811]

Hello @Ram81! I'd love to work on this proposal as a part of GSOC'20.

[RishabhJain2018]

Hi @spectre1811, Thanks a lot for your interest in the project. Looking forward to your proposal.

[madrix01]

Hello @Ram81 , I would like to contribute to the CloudCV as a part of GSOC'20 , I have fair knowledge about Django, Rest-apis and python. Can you help me getting started?

[mrinalwahal]

Hey @Ram81 and @RishabhJain2018 ,

I've tried reaching you through emails, however, I failed at it. Therefore, writing my queries here itself. I'm writing this to inform that I'm quite interested in this GSoC project.

I've worked on a quite similar GSoC project last year as well for INCF, Germany, wherein the end goal was more or less similar to this one, and since I quite enjoyed working on that project, I'm choosing to apply for this project this year.

Here's what I'm gonna do now:

• I've started launching EvalAI docker containers and getting to know around the tool.
• Signed up on the hosted online version as well. Username: @wahal

Here's what I want to know as of now:

1. Do you have any specific requirements for a particular CI/CD tool, or can we simply standardize GitHub workflows? They have a considerably fine API as well to integrate. I've worked with Drone CI, GitLab CI/CD and Github Workflows, I guess ramping up on new CI tools, if required, shouldn't be a problem.
2. What is a standard challenge configuration that needs to be tested? Can you give me some examples?
3. This project will be deployed as a fresh micro-service (ideally) or integrated into the existing backend? I'm fine either way but will have to plan the work accordingly.

Authentication of users and pushing updates to DB seem more straightforward, so I'll skip those as of now.

Do let me know if you want me to look for something really specific or even if you have any experimental ideas to improve the project. :)

[Rubix982]

Hi @Ram81 and @RishabhJain2018 , Sounds pretty cool, yet ambitious. I'm willing to take part in this if I do not qualify for the 'code pipeline' idea. As for my skill level, I can take on Python, Django. CI/CD tools is something I have yet to learn. Should definitely get to learning that.