Enhancements in code upload pipeline

[Sanji515]

Project Title: Enhancements in code upload pipeline

Description:

As EvalAI hosts more code-upload challenges and researchers utilizes our modular kubernetes based infrastructure for hosting these challenges, we would like to automate this pipeline as much as possible to enhance user experience. During GSoC 2019, we built this pipeline for evaluating AI model’s code by running it against unseen test environments in real time and this year the plan is to add features like start, stop, restart, delete cluster, etc. so as to give challenge hosts more control over their challenge evaluation cluster. This will not only involve control over the nodes running evaluation but also viewing logs which are being updated in real-time. Finally, the plan is to give challenge hosts the capability to run the evaluation cluster in their cloud by simply plugging in their keys and rest all will be taken care by EvalAI.

Deliverable:

• [ ] Dockerize the code upload challenge submission worker.
• [ ] Create a dashboard to manage challenge workers by the challenge hosts.
• [ ] Add APIs to send cloudwatch logs from the workers to challenge hosts so as to debug or monitor the submissions.
• [ ] Add API’s for challenge hosts to manage (start/stop/restart/delete) their challenge kubernetes cluster after the challenge is approved by the admin.
• [ ] Add CRUD operations for challenge hosts for kubernetes config for the challenge cluster.
• [ ] Setup a pipeline such that a node in kubernetes should only spawn when a submission comes in and shuts down after the evaluation is completed.
• [ ] Setup a pipeline so that a challenge host can add their AWS credentials with the challenge and the evaluation kubernetes cluster is setup onto challenge hosts cloud while the challenge is still hosted on EvalAI.
• [ ] Add a feature to use GCP instead of AWS for setting up cluster
• [ ] As a part of this project we will also focus on improving the CI/CD pipeline so that we can push the changes from GitHub to the production servers seamlessly with minimal supervision.
• [ ] Write robust test cases for the added APIs and scripts.
• [ ] Add documentation for the newly added features.

Mentor: Kartik Verma @vkartik97 , Rishabh Jain @RishabhJain2018

Skills: Docker, Kubernetes, AWS, Django, DRF

Skill Level: Difficult

Get started: Try to fix some issues in EvalAI (note that there are some issues labeled with GSOC-2020)

Tutorials:

• https://docker-curriculum.com/
• https://cloudacademy.com/blog/how-to-use-aws-cli/
• https://kubernetes.io/docs/tutorials/

Important Links:

• EvalAI Website: evalai.cloudcv.org
• EvalAI Github repository: Cloud-CV/EvalAI
• EvalAI Docs: http://evalai.readthedocs.io/en/latest
• GSoC Proposal Template: Cloud-CV/GSoC-Ideas/wiki/GSOC-2020-Proposal-Template
• Gitter Channel: gitter.im/Cloud-CV
• Mailing list: groups.google.com/forum/#!forum/cloudcv

[Ayukha]

@RishabhJain2018 @vkartik97 This sounds exciting. I also would love to work on this project.

[RishabhJain2018]

Thanks, @Ayukha for the interest. Looking forward to your proposal.

[gauravsomani336]

@RishabhJain2018 @vkartik97 this project looks very interesting. from where can I start contributing to this project. I can't seem to understand if I cm good enough to contribute to this project. I am well versed in python. your help would be really appreciated

[abhinav9414]

@RishabhJain2018 @vkartik97 this is interesting. I would like to join this kind of project.

[RishabhJain2018]

Hi @gauravsomani336, Welcome to CloudCV! You can start on this project by setting up the development environment for EvalAI and looking into the issues which you can solve.

[RishabhJain2018]

Hi @abhinav9414, Welcome to CloudCV and thanks a lot for your interest. Looking forward to your contribution and the GSOC proposal.

[yashdusing]

@RishabhJain2018 @vkartik97 This looks amazing. Looking forward to contribute

[RishabhJain2018]

Hi @yashdusing, Thanks a lot for your interest in the project. Looking forward to your GSoC Proposal.

[ankitjain28may]

Hi, I am interested in mentoring this project along with other mentors, Do we have more explanatory docs regarding the infrastructure we currently have, any diagrams so we can have more understanding of the project.

[RishabhJain2018]

Hi @ankitjain28may, Welcome to CloudCV! and thanks for reaching out. Please fill this form for mentoring this year with CloudCV and we will reach out to you in a few days.

[sulabh2000]

@RishabhJain2018 @vkartik97 Great idea...interested in contributing to this.

[MichaelNkrumah]

I'm Scared tackling this project I don't have enough knowlegde on this structure, yet I want to contribute...

[chialin-liu]

@RishabhJain2018 I had the experience of building multiple container apps using docker and developing backend system in django. I am very interested in this project

[Rubix982]

Highly interested in this project, as it relates to my ML background, but on a deployment level as well .Although I'm not familiar with Kubernetes and AWS, it's something I believe I can pick up pretty quickly. Never got to learn them because I never really found a use case for them in my day to day knowledge up till now. Experience wise, I think this would be good start for me. I have familiarity with Django and once wrote some code to deploy a local machine learning project using Django's rest API by following.