Static code upload challenge evaluation and enhancements in GitHub based challenge creation

[Ram81]

Project Title: Static code upload challenge evaluation and enhancements in GitHub based challenge creation

Description:

EvalAI is a platform to host and participate in AI challenges around the globe. To a challenge host, reproducibility of submission results and privacy of the test data are the main concerns. Towards this, the idea is to allow users to submit a docker image for their models and evaluate them on static datasets. In order to achieve this we want to build a pipeline which will use the dockerized models and run it on kubernetes based infrastructure with stored test annotations and report the results on the EvalAI leaderboard. Another part of the project is to streamline our challenge creation pipeline. Last year we added support for github based challenge creation which allows challenge hosts to use a private github repository to create and manage updates in a challenge. The goal for this year is to support bi-directional updates for challenges created using github. This feature will allow hosts to sync changes from EvalAI UI to their challenge github repository. The goal is to enhance the challenge creation experience for challenge hosts involving minimal support from the EvalAI team.

Deliverable:

• [ ] Add is_static_dataset_docker_based_challenge field in Challenge model and accept the parameter as part of challenge config in Challenge creation API
• [ ] Add submission_input_file field in submission model for static code upload submission.
• [ ] Add is_static_code_upload_submission flag in SQS queue message for submissions made to static code upload challenge. This flag will allow us to use same queue for 2 step evaluation
• [ ] Add static code upload challenge test dataset download function in code upload worker.
• [ ] Add methods in the code upload worker to spawn a single Kubernetes pod for static code upload evaluation.
• [ ] Create single kubernetes container using the submission docker image
• [ ] Mount test dataset volume on the container at a fixed path
• [ ] Add submission_pk as environment variable
• [ ] Add Kubernetes pre stop hook to submit the generated output file submission.json or submission.csv to EvalAI as a submission using submission_pk environment variable of the kubernetes pod.
• [ ] Add support for a job (it can be a cron job) to monitor evaluation time and kill the job if it exceeds the submission time limit.
• [ ] Add support to report error logs from job pods to submission stderr file
• [ ] Write tests for the static code upload challenge worker
• [ ] Add github token model field in challenge model to sync challenge changes to github repo
• [ ] Add post save hooks to generate challenge details update and submit as a pull request to the hosts github repo using Github APIs.
• [ ] Write tests for challenge details sync post hook methods

Mentor: Khalid Riyaz (@KhalidRmb), Ram Ramrakhya (@Ram81), Rishabh Jain (@RishabhJain2018)

Skills: Python, Django, Kubernetes, Docker

Skill Level: Hard

Get started: Try to fix some issues in EvalAI (note that there are some issues labeled with GSOC-2020)

Important Links:

• EvalAI Website: eval.ai
• EvalAI Github repository: Cloud-CV/EvalAI
• EvalAI-ngx Github repository: Cloud-CV/EvalAI-ngx
• EvalAI Docs: http://evalai.readthedocs.io/en/latest
• GSoC Proposal Template: Cloud-CV/GSoC-Ideas/wiki/GSOC-2020-Proposal-Template
• Gitter Channel: gitter.im/Cloud-CV
• Mailing list: groups.google.com/forum/#!forum/cloudcv

[savish28]

Hey @Ram81, I am intersted in this project and would love to work on issues regarding the same. Most of the issues mentioned Gsoc are assigned so could I start working of making pull request for some of the initial deliverables for this?

[Cenrax]

Hey @Ram81, I am interested in this project. I have experience in working with Kubernetes in AWS and Azure and also managing of CNI. I want to contribute in this project, can you please guide me on what to do next?

[akshat-crypto]

Hey @Ram81, I want to contribute in this project, I have worked on Kubernetes, Aws, docker, Jenkins in the past few months and i would like to gain more knowledge in this domain. Pls guide me on how to proceed further

[Anubhav1107]

Hello @Ram81, @KhalidRmb @RishabhJain2018, I am interested in this project and want to work on it. I normally work in Python and have tinkered with docker and have knowledge in AWS, I will learn Kubernetes as the Project goes on. Can you please guide me on where to start this, thanks !

[Garoradua]

Hey! @Ram81, @KhalidRmb @RishabhJain2018, I am a very keen learner of this project. Because I generally worked on Docker from Basic launch its image to the advance making apache webserver manually and fully automated from making scripts on Dockerhub. I am doing automation on the docker like using MySQL database to launch fully working our own WordPress website. so, please involve and guide me in this project.

[AmnaFish]

Not something that I have experience in, but I am super eager to work on this project. @Ram81, @KhalidRmb @RishabhJain2018 I am pretty sure that with your guidance Ill make it through this project. I have worked with Python, and as for Docker, Kubernetes and every other new tech that will be required, I'll learn and grasp them as soon as possible. This project will give me a ton to learn and I believe I'll make it to the end successfully. Cant wait to get started!

[kshitijdhara]

Hi, @Ram81, @KhalidRmb, and @RishabhJain2018 I want to contribute to this project, I have worked on Kubernetes, AWS, GCP, Docker, Jenkins, and Cloud Code in the past few months and I would like to gain more knowledge in this domain. Please guide me on how to proceed further.

[Saty248]

Hi, @Ram81 , @KhalidRmb , and @RishabhJain2018 .. this is Sayantan mondal. I am a 2nd year btech student . I was searching for open source project to contribute . I found this project to be interesting . I did a devops course in Udemy where I learnt about aws , docker , kubernetes and I also made from websites using python and django .. so how should I begin ? can you guys please guide me through this? I am fast learner and I won't dissapoint you guys

[huskhi]

Hey @Ram81 @RishabhJain2018 @KhalidRmb, I'm a 2nd year B.Tech student. I' m new to open source but I'm motivated and interested to work on this project. Please guide me on how can I get started with this?