Evaluating submission code in Docker containers

[RishabhJain2018]

Project Title: Evaluating submission code in Docker containers on EvalAI

Description:

The rise of reinforcement learning based problems or any problem which requires that an agent must interact with an environment introduces additional challenges for benchmarking. In contrast to the supervised learning setting where performance is measured by evaluating on a static test set, it is less straightforward to measure generalization performance of these agents in the context of the interactions with the environment. Evaluating these agents involves running the associated code on a collection of unseen environments that constitutes a hidden test set for such a scenario. The goal of this project is to set up a robust pipeline for uploading prediction code in the form of Docker containers (as opposed to test prediction file) that will be evaluated on remote machines and the results will be displayed on the leaderboard.

Deliverable:

• [ ] Setup the pipeline for uploading the image on ECR registry
• [ ] Add feature to display the image uploaded by a user on UI
• [ ] Setup the pipeline to spawn a GPU based AWS machine automatically and start the evaluation of the uploaded docker image
• [ ] Pipeline to send the evaluation results back to EvalAI and kill the spawned instance
• [ ] Display the stderr, stdout, results corresponding to the submission on EvalAI
• [ ] Setup a dummy environment based challenge on EvalAI
• [ ] Optimize the submission evaluation by evaluating it on the cluster of machines
• [ ] Write robust test cases for the code
• [ ] Write Documentation for setting it up on server

The aforementioned features should be implemented by the end of the GSoC period. Moreover, we expect students to also have implemented improvements that they have proposed.

Mentors: Deepesh Pathak @fristonio , Rishabh Jain @RishabhJain2018 , Deshraj @deshraj

Skills Required: Docker, AWS-CLI, Django, DRF [Knowledge of Reinforcement Learning is not necessary.]

Skill Level: Difficult/Ambitious

Get started: Try to fix some issues in EvalAI (note that there are some issues labeled with GSOC-2019)

Tutorials:

a) Docker b) AWS-CLI c) Buildpack d) jupyter-repo2docker

Important Links:

• EvalAI Website: evalai.cloudcv.org
• EvalAI Github repository: Cloud-CV/EvalAI
• EvalAI Docs: http://evalai.readthedocs.io/en/latest
• GSoC Proposal Template: Cloud-CV/GSoC-Ideas/wiki/GSOC-2019-Proposal-Template
• Gitter Channel: gitter.im/Cloud-CV
• Mailing list: groups.google.com/forum/#!forum/cloudcv

[krtkvrm]

@deshraj @RishabhJain2018 @fristonio
This sounds exciting! I also would love to work on this project. 😄

[RishabhJain2018]

Thanks for your interest @vkartik97! Looking forward to your GSOC Proposal :)

[adi0509]

I am also interested, but have knowledge of python and some basics of docker. I am a beginner but interested in this.

[fristonio]

Thanks for your interest in the project @adi0509 The project is mostly around docker so I would recommend you to familiarize yourself with AWS and docker ecosystem. Here are some links for the same.

• https://labs.play-with-docker.com/
• https://github.com/open-guides/og-aws

Let me know if you have some specific doubts.

[adi0509]

Thanks for your interest in the project @adi0509 The project is mostly around docker so I would recommend you to familiarize yourself with AWS and docker ecosystem. Here are some links for the same.

• https://labs.play-with-docker.com/
• https://github.com/open-guides/og-aws

Let me know if you have some specific doubts.

@fristonio Thanks for your suggestion. I will surely follow it.

[Helinia]

@deshraj @RishabhJain2018 @fristonio Thank you all for your kind help! I am interested in this topic and will look into it. Cheers!

[RishabhJain2018]

Hi @Helinia, Thanks for your interest. Looking forward to your proposal.

[chidauri]

Hi @RishabhJain2018 ,i am interested in this topic want to contribute, i do not fulfill all the prerequisites though :(

[yashbhutoria]

Hi, @RishabhJain2018 I have good knowledge of Azure Enterprise Cloud and Python and Docker. I Believe I can contribute to this one.

[sanjaymaniam]

@fristonio @RishabhJain2018 Hi, this seems pretty interesting! Just a quick question. What level of comfort with Django would this demand? And on that note, for someone fairly proficient in Python, what would be an efficient way to pick Django up?

[krritik]

This looks cool and I would love to work on this project.

[Aaryaman09]

Hi Everyone, i liked the project and i'm very much excited to contribute in it.I have intermediate knowledge of python and beginner of docker. @RishabhJain2018 Please provide me with the link (if required) so i can catch up to the pace.Thank you

[RishabhJain2018]

Hi @yashbhutoria @cyphysan @krritik @Aaryaman09 Thanks for your interest and looking forward to your GSoC proposals.

[RishabhJain2018]

Just a quick question. What level of comfort with Django would this demand? And on that note, for someone fairly proficient in Python, what would be an efficient way to pick Django up?

@cyphysan We don't demand a student to be an expert but should have a basic understanding of the concepts and workflow in Django. In order to get started with Django, please refer to the documentation and its tutorial app.

[RishabhJain2018]

please provide me with the link (if required) so I can catch up to the pace.Thank you

@Aaryaman09 What link are you talking about? Please join the gitter channel in order to stay updated with the project.

[mani1soni]

Hey @RishabhJain2018 i liked the project. I am intermediate in docker,aws and python. i would love to work on this project.

[fristonio]

Thanks for your interest @mani1soni looking forward to your proposal.

[vardotexe]

Hey, @RishabhJain2018 I really liked the project. I did work on docker and python but intermediate in aws. I would Love to contribute.

[9916103020]

Hey,@RishabhJain2018 I really liked the project. I have done mt last project on Django, python and this time I am working on aws but I am intermediate in docker. I would love to contribute.

[theodore3131]

Hi, @RishabhJain2018 I am interested in this project. I have worked on CI/CD pipeline with Docker during my internship in RedHat and I am familiar with python. (But starter in Django). I have worked on some research in deep learning and I am happy to contribute.

[RishabhJain2018]

Hey @NagpalVarun, @9916103020, @theodore3131 Thanks for your interest in the project. I'm looking forward to your GSoC proposals. Please feel free to share with us so that we can review before the final submission to google.

[yashdusing]

@RishabhJain2018 Why does the participant provide the docker container for evaluating the code? I mean, the submission of model file could have a predict method which could be used instead and the evaluation code docker container can be provided by the challenge host (Also, the context of user in the description is related to challenge host or participant ?)

[deshraj]

@yashdusing please read section 6.2 of this paper https://arxiv.org/pdf/1902.03570.pdf. This will clear your doubt. :)

[yashdusing]

So, the participant submits the same docker image with prediction code which he used for training (with little changes but the underlying API for simulation of environments stays the same. The only change is in the environment files)? If so, I wrongly assumed that the challenge host specifies the docker image. But, the latter approach could actually abstract the details of the implementation from the participant and make the challenge host implement the docker script. This could be easier for participants because most of the participants need not know of docker (it's better to assume that the challenge host knows of docker than the participant). This way, the participant focuses on the model snapshot and the challenge host focuses on creating the docker image (Also, we have to trust the participant to not do something malicious in the former approach while in the latter approach all we are getting from the participant is the model file) @deshraj @RishabhJain2018 can you please tell me what I’ve written is right or wrong

[cristomathew]

Hi, Im intrested in this project, I know some basics of Aws(worked on it during my first internship at TCS) , I am familiar with python and kinda starting with Django

[zduymz]

I would love to join in this project. Could someone show me where can i start digging in because this is my first time to GSOC.

[9916103020]

@zduymz Firstly become familiar with the whole idea of the project using this link:https://arxiv.org/pdf/1902.03570.pdf and then see the template of the GSoC 2019 link: https://github.com/Cloud-CV/GSoC-Ideas/wiki/GSOC-2019-Proposal-Template

[9916103020]

Hello Sir, Divyansh Here @deshraj @RishabhJain2018 @Ram81 @fristonio Could someone please review my proposal? Only a few days are left so it would be really helpful if someone could give feedbacks. Thanks in advance.

[9916103020]

Sir, would you please reply?

[seeni-dev]

I am starting to work on this. If anyone is already working on this, ping me.

[spectre1811]

@RishabhJain2018 is the issue still open for GSOC'20 or we'll have to wait for some other issues?

[RishabhJain2018]

Hi @spectre1811 , Thanks a lot for your interest in the organization. We will be opening new issues for this year's GSoC.