Implementing python package and REST APIs for EvalAI

[RishabhJain2018]

Project Title: Implementing python package and new features for EvalAI

Description: a) Currently, if someone wants to make a submission to a challenge on EvalAI then he has to login to EvalAI and upload a file. We want to make this process even easier for researchers who really like their terminal screen. The first main goal of this project is to create a python package for EvalAI which lets the participants to import evalai as a python package and then make submissions through their python script instead of logging in to the website and then doing it.

b) EvalAI uses Django REST Framework (DRF) at the backend. We want to add a lot of new features this summer where we want the student to implement RESTful web services using DRF. We expect the student to write simple and clean REST APIs, unit tests and an exhaustive documentation for each feature that he is building. Please see the deliverables section to get more details about the project.

Deliverables:

1. EvalAI Python Package:

This package should include the following functionalities (but students are strongly encouraged to come up with their ideas/features and should definitely mention in their proposal):

• For Participants:

  • Show all the challenge details (except test annotations file and other private details)
  • Show the number of challenges in which the participant has participated
  • Pretty print leaderboard for each challenge
  • Ability to make submissions using the python package
  • Show the number of overall submissions happened between two dates for each challenge phase of a challenge

• For Hosts:

  • Show the number of overall submissions happened between two dates for each challenge phase of a challenge
  • Check evalai.cloudcv.org for other host related analytics features that can be added here
  • There are a lot of analytics related feature that can be added here. We strongly encourage prospective students to brainstorm and add it in their proposal.

2. Implementing RESTful web services

• Analytics (implement REST APIs and UI for following features):
  • Show how the rank has dropped/increased for a participant in a challenge
  • Pie chart for the status of the submissions for a participant in a particular challenge
  • Graph for the number of submissions happening per day
• Add the functionality to show the challenges details when the user is not logged in
• Create an admin dashboard where EvalAI admin can do following things:
  • Admin can send email to all/some users informing them about a new challenge. Also, setup the email formatting in such a way so that we can add images (or a HTML) in the email content.
• API versioning

3. Implement webhooks for retrieving challenge submissions and results:
   • The aim of this feature to retrieve the results of a challenge from some forked version of EvalAI to the main EvalAI server. This will help the challenge hosts who are running forked EvalAI on their private servers to show their results on the main Evaluation server (i.e evalai.cloudcv.org).
   • Note that there are a lot things to be taken care of like:
     • How will make sure that the data is authentic?
     • How will you make sure that the results are retrieved in real time?
   • Also have a webhook for bots (slack/ messenger / github etc). Triggered when new submission, leaderboard changes, rank increases or decreases etc.

The above mentioned features should be implemented by the end of the GSoC period. Moreover, we expect students to also have implemented improvements that they have proposed.

Mentors: Rishabh Jain @rishabhjain2018, Shiv Baran Singh @spyshiv

Co-Mentor(s): Deshraj Yadav @deshraj

Skills: AngularJS, Python, Django, Django REST Framework, D3.JS, AWS

Skill Level: Difficult/Ambitious

Getting started:

We expect the students to solve at least one (the more the better) from the following list of issues which are super relevant to the project. Here is the list of getting started issues solve (some of these would already be open issues on Github):

• Create a unique link for invitation to the team -- And the person who is inviting gets to see the link only. If he wants, then he can share the link with other future team members.
• Create an API for adding the affiliation to the participants
• On UI, show them dropdown with the existing affiliation in the database
  • Show 'verified' for each of the admin approved affiliations
  • Implement the feature of setting only allowed email domains and only blocked email domains for a particular challenge.
• Modify Challenge creation using zip file API -- unzip the challenge details after it gets approved by EvalAI admin (go through the codebase for more details and ask questions on Gitter).

Recommended: Try to fix some issues (note that there are some issues labeled with GSOC).

Important Links:

• EvalAI Website: https://evalai.cloudcv.org
• EvalAI Github repository: https://github.com/Cloud-CV/EvalAI
• EvalAI Docs: http://evalai.readthedocs.io/en/latest/
• GSoC Proposal Template: https://github.com/Cloud-CV/GSoC-Ideas/wiki/GSOC-2018-Proposal-Template
• Gitter Channel: https://gitter.im/Cloud-CV/
• Mailing list: https://groups.google.com/forum/#!forum/cloudcv

[guyandtheworld]

@deshraj @RishabhJain2018 This sounds exciting! Wouldn't challenge creation using UI also be a good project for Summer of Code? I also would love to work on the EvalAI Python Package. :smile:

[deshraj]

Hi @isht3, glad to hear that the project sounds exciting. Thanks for your interest in this project. :)

Wouldn't challenge creation using UI also be a good project for Summer of Code?

We already have a Github Pull Request for this feature but this is in review.

[meshubhama01]

sounds exciting... I have few suggestions though. 1.for participants- they can make their groups too for discussion 2.for host - 1.no. of the wrong submission should be recorded for each participant so that host can send better-personalized email to participants. 2.can we afford to do more analytics(data) in order to have a more personalized feed for each participant.. fo e.g. the type of error they generally do or the type of problem they always complete or the syntax error they tend to repeat....etc..?

p.s- sorry if above sound silly as this is my first time. any guidance or help is welcome

[deshraj]

Thanks @meshubhama01 for the suggestions. I did really like the following two points:

2.for host - 1.no. of the wrong submission should be recorded for each participant so that host can send better-personalized email to participants. 2.can we afford to do more analytics(data) in order to have a more personalized feed for each participant.. fo e.g. the type of error they generally do or the type of problem they always complete or the syntax error they tend to repeat....etc..?

Since the project is still evolving, so there are plenty of things that we can add. If you are planning to apply for this project idea, we would love to have see list of some new features in the GSoC Proposal.

Last, the suggestions are not at all silly and I hope you enjoy working on this project. Let us know if you have questions/doubts.

[meshubhama01]

as this is my first time and I don't have friends in this area I am quite confused, how to apply for this project. any help?

[deshraj]

I would recommend you to read the GSoC Manual here https://google.github.io/gsocguides/student/. This will give you a better idea how to start.

[gcmurillo]

I'm really interested in this project, in the API Rest especially, in the the coming days I will send my proposal for GSoC. I hope not so late :(

[RishabhJain2018]

Hi @gcmurillo , Thanks for your interest in this project. :) No, you are not at all late for submitting the proposal. Looking forward to see your proposal soon!

[saisusrith]

Hi @deshraj, I lived the key idea of the project and the way it saves a lot of time. I would love to work on the project as a part of summer of code.

[advait-net]

hi @deshraj, i like the idea as probably this will make a common platform. I would love to work on this project as a part of summer of code. I hope i can submit the proposal by 21:30 today.

[RishabhJain2018]

Closing this issue since GSOC 2018 is over.