Evaluation Infrastructure Optimization

[gchhablani]

Project Title: Evaluation Infrastructure Optimization

Description: This project aims to enhance EvalAI's functionalities through automating large worker deployments in AWS, adding relevant features for efficient challenge management and also writing a robust and efficient test suite. The focus of the project is two-fold:

• To automate large worker deployment processes on EvalAI using AWS EC2 instances or spot instances, make challenge management seamless and less reliant on the admins. We want to reduce the dependency of challenge hosts on EvalAI admins.
• To make EvalAI more reliable and error-free by incorporating tests for different frontend and backend components. Having robust tests prevents making code-breaking changes to the codebase. This task will include adding unit tests for the API suite, prediction upload evaluation workers, code upload evaluation workers (on EKS), and integration tests for end-to-end testing of all components.

Deliverable:

• Infrastructure Optimization
  • [ ] Implement auto-scaling for the challenges being hosted on independent EC2 instances.
  • [ ] Add feature to allow hosts to use custom docker images based on EvalAI worker images for submission evaluation.
  • [ ] Add features to create a forum for the challenge during challenge creation.
  • [ ] Add feature in GitHub-based challenge to manage multiple challenge configs over the years for the same challenge.
• Building a Test Suite
  • [ ] Add tests for GitHub-based challenge creation on EvalAI.
  • [ ] Add tests for code upload evaluation workers, including unit tests for individual components and integration tests for the worker.
  • [ ] Add tests for code-upload challenge evaluation and static code upload challenge evaluation pipelines.
  • [ ] Add unit tests of Kubernetes components using mock.
  • [ ] Add tests for frontend components:
  • Challenge Page
  • Make submission page
  • My submissions and All submissions page
  • Settings tab
  • Dashboard with tabs for all challenges, hosted challenges, and participated challenges.

Mentor: Gunjan Chhablani, Ram Ramrakhya

Skills: Python, Django, AngularJS, AWS

Skill Level: Medium

Get started: Try to fix some issues in EvalAI (note that there are some issues labeled with GSOC-2023)

Important Links:

• EvalAI Website: eval.ai
• EvalAI Github repository: Cloud-CV/EvalAI
• EvalAI Docs: http://evalai.readthedocs.io/en/latest
• GSoC Proposal Template: Cloud-CV/GSoC-Ideas/wiki/GSOC-2020-Proposal-Template
• Gitter Channel: gitter.im/Cloud-CV
• Mailing list: groups.google.com/forum/#!forum/cloudcv

[thesakshidiggikar]

@Ram81 Hey I Would like to be part of this Project mentioned in the idea list, can you guide me please also the provided gitter channel show that it's the site is not found could you please provide me the discussion tab please?

[marinicadenisandrei]

Hi,

I am very interested in participating for the "Evaluation Infrastructure Optimization" program offered by GSOC 2023.

I am very passionate about this field, I currently work for companies that use the same technologies and I am also a master's student with a profile focused on these technologies. I would like to interact with a very well developed team, from which I can learn a lot of things and to be able to interact and implement all the knowledge to contribute to this project.

I am very passionate, I want to devote myself a lot to the project and to be able to participate in this GSOC 2023 program.

[Kanika637]

Hey, I would like to be a part of the project. But I can't see any issues in EvalAI with the label GSOC-2023.

[Nishant123455]

hello I am Nishant Mishra and I currently a student at Manipal University Jaipur India pursing Information Technology BTech my skills are :- Python django React.js c/c++ git

I am interested to contribute on this project Can you guide me for that And what to do to get into this project where to discuss about the project (note ; --channel link is not working)

[gchhablani]

@Nishant123455 @Kanika637 @marinicadenisandrei @thesakshidiggikar Can you please join this Slack: https://join.slack.com/t/evalai-gsoc/shared_invite/zt-1rkm0m8pw-TBkjRjx~_GyIvMVlF5iDHg?

We will update the link on the GSoC page. Also, please join #gsoc-23-interested-participants channel. I will be putting up more information soon.

[Raghucharan16]

Hi everyone, Myself N.V.RaghuCharan CSE Undergraduate.

I'm interested in this project as I have prior knowledge in the skill set required for the project. I'm always a fan of opensource and want to contribute my part to it. I have decent knowledge in python, ML, Cloud service Providers (AWS, Azure, GCP). very excited to kickstart my open source journey through this project.

[gchhablani]

@Raghucharan16 Welcome to CloudCV-Community. Please join the following workspace: https://join.slack.com/t/evalai-gsoc/shared_invite/zt-1rkm0m8pw-TBkjRjx~_GyIvMVlF5iDHg

[sujitdhar014]

Hi, I’m Sujit Sutradhar, B.Tech. (CSE) pre-final year student, and I am very interested in participating in this project. I want to become a part of the Cloud-CV team so I can learn from and contribute to this project. Could you assist me with that?

[Shubhamraut01]

Hello Everyone ,

I am in last year of masters in Computer Science from Fergusson College , Pune . I have 7 month experience working in Tech Industry . I have underwent AWS training and I will be appearing for certification soon .I can perform AWS tasks with ease. I have experience in working with ReactJS ,AngularJS ,HTML ,CSS ,Python ,MongoDB , Sql ,git and Django .I have done a project in which we can create multiple rooms and chat similar to Discord ( link : https://github.com/Shubhamraut01/StuddyBuddy).

I have Knowledge of UI/UX using Figma and Adobe XD . I have done google UX Specialization here are my UI/UX projects : https://designxt.github.io

I have all the essential Skills required to enhance the UI of the project and enhance its features .

I am looking forward to work on this wonderful project with the CloudCV team / mentors together we can create a better user experience for the users. please checkout my portfolio at : https//shubhamraut.me to know more .

Thankyou, Shubham Raut shubhamrao12321@gmail.com

[Rukiyav]

@Ram81 @gchhablani HI GUYS, I am Vaishnavi I would like to be part of this project can to please tell me about it.