Proposal to Make LLM-AutoEval Repository Multi-Cloud Compatible

[adithya-s-k]

Subject: Proposal to Make LLM-AutoEval Repository Multi-Cloud Compatible

---

Issue Title: Enhancing LLM-AutoEval for Multi-Cloud Compatibility using Sky Pilot

Description:

I am proposing a significant enhancement to the LLM-AutoEval repository to make it multi-cloud compatible. Currently, the repository only supports the runpod GPU provider, and my goal is to extend its capabilities to run seamlessly on various cloud platforms using the Sky Pilot framework.

Proposed Roadmap:

1. Dynamic GPU Configuration:

   • Adapt the existing execution script to support different GPU configurations dynamically.
   • Allow users to specify their GPU preferences or let the system auto-detect and utilize available resources efficiently.

2. Integration with Sky Pilot:

   • Modify the execution script to integrate with the Sky Pilot framework.
   • Leverage Sky Pilot's capabilities for running LLMs, AI, and batch jobs on any cloud provider.

3. Multi-Cloud Testing:

   • Thoroughly test the modified script on multiple cloud providers (e.g., AWS, Azure, GCP) to ensure consistency and efficiency.
   • Validate performance and resource utilization across different cloud environments.

4. Documentation:

   • Create comprehensive documentation guiding users on how to:
     • Specify cloud provider preferences and credentials.
     • Use the modified script for multi-cloud deployment.
     • Troubleshoot common issues and optimize performance.

Expected Outcome:

The enhanced LLM-AutoEval will empower users to choose their preferred cloud provider, customize GPU configurations, and seamlessly run experiments using the Sky Pilot framework. This will significantly improve accessibility and usability, fostering collaboration and adoption within the community.

Contributor - @adithya-s-k

[adithya-s-k]

@mlabonne Would like to contribute to the repo. It would be great if you can assign me this issue