Integrate web-llm as an LLM option

[singerbj]

⚠️ Please check that this feature request hasn't been suggested before.

• [X] I searched previous Ideas in Discussions didn't find any similar feature requests.
• [X] I searched previous Issues didn't find any similar feature requests.

🔖 Feature description

It would be cool to be able to leverage web-llm as a free model option that runs locally via the webgpu API. It would also be good from a security perspective to allow people to run and train the model locally vs using a 3rd party provider. Honestly, I am not sure it's possible as I am not a data science expert.

📝 Additional Context

https://github.com/mlc-ai/web-llm

Acknowledgements

• [X] My issue title is concise and descriptive.
• [X] I have searched the existing issues to make sure this feature has not been requested yet.
• [X] I have provided enough information for the maintainers to understand and evaluate this request.

[VJlaxmi]

Hi @singerbj, this integration sounds very interesting! I'd be keen to help explore the feasibility and contribute to adding web-llm as an LLM option.

Here's my initial approach:

• Compatibility Analysis: I'll thoroughly review web-llm's documentation and your project's architecture to determine any potential alignment or misalignment points.
• Integration Plan: Based on the analysis, I'll outline a potential integration strategy, addressing areas like API calls, configuration, and user interface updates.
• Prototyping: If feasible, I'll create a proof-of-concept to demonstrate how web-llm could function within your project.

I'm familiar with web technologies and have some experience with language models. Let me know if this approach sounds reasonable, and I'm happy to collaborate!