Running with open source models from huggingface

evanhu1 / talk2arxiv-server

Apache License 2.0

45 stars 6 forks source link

Running with open source models from huggingface #1

Open damianr13 opened 8 months ago

damianr13 commented 8 months ago

Great repository I was excited to play with this but I don't have keys to Open AI and Cohere API.

I think it would be easier for a newcomer to run this project if it enabled running some open source LLM locally.

evanhu1 commented 8 months ago

I felt that for newcomers just using API keys would be easiest time wise! But to plug in a local LLM, you'd just edit the chat route endpoint in app.py to call your own local LLM's inference instead of OpenAI.