search

AugustDev / enchanted

Enchanted is iOS and macOS app for chatting with private self hosted language models such as Llama2, Mistral or Vicuna using Ollama.
Apache License 2.0
2.29k stars 150 forks source link

add support for Groq #61

Open jtoy opened 3 months ago

jtoy commented 3 months ago

I dont think it makes sense to support tons of other models, but since groq is 10x faster and has free api keys to mixtral at 30 reqs a minute, I think worth it :) https://groq.com/

omniwired commented 2 months ago

I think the whole point of this project is to allow local LLM via ollama to be run easily.

adrianliechti commented 2 months ago

You can use an LLM Gateway for such usecases. Mine should work for that use case also: https://github.com/adrianliechti/llama

image

Enchanted Settings:

image

LLM Platform Config:

config.yaml

providers:
  - type: groq
    token: ${GROQ_API_TOKEN}

    models:
      llama3-70b-instruct:
        id: llama3-70b-8192

      mixtral-8x7b-instruct:
        id: mixtral-8x7b-32768