search

Gene-Weaver / VoucherVision

Initiated by the University of Michigan Herbarium, VoucherVision harnesses the power of large language models (LLMs) to transform the transcription process of natural history specimen labels.
https://huggingface.co/spaces/phyloforfun/VoucherVision
GNU General Public License v3.0
11 stars 1 forks source link

Consider supporting groq #6

Open nickynicolson opened 1 week ago

nickynicolson commented 1 week ago
Groq provides an OpenAI compatible API to several LLMs e.g. LLaMA3 8b, LLaMA3 70b, Mixtral 8x7b, Gemma 7b (documented on the models page) and free access - as of 2024-06-24 usage limits are: ID Request per minute Requests per day Tokens per minute
llama3-8b-8192 30 14,400 30,000
gemma-7b-it 30 14,400 15,000
mixtral-8x7b-32768 30 14,400 5,000
llama3-70b-8192 30 14,400 6,000

Is it worth considering supporting this as an LLM API provider?

Gene-Weaver commented 1 week ago

Absolutely! This is certainly on the todo list. Mixtral-8x7b and llama3-70b are both good candidates. This might also help you scale up significantly. I will make sure it happens soon. Groq would also be good for the agents workflow given the huge speed advantage.

I also have a sandbox script that calls 256 OpenAI API calls simultaneously, so that can achieve more or less 1,000 transcriptions per minute too (assuming we want to pay for it!).