buy or host OpenAI compatible API that can serve more than 1000 tokens for the showcase instance

[ycfreeman]

Azure free tier is 1000 token per minute, which means it's almost just 1-2 prompt per minute, not quote useful

we'll either need to host and expose a machine, or buy a hosted service. would need some research on how and how much

[ycfreeman]

cloud hosting

• still haven't found anything that is free yet
  • all are expensive, doesn't look like there's any kind of discount for student or something

self hosting

• https://github.com/ray-project/kuberay
  • probably overkill with kube, and doesn't look like there's any out of the box openai api
• ollama
  • looks like this is only good for local dev use - from reddit comments
• https://github.com/vllm-project/vllm
  • this look like it would be something to look closer to

[ycfreeman]

@Rzz8 suggested Hugging Face Inference

and after a bit of dig through, it seem to actually relatively cheap and easy to use

paid for the PRO tier and it pretty much works without much issue so far

https://huggingface.co/docs/api-inference/en/getting-started

will lock down to this for now