search

microsoft / onnxruntime-genai

Generative AI extensions for onnxruntime
MIT License
259 stars 66 forks source link

Extensions for LLM #442

Open hannespreishuber opened 1 month ago

hannespreishuber commented 1 month ago

the idea. Write a litte app - put it on a raspberry to swith lights on or off Would start with phi-3 (which works as .net app) extend with whisper.. Would need a feature like embeddings or functions Had a look at semantic kernel but needs a http Rest endpoint, which makes no sense. Any advice?

yufenglee commented 1 month ago

I didn't have experience to run a LLM on raspberry device. How does the perf look like, for example to run a phi3 model?

natke commented 3 weeks ago

@hannespreishuber Your app idea sounds interesting! Can you elaborate on the flow of the application a little more?

Did you try to run phi-3 on raspberry pi already?

Fyi, we are in the process of adding support for whisper.