search

edgar971 / open-chat

A self-hosted, offline, ChatGPT-like chatbot with different LLM support. 100% private, with no data leaving your device.
MIT License
63 stars 8 forks source link

Hardwareacceleration with Google Coral? #13

Open masterdot opened 8 months ago

masterdot commented 8 months ago

Hey there,

does anyone know if the invest in a usb coral device is worthy? The resposes are really slow and kicking the cpu in extreme load :) would be cool to speed it up a bit (or lot).

thanks in advance

TheQuickestFox commented 7 months ago

The coral is focused around image recognition for things like person identification and classification, I think it has no onboard memory which is what seems to be critical with LLMs. So I don't see it being a viable solution.

As to the CPU load, there is currently another active issue #12 where multiple users report no GPU load and full CPU load - I think there's something misconfigured in this image causing it not to use the GPU correctly.