search

kennethnym / infinifi

infinifi plays gentle lofi music in the background indefinitely
https://infinifi.cafe
Apache License 2.0
278 stars 14 forks source link

small chat feature #11

Open not-nullptr opened 3 months ago

not-nullptr commented 3 months ago

would be super cute honestly !! it already has a tuned in counter and a websocket server running already. would be happy to implement this if i knew it was gonna be merged :3

kennethnym commented 3 months ago

that sounds like a really cool idea!! my only worry is that it's gonna be ruined by spam and the likes :(

not-nullptr commented 3 months ago

hmm.. well it's already running inference for MusicLM which is notoriously hard to run already. does the server have enough VRAM left for a small LLM for sentiment anaylsis, maybe gemma2:2b which is insanely small for the quality? https://ollama.com/library/gemma2:2b

kennethnym commented 3 months ago

i have 6gb of vram left, should be able to run a small LLM? but the gpu is basically always at 100% usage due to it constantly churning out new clips, so i don't know if it can handle another llm. i definitely CANNOT afford to spin up another gpu 😭

Screenshot 2024-08-12 at 23 26 40
not-nullptr commented 3 months ago

i've just looked into it, running an LLM probably isn't worth it. after some local testing they're way too overbearing to censor regular conversation. might just be worth having an IP rate limit

kennethnym commented 3 months ago

hmm i see, i will keep this opened for now, i do want to implement this in v2, but main focus of v2 right now is a fine-tuned model + dynamic prompt generation for more variety. thank u for this cute suggestion though!

feel free to drop a PR if u want, but again the model goes first, i will review it once the model is trained :)

aryanranderiya commented 1 month ago

Hey you could look into Cloudflare Workers AI. You get like a 100k requests a day for free and maybe it's something that'll work out of the box? Not for fine-tuning tho