search

kennethnym / infinifi

infinifi plays gentle lofi music in the background indefinitely
https://infinifi.cafe
Apache License 2.0
231 stars 11 forks source link

small chat feature #11

Open not-nullptr opened 1 month ago

not-nullptr commented 1 month ago

would be super cute honestly !! it already has a tuned in counter and a websocket server running already. would be happy to implement this if i knew it was gonna be merged :3

kennethnym commented 1 month ago

that sounds like a really cool idea!! my only worry is that it's gonna be ruined by spam and the likes :(

not-nullptr commented 1 month ago

hmm.. well it's already running inference for MusicLM which is notoriously hard to run already. does the server have enough VRAM left for a small LLM for sentiment anaylsis, maybe gemma2:2b which is insanely small for the quality? https://ollama.com/library/gemma2:2b

kennethnym commented 1 month ago

i have 6gb of vram left, should be able to run a small LLM? but the gpu is basically always at 100% usage due to it constantly churning out new clips, so i don't know if it can handle another llm. i definitely CANNOT afford to spin up another gpu 😭

Screenshot 2024-08-12 at 23 26 40
not-nullptr commented 1 month ago

i've just looked into it, running an LLM probably isn't worth it. after some local testing they're way too overbearing to censor regular conversation. might just be worth having an IP rate limit

kennethnym commented 1 month ago

hmm i see, i will keep this opened for now, i do want to implement this in v2, but main focus of v2 right now is a fine-tuned model + dynamic prompt generation for more variety. thank u for this cute suggestion though!

feel free to drop a PR if u want, but again the model goes first, i will review it once the model is trained :)