Using local LLMs

[heyarne]

Hi!

I think this is an interesting tool and I'm very sympathetic to trying out different ways to sort the fediverse timelines algorithmically. Very cool idea!

There are some issues I see however in taking what other people post and sending it to a company that siphons off a large part of the web and re-sells it to you. There have been instances of other peoples' prompts leaking, so this may be a privacy issue as well. This may be especially problematic when direct messages are handled indiscriminately.

Can you clarify how much is sent to OpenAI when setting the URL / configuring the API key? I understand that by default it sends nothing, which is nice. And second, is it possible to run this with a local LLM? I guess this is possible via plugins, is there a built-in method? Is there anything to look out for when picking a particular model?

Thanks! Looking forward to playing around with it.

[tkellogg]

Ah! Great questions. First:

this may be a privacy issue as well. This may be especially problematic when direct messages are handled indiscriminately.

[Mastodon-specific] WARNING: your DMs aren't encrypted and are only private if you trust your admin (and the other admin) to keep them private.

Can you clarify how much is sent to OpenAI when setting the URL / configuring the API key?

There are two AI interactions

1. Embedding model (not really an LLM, but similar) — the full text of the toot is sent, with nothing else (you don't want to prompt embedding models as extra info confuses them)
2. The summarizer — by default I'm using gpt-3.5-turbo, the full text of all toots in a group are sent (unless they don't fit, in which case they're truncated), along with a prompt instructing it to summarize.

Right now, user names aren't sent, nor is any other metadata. But that could change if it's beneficial to some algorithm.

And second, is it possible to run this with a local LLM? I guess this is possible via plugins, is there a built-in method? Is there anything to look out for when picking a particular model?

Yes, it's not fully baked yet, but it uses llm to manage models, so you should be able to have access to the full set of models that llm can install via it's own plugin system.

I'll be experimenting with local models soon. I hope to see big accuracy gains with the embedding model (OpenAI's ada-002 isn't that good). Most small LLMs should also work well for summarizing text too. Regardless, I think you'll have to expect that a lot of people will still use OpenAI anyway, because it's easier to setup or they don't have the hardware to run a bigger model.

Let me know if you have any other questions.

[heyarne]

[Mastodon-specific] WARNING: your DMs aren't encrypted and are only private if you trust your admin (and the other admin) to keep them private.

I'm not sure I understand. Is this a quote or something your wrote? I remember reading something similar to this and it referred to admin's of each party involved in a private conversation being able to access direct messages. So you have to trust them not to.

Building a client that sends all content to a third party is a different thing, isn't it? I'm not sure encryption would help, since the client needs access to the decrypted message in order to display it to the user.

Phrasing it like this I'm actually unsure it's legal to do that? I don't know if there's a precedent for that, but at least in the EU you have to very explicitly consent to how your data is used. I think people would be upset if you took all e-mails between the both of you and sent them to a third party?

There are two AI interactions […]

That's super helpful information, thanks.

Yes, it's not fully baked yet, but it uses llm to manage models, so you should be able to have access to the full set of models that llm can install via it's own plugin system. I'll be experimenting with local models soon.

Cool, super exciting! I'll keep an eye on it

[tkellogg]

I wrote everything, 100%. But I am paraphrasing something I heard about a year ago. The gist is that nothing is/was encrypted in the database, so any admin can see. Potentially this changed, idk. But yeah, I don't think it is compliant with EU law. Shadow IT, I guess.