search

hoarder-app / hoarder

A self-hostable bookmark-everything app (links, notes and images) with AI-based automatic tagging and full text search
https://hoarder.app
GNU Affero General Public License v3.0
2.36k stars 78 forks source link

How do I set the variable "user" or "system" for AI inference #262

Open Silverdutchman opened 3 days ago

Silverdutchman commented 3 days ago

Hello,

I use Perplexity to connect Hoarder. But I get the following error message.

2024-06-30T08:42:33.745Z error: [inference][31] inference job failed: Error: 400 Last message must have role user.

it seems that Perplexity, that's how I interpretate it, need the variabele "role" to be set. My question is: how and where can I do that in the config?

kamtschatka commented 3 days ago

Perplexity is not on the supported list, what gives you reason to believe, that this should work?

MohamedBassem commented 3 days ago

@kamtschatka we support openAI compatible APIs and perplexity is one of them (https://docs.perplexity.ai/docs/getting-started)

Silverdutchman commented 3 days ago

@kamtschatka I never stated that I believed that Perplexity is supported, although it is an openAI compatible API that is supported by the Hoarder Team, that doesn't mean I should/could work out of the box. Hence the question, nothing more and nothing less 🙏

@MohamedBassem thank you for the clarification, it was the foundation on why I asked this.

kamtschatka commented 3 days ago

@MohamedBassem we should probably update that then and state that we only support OpenAI. It doesn't work for https://github.com/hoarder-app/hoarder/issues/137 and it does not work here.

MohamedBassem commented 2 days ago

@kamtschatka If it doesn't work for azure and doesn't work for perplexity, it doesn't mean that it doesn't work for other openAI compatible APIs. I'd argue that the problem here is that perplexity is out of spec (or more restrictive) than openAI.

@Silverdutchman I don't think the problem here is a variable that you have to tune or a role that you have to set. It's probably because we're requesting the inference with the message having the role system here (https://github.com/hoarder-app/hoarder/blob/main/apps/workers/inference.ts#L46) while perplexity wants it to be user instead of system. I think user should work, I'm just a bit concerned if this might degrade the inference performance in any ways.

MohamedBassem commented 2 days ago

I'm happy to try to changing the role to user, I don't expect this to cause problems.