Integrating LLMs into search (link prediction, top-site summarization, stable diffusion images, academic articles)

[alexlyee]

There are plenty of amazing solutions for using large language models (LLMs) to help with searching. For sake of compressing this request, I'll point out four kinds of them that I want in a modern search engine:

• Metaphor systems has a working beta which is trained to predict links based on prompts. This is much more like a classic search engine, except, my god, it already has been so much better at getting useful articles than any single search engine has been for my searches.
• Perplexity, KOMO, and Uminal have created chat bots that take any text sent to them, forward it to a search engine, pick the top x resulting sites, and feed them through an LLM that summarizes them. They all have been fantastic for cutting through the clutter that I'm so used to having to sift through on the internet and getting exactly the answer I want. The first two include citations, links to the sites where they got the facts in their answers from, which is amazing. Uminal may be the most useful to us though, as I'm aware the owner is considering open-sourcing it (!)
• Though I know there are many, Lexica is the longest-standing AI image search engine I know of.
• Elicit is the best solution I know of today for searching scholarly articles using AI, and it's fantastic.

It would be great if we could implement a menu on the side for link-prediction and top-site summarizing LLMs like those outlined. Perhaps stable-diffusion image and AI-based academic article searchers could be implemented as engines, or perhaps they ought to have their own tabs?

If there isn't an API available, is it a possibility to have a headless browser scrape a response page?

Maybe it would be a good idea to cache responses to eliminate unnecessary duplicate requests to these aforementioned free providers?

I'm aware of #2009, but I don't think with these offerings that already exist we should use ChatGPT to answer prompts directly, though that is certainly an option (!)

I'd be happy to work on this here if we think that this is the way forward. Please give input if you know of better ways of achieving this right now!

cheers, alex

---

Related:

• https://github.com/searxng/searxng/issues/2008
• https://github.com/searxng/searxng/issues/2273

[blob42]

This might be slightly related to #2129 .

I am currently adding a PR to langchain that would allow the usage of SearxNG API to be used as a search engine for LLMs.

If SearX is to be used with other LLM frameworks or libraries, for a quick fix I needed to allow whitelising for the rate limiter for those who self host instances. A proper solution is to allow unlimited usage of the API using a key.

[blob42]

@alexlyee

Follow up to my previous comment to add more context and reply to the original topic of this issue.

First, I believe integrating smart answers/prompting would really make Searx shine as an alternative free(libre) meta search engine. As a searx user (self hosted), my main reason is privacy and control of my search results.

Prompt/LLM based search assistants sounds amazing but right now the main problem is that for each tool/assistant I am required to login with Google/Discord ... and provide my private data to be harvested. Given the unlimited potential of LLMs for enhanced information retrieval, at this rhythm we will have a ridiculous number of tools, each one requiring signup or some sort of API key to use.

Hence, my interest for using langchain with searx. Langchain is a toolkit to build LLM based smart agents, chat bots etc. It allows you to quickly hook LLM chain calls to the internet or other APIs. Almost all the tools that where mentioned as examples could be reproduced with minimal code using Langchain.

In order to use LLMs langchain made an abstract api to hook in any LLM, right now I believe it's mainly OpenAI, but they will be able to handle other LLMs including free and open source ones in the future.

In summary, SearX could offer LLM assisted search by developing its own modules that would use Searx itself for searching. A typical use case would be to allow users to input their OpenAI API key and enable AI based search. A big advantage as well, langchain is a python project and so is searx, it would be easy to quickly come up with equivalents to many of the proprietary tools that are proliferating at an insane pace.

[alexlyee]

@blob42 you bring up an excellent point that currently openAI requires a phone number to sign up which can be used to identify you. It's not easy (and it's cost-prohibitive) to obtain an anonymized VoIP number for use with openAI. The other services I mentioned are usable with just an email, but I wonder if it will stay that way.

You make several other great points, like the fact that they are both Python based, however, I don't think it's necessary to interact with OpenAI at this point now that tools like metaphor.systems and perplexity exist.

[blob42]

@alexlyee using langchain does not require OpenAI, they already adapted the hugging face library as an LLM and many others as well. The LLM part is only a small piece that is required to build more advanced assistants. Say in a year or two OpenAssistant is ready it will be trivial to add to LangChain.

My point being, instead of integrating specific tools with Searx, it would be great to integreate a framework that would allow anyone to build their own assistant experience the same way it is done with engines. On the searx side an API needs to be defined for developping assistants or agents, then they can be activated or not in the settings or UI.

To use some of your examples, let's pick Metaphor. In langchain a similar can be made in many ways, for example using ReAct.

I will find some time soon to make a quick notebook demonstrating the feature using langchain and searx results.

[alexlyee]

@blob42 I see, and yeah I was thinking the same when I found out about OpenAssistant only a few days ago. That sounds excellent! I agree that it would be better to go about it that way. I suppose I figured that it would be quickest to go about it by using the aforementioned already built services, and that would come later. I've got to learn more about langchain, it seems to be something really good to know about. Don't know where to start. Also I noticed that someone took the time to 👎 this topic-- is there something we've got really wrong here?

[blob42]

We could create an MVP with some simple LLM assistant. My proposal would be to split the work in these steps:

1. Prototype some assistant scenarios in notebooks, it will be easy when my Searx wrapper for langchain will be merged.
2. Draft some ideas for an API / conventions for adding assistants the same way as engines.
3. What would the UI side look like ? I would consider adding them in a different view than the default one so the Searx experience stays exactly the same for those who are not interested in this.

In the meantime let's wait for some feedback from other users and core maintainers.

[alexlyee]

Agreed. I think the link-predicting models like metaphor should be made as an engine, while all the others that return anything more than links should be classified as assistants. Like I mentioned before, the assistants could be placed on the right hand side of the search results in their own boxes. Do you think it would be best to have a chat box in there or a link to a page where one can continue the conversation in a place not unlike ChatGPT's UI?

[bgvaughan]

99% of the reason I set up a SearXNG instance for my personal use is that I was hoping to filter out LLM-generated nonsense. I would consider deliberately including it grossly unethical.

[alexlyee]

@bgvaughan thank you for your opinion, I was wondering what the stance was of the people downvoting! I guess I can understand that, although I don't see the harm in making it an option. I wouldn't have opened this suggestion if I didn't find tools like metaphor.systems extremely useful myself. Since I made that initial post, probably a quarter of my searches online have been made using that tool, and it's been far more effective than a regular search engine for certain things. Massive productivity booster for me, and so I'd love to have it somehow integrated into my every search, and I'm sure I'm not alone in that opinion

[dinvlad]

Tbh while I remain skeptical about LLMs too, I agree with @alexlyee this should at least be an opt-in. As these technologies mature, while they will likely never be perfect, they can at least be useful to some. Like @blob42, I also think that LangChain goes a long way towards preserving privacy, for those like myself who might prefer to have a locally-hosted option.

[rasos]

With Mistral's open sourced 8x7B model there is a new powerful LLM which one could even host locally https://mistral.ai/news/mixtral-of-experts/

Agree that results from any LLM should be opt-in and always clearly indicated as AI generated.

[dalf]

FYI Perplexica is the other way around: it relies on SearXNG among other projects to provide an AI assistant.