This project gives local LLMs the ability to search the web by outputting a specific command. Once the command has been found in the model output using a regular expression, duckduckgo-search is used to search the web and return a number of result pages. Finally, an ensemble of LangChain's Contextual compression and Okapi BM25 (Or alternatively, SPLADE) is used to extract the relevant parts (if any) of each web page in the search results and the results are appended to the model's output.
update_wizard
script inside the text-generation-webui folder
and choose Install/update extensions requirements
. This installs everything using pip
,
which means using the unofficial faiss-cpu
package. Therefore, it is not guaranteed to
work with your system (see the official disclaimer).text-generation-webui/extensions/LLM_Web_search
in a terminal or conda shell.
If you used the one-click install method, run the command
conda env update -p <path_to_your_environment> --file environment.yml
,
where you need to replace <path_to_your_environment>
with the path to the
/installer_files/env
subfolder within the text-generation-webui folder.
Otherwise, if you made your own environment,
use conda env update -n <name_of_your_environment> --file environment.yml
python server.py --extension LLM_Web_search
If the installation was successful and the extension was loaded, a new tab with the title "LLM Web Search" should be visible in the web UI.
See https://github.com/oobabooga/text-generation-webui/wiki/07-%E2%80%90-Extensions for more information about extensions.
Search queries are extracted from the model's output using a regular expression. This is made easier by prompting the model
to use a fixed search command (see system_prompts/
for example prompts).
Currently, only a single search query per model chat message is supported.
An example workflow of using this extension could be:
The default regular expression is:
Search_web\("(.*)"\)
Where Search_web
is the search command and everything between the quotation marks
inside the parentheses will be used as the search query. Every custom regular expression must use a
capture group to extract the search
query. I recommend https://www.debuggex.com/ to try out custom regular expressions. If a regex
fulfills the requirement above, the search query should be matched by "Group 1" in Debuggex.
Here is an example of a more flexible, but more complex, regex that works for several different models:
[Ss]earch_web\((?:["'])(.*)(?:["'])\)
Experimental support exists for extracting the full text content from a webpage. The default regex to use this functionality is:
Open_url\("(.*)"\)
Note: The full content of a web page is likely to exceed the maximum context length of your average local LLM.
This is the default web search backend.
Rudimentary support exists for SearXNG. To use a local or remote SearXNG instance instead of DuckDuckGo, simply paste the URL into the "SearXNG URL" text field of the "LLM Web Search" settings tab. The instance must support returning results in JSON format.
To modify the categories, engines, languages etc. that should be used for a specific query, it must follow the SearXNG search syntax. Currently, automatic redirect and Special Queries are not supported.
This extension comes out of the box with Okapi BM25 enabled, which is widely used and very popuplar for keyword based document retrieval. It runs on the CPU and, for the purpose of this extension, it is fast.
If you don't run the extension in "CPU only" mode and have some VRAM to spare,
you can also select SPLADE in the "Advanced settings" section
as an alternative. It has been shown to outperform BM25 in multiple benchmarks
and uses a technique called "query expansion" to add additional contextually relevant words
to the original query. However, it is slower than BM25. You can read more about it here.
To use SPLADE, you have to install the additional dependency qdrant-client.
Simply activate the conda environment of the main web UI and run
pip install qdrant-client
.
To improve performance, documents are embedded in batches and in parallel. Increasing the
"SPLADE batch size" parameter setting improves performance up to a certain point,
but VRAM usage ramps up quickly with increasing batch size. A batch size of 8 appears
to be a good trade-off, but the default value is 2 to avoid running out of memory on smaller
GPUs.
Naively partitions a website's text into fixed sized chunks without any regard for the text content. This is the default, since it is fast and requires no GPU.
Tries to partition a website's text into chunks based on semantics. If two consecutive sentences have very different embeddings (based on the cosine distance between their embeddings), a new chunk will be started. How different two consecutive sentences have to be for them to end up in different chunks can be tuned using the sentence split threshold
parementer in the UI.
For natural language, this method generally produces much better results than character-based chunking. However, it is noticable slower, even when using the GPU.
If you (like me) have ≤ 12 GB VRAM, I recommend using Llama-3.1-8B-instruct or gemma-2-9b-it.