[New feature] Developing online search feature

[donlinglok1]

So I found a demo project that can let the chatbot search on the internet using Selenium and BeautifulSoup. https://github.com/THUDM/WebGLM/ https://github.com/LemonQu-GIT/ChatGLM-6B-Engineering

I wonder can this use on SillyTavern? But I am not familiar with the sillytavern extra extension, so this is still on developing and welcome contributors. Hope can become a useful feature of SillyTavern-Extras. https://github.com/donlinglok/SillyTavern-Extras/tree/wikibook

My idea is to add online internet data in user input and pass it to LLM to process, I am not sure if this kind of lorabook thing. Or can implement to chromadb. Currently, I only use BeautifulSoup to parse the https://en.wikipedia.org/, I think other websites can do so. below is the demo:

$py wikilorabook_module.py '""Unity Game""' 1 1
['""Unity Game""']
https://en.wikipedia.org/w/index.php?fulltext=1&ns0=1&search=Unity%20Game
Finish search_wiki_list on 0.79537034034729s

https://en.wikipedia.org/wiki/Unity_(game_engine)
Finish read_wiki_page on 0.433774471282959s

{"lorebookVersion": 1, "entries": [{"text": "Unity (game engine) is \nUnity is a cross-platform game engine developed by Unity Technologies, first announced and released in June 2005 at Apple Worldwide Developers Conference as a Mac OS X game engine. The engine has since been gradually extended to support a variety of desktop, mobile, console and virtual reality platforms. It is particularly popular for iOS and Android mobile game development, is considered easy to use for beginner developers, and is popular for indie game development.", "contextConfig": {"prefix": "", "suffix": "\n^Unity (game engine)(https://en.wikipedia.org/wiki/Unity_(game_engine))", "tokenBudget": 2048, "reservedTokens": 0, "budgetPriority": 400, "trimDirection": "trimBottom", "insertionType": "newline", "insertionPosition": -1}, "displayName": "Unity (game engine)", "keys": ["Unity (game engine)"], "searchRange": 1000, "enabled": true, "forceActivation": false}]}
Finish run on 1.234328269958496s

pretty json

{
   "lorebookVersion":1,
   "entries":[
      {
         "text":"Unity (game engine) is \nUnity is a cross-platform game engine developed by Unity Technologies, first announced and released in June 2005 at Apple Worldwide Developers Conference as a Mac OS X game engine. The engine has since been gradually extended to support a variety of desktop, mobile, console and virtual reality platforms. It is particularly popular for iOS and Android mobile game development, is considered easy to use for beginner developers, and is popular for indie game development.",
         "contextConfig":{
            "prefix":"",
            "suffix":"\n^Unity (game engine)(https://en.wikipedia.org/wiki/Unity_(game_engine))",
            "tokenBudget":2048,
            "reservedTokens":0,
            "budgetPriority":400,
            "trimDirection":"trimBottom",
            "insertionType":"newline",
            "insertionPosition":-1
         },
         "displayName":"Unity (game engine)",
         "keys":[
            "Unity (game engine)"
         ],
         "searchRange":1000,
         "enabled":true,
         "forceActivation":false
      }
   ]
}

longer demo:

$py wikilorabook_module.py 'Did you hear about ""LK99""? It is a new ""materials"", checkout here ""https://en.wikipedia.org/wiki/Ambient_pressure""".'
['""LK99""', '""materials""', '""https://en.wikipedia.org/wiki/Ambient_pressure""']
https://en.wikipedia.org/w/index.php?fulltext=1&ns0=1&search=LK99
Finish search_wiki_list on 0.8338730335235596s

https://en.wikipedia.org/wiki/LK-99
Finish read_wiki_page on 0.4473867416381836s

https://en.wikipedia.org/w/index.php?fulltext=1&ns0=1&search=materials
Finish search_wiki_list on 1.572911024093628s

https://en.wikipedia.org/wiki/Material
Finish read_wiki_page on 0.255573034286499s

https://en.wikipedia.org/wiki/Ambient_pressure
Finish read_wiki_page on 0.24683809280395508s

{"lorebookVersion": 1, "entries": [{"text": "LK-99 is LK-99 (from the Lee-Kim 1999 research)is a gray–black, polycrystalline compound, identified as a copper-doped lead‒oxyapatite. A team from Korea University led by Lee Sukbae (이석배) and Kim Ji-Hoon (김지훈) began studying this material as a potential superconductor starting in 1999.  In July 2023, they published preprints claiming that it acts as a room-temperature superconductor  at temperatures of up to 400 K (127 °C; 260 °F) at ambient pressure. \n", "contextConfig": {"prefix": "", "suffix": "\n^LK-99(https://en.wikipedia.org/wiki/LK-99)", "tokenBudget": 2048, "reservedTokens": 0, "budgetPriority": 400, "trimDirection": "trimBottom", "insertionType": "newline", "insertionPosition": -1}, "displayName": "LK-99", "keys": ["LK-99"], "searchRange": 1000, "enabled": true, "forceActivation": false}, {"text": "Material is Material is a substance or mixture of substances that constitutes an object. Materials can be pure or impure, living or non-living matter. Materials can be classified on the basis of their physical and chemical properties, or on their geological origin or biological function. Materials science is the study of materials, their properties and their applications.\nRaw materials can be processed in different ways to influence their properties, by purification, shaping or the introduction of other materials. New materials can be produced from raw materials by synthesis.\n", "contextConfig": {"prefix": "", "suffix": "\n^Material(https://en.wikipedia.org/wiki/Material)", "tokenBudget": 2048, "reservedTokens": 0, "budgetPriority": 400, "trimDirection": "trimBottom", "insertionType": "newline", "insertionPosition": -1}, "displayName": "Material", "keys": ["Material"], "searchRange": 1000, "enabled": true, "forceActivation": false}, {"text": "Ambient pressure is The ambient pressure on an object is the pressure of the surrounding medium, such as a gas or liquid, in contact with the object.Within the atmosphere, the ambient pressure decreases as elevation increases. By measuring ambient atmospheric pressure, a pilot may determine altitude (see pitot-static system). Near sea level, a change in ambient pressure of 1 millibar is taken to represent a change in height of 9 metres (30 ft).[citation needed]\n", "contextConfig": {"prefix": "", "suffix": "\n^Ambient pressure(https://en.wikipedia.org/wiki/Ambient_pressure)", "tokenBudget": 2048, "reservedTokens": 0, "budgetPriority": 400, "trimDirection": "trimBottom", "insertionType": "newline", "insertionPosition": -1}, "displayName": "Ambient pressure", "keys": ["Ambient pressure"], "searchRange": 1000, "enabled": true, "forceActivation": false}]}
Finish run on 3.3680498600006104s

[Cohee1207]

That sounds like a great idea. You can send a PR for your Extras improvements, and we'll think about the UI part for the SillyTavern extension later on.

[donlinglok1]

Great, i have completed the backend api, and no additional dependency require. (maybe can have a frontend ajex version in future) The pull request is here: https://github.com/SillyTavern/SillyTavern-Extras/pull/166

The workflow should like this: User submit text -> Call online_lorabook api -> Import the result lorabook json -> Continue with the normal lorabook flow.

I will also update a fandom parser soon.