Better default web search function

[cpacker]

Is your feature request related to a problem? Please describe.

• The current http_request function included in memgpt_extras isn't meant for serious web browsing use
• We should include a better set of web browsing related functions exposed as functions/tools to the LLM

Describe the solution you'd like

• search_web tool to execute basic web search
• browse_web tool to OCR a website

We should keep http_request in for API uses, but it should be delegated to APIs and not used for general web browsing.

Can take inspiration from:

• https://github.com/RandomInternetPreson/LucidWebSearch
• https://github.com/oobabooga/text-generation-webui/discussions/932

[agiletechnologist]

There are some considerations about search engine terms of service. Google and Bing specifically have APIs and associated costs. Accessing Bing or Google with a program requires using the API. This would be Ideal for MemGPT to use.

There are Free Tiers for both Google and Bing. This is good for occasional use, but for serious usage, you can need to pay. You will also need a way to add API Keys for the Search.

Google API Info: https://developers.google.com/custom-search/v1/overview Google API Pricing: https://support.google.com/programmable-search/answer/9069107?hl=en

Bing API Info: https://www.microsoft.com/en-us/bing/apis/bing-web-search-api Bing API Pricing: https://www.microsoft.com/en-us/bing/apis/pricing

[TheOnlyWiseJEDI]

You might build off of my google api function (search_api) in my repository, it is configured for google custom search and can be expanded. I had adjusted things to limit the amount of info returned in the name of context window savings. Also, the free tier for google search allows a lot of free searches per month. Just going off memory I'm thinking I read 100,000? idk it was awhile ago but when I saw it I know I said to myself, no way I'm hitting that limit.