search

UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations
https://inspect.ai-safety-institute.org.uk/
MIT License
580 stars 102 forks source link

Use of paid API for `web_search()` tool #576

Closed tadamcz closed 1 week ago

tadamcz commented 1 week ago

Hi! :)

Your docs explain:

The web_search() tool uses Google Programmable Search Engine. To use it you will therefore need to setup your own Google Programmable Search Engine and also enable the Programmable Search Element Paid API.

The Programmable Search Element Paid API page says that:

The Programmable Search Element now offers a paid ads-free version. By enabling the Programmable Search Element Paid API your Programmable Search Element will continue to function exactly the same as the free ad-supported version, except that instead of ads being displayed along with search results you'll instead be charged a small fee for each search.

To clarify: is the reason you require the paid API entirely to avoid ads? Or is there some other reason you are using the paid API?

jjallaire-aisi commented 1 week ago

AFAIK the paid API is the only way to get the REST API that we use in Inspect (they don't have a free REST API for search). This may have changed recently but that was the state of things when we implemented this.

Okay, I checked, and it does look like you can now get 100 queries per day for free: https://developers.google.com/custom-search/v1/overview (not sure if this is enough for your evals?). This should work now with Inspect.

A couple of other search APIs that people like are Bing and Brave (don't know if they have free tiers). If you look at the source code for web_browser() you'll see that it is well set up for additional providers, so if you know of a free provider you want to add support for we'd be glad to take a PR.

tadamcz commented 1 week ago

Makes sense, thanks!

I had looked at https://developers.google.com/custom-search/v1/overview but hadn't seen that they limit API access to 100 queries per day.

Since that's so few in our context (e.g. GAIA has 450 samples), it's probably fine to keep your docs as written, and require the two environment variables as you currently do.