Scrape results from Serper results using Olostep

khoj-ai / khoj

Your AI second brain. Get answers to your questions, whether they be online or in your own notes. Use online AI models (e.g gpt4) or private, local LLMs (e.g llama3). Self-host locally or use our cloud instance. Access from Obsidian, Emacs, Desktop app, Web or Whatsapp.

https://khoj.dev

GNU Affero General Public License v3.0

12.63k stars 640 forks source link

Scrape results from Serper results using Olostep #627

Closed sabaimran closed 7 months ago

sabaimran commented 7 months ago

Use Olostep to go through top results from Serper's google search
Add an intermediate step that summarizes extracted web results for further response downstream
If answerBox is present in the query results, no need to do the additional scraping
Add truncation logic to the model wrapper

debanjum commented 7 months ago

Can we add chat actor, director tests to verify the functioning of this /online with website crawl flows? It's getting a little complicated. It'd be good to see what kind of stuff Khoj can do with this vs without it (It'll improve insight into our chatquality benchmarking via pytests)