Closed Tom-Neverwinter closed 2 weeks ago
Hi, we appreciate your suggestion here. It's a handy tool for individual developers to run STORM with minimum usage.
However, we do want to stress that we should respect to the crawling / scraping policy, and for most of the time it's non trivial. It would be problematic if we query search engine extensively without using their API that could potentially lead to serious trouble.
There are many alternative / more affordable search engine API options. We are happy to review and merge effort down this line.
Summary Implement a feature that allows the system to perform automated web searches and scrape relevant content using Selenium and Scrapy.
This is a quick and dirty framework of the idea, this is in no way final or working.
lets be honest here nobody wants to pay for YouRM, BingSearch when we can do this ourselves
Motivation This feature will enable the Storm-oval project to search the web for relevant articles, gather information, and scrape necessary content without relying on external APIs. This can be useful for building a local knowledge base, gathering data for analysis, or enhancing the functionality of the virtual assistant.
Proposed Implementation Dependencies Installation:
Install Scrapy and Selenium:
WebDriver Setup:
Download the WebDriver for your browser (e.g., ChromeDriver for Chrome) and ensure it is accessible from your system path. Scrapy Project Setup:
Create a new Scrapy project:
Selenium Script for Web Search:
Create a Python script (search.py) to perform web searches and save URLs:
Scrapy Spider for Content Scraping:
Modify search_scraper/spiders/search_spider.py to scrape content from the URLs:
Running the Scrapy Spider:
Execute the spider to scrape the content and save it to a file:
Benefits Automation: Automatically perform web searches and scrape content without manual intervention. Cost-Efficient: Avoid the costs associated with using external APIs. Customization: Tailor the search and scraping process to specific needs and preferences. Potential Use Cases Building a local knowledge base. Gathering data for research and analysis. Enhancing the functionality of the virtual assistant.