search

NikolaiT / GoogleScraper

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.
https://scrapeulous.com/
Apache License 2.0
2.6k stars 734 forks source link

Getting a KeyError: search_engine_name when I made my own .db file #196

Open domtuazon opened 6 years ago

domtuazon commented 6 years ago

I'm not sure if I'm missing any steps, but I haven't seen any on the readme.

Steps I did is: I copied the google_scraper.db file, renamed it and added 'database_name' = 'google_scraper_new' to the config dict.

After that I'm getting the following error:

Traceback (most recent call last): File "GoogleSearchWebCrawler.py", line 57, in <module> search = scrape_with_config(config)

File "d:\users\dltuazon\documents\pyscripts\googlescraper-master\GoogleScraper\core.py", line 64, in scrape_with_config return main(return_results=True, parse_cmd_line=False, config_from_dict=config)

File "d:\users\dltuazon\documents\pyscripts\googlescraper-master\GoogleScraper\core.py", line 362, in main scrape_jobs = cache_manager.parse_all_cached_files(scrape_jobs, session, scraper_search)

File "d:\users\dltuazon\documents\pyscripts\googlescraper-master\GoogleScraper\caching.py", line 413, in parse_all_cached_files store_serp_result(serp, self.config)

File "d:\users\dltuazon\documents\pyscripts\googlescraper-master\GoogleScraper\output_converter.py", line 121, in store_serp_result print(serp)

File "d:\users\dltuazon\documents\pyscripts\googlescraper-master\GoogleScraper\database.py", line 94, in __str__ **self.__dict__) KeyError: 'search_engine_name'

Edit: This is running on a Windows 10 machine, Python 3.5 version

domtuazon commented 6 years ago

Sooo I got this to work.

If you encounter this error, make sure to clear out your .scrapecache folder as well when creating a new db. I think some part of the code is referencing them. Haven't dug through the codes enough that deep to check.

Thanks.