SEO python scraper to extract data from major searchengine result pages. Extract data like url, title, snippet, richsnippet and the type from searchresults for given keywords. Detect Ads or make automated screenshots. You can also fetch text content of urls provided in searchresults or by your own. It's usefull for SEO and business related research tasks.
I've been using this package for months but this week it starting only returning SERPs with the serp_type value of "shopping". Here is the code I'm running...
scrap = serpscrap.SerpScrap()
scrap.init(config=config.get(), keywords="best dog food")
results = scrap.run()
results -
{'query_num_results_total': '0', 'query_num_results_page': 3, 'query_page_number': 1, 'query': 'best dog food', 'serp_rank': 1, 'serp_type': 'shopping', 'serp_url': 'https://wildearth.com/products/clean-protein-dog-food?variant=31491525673003', 'serp_rating': None, 'serp_title': 'Wild Earth - Healthy Protein Dog Food, Clean High-Protein Formula, 18lbs', 'serp_domain': 'wildearth.com', 'serp_visible_link': None, 'serp_snippet': 'Wild Earth - Healthy Protein Dog Food, Clean High-Protein Formula, 18lbsWild Earth - Healthy Protein Dog Food, Clean High-Protein Formula, 18lbs$70.00Wild EarthWild Earth(5)Wild Earth - Healthy Protein Dog Food, Clean High-Protein Formula, 18lbs$70.00Wild Earth(5)', 'serp_sitelinks': None, 'screenshot': '/tmp/screenshots/2020-10-04/google_best dog food-p1.png'}
{'query_num_results_total': '0', 'query_num_results_page': 3, 'query_page_number': 1, 'query': 'best dog food', 'serp_rank': 2, 'serp_type': 'shopping', 'serp_url': 'https://www.petco.com/shop/en/petcostore/product/orijen-original-dry-dog-food-25-lbs-2992171', 'serp_rating': None, 'serp_title': 'ORIJEN Original Dry Dog Food, 25 lbs.', 'serp_domain': 'www.petco.com', 'serp_visible_link': None, 'serp_snippet': 'CURBSIDE PICKUPPick up todayORIJEN Original Dry Dog Food, 25 lbs.ORIJEN Original Dry Dog Food, 25 lbs.$90.99Petco.comPetco.com(1k+)CURBSIDE PICKUPPick up todayORIJEN Original Dry Dog Food, 25 lbs.$90.99Petco.com(1k+)', 'serp_sitelinks': None, 'screenshot': '/tmp/screenshots/2020-10-04/google_best dog food-p1.png'}
{'query_num_results_total': '0', 'query_num_results_page': 3, 'query_page_number': 1, 'query': 'best dog food', 'serp_rank': 3, 'serp_type': 'shopping', 'serp_url': 'https://www.chewy.com/blue-buffalo-life-protection-formula/dp/170938?utm_source=google-product&utm_medium=organic&utm_campaign=f&utm_content=Blue%20Buffalo&utm_term=%7Bkeyword%7D', 'serp_rating': None, 'serp_title': 'Blue Buffalo Life Protection Formula Adult Chicken & Brown Rice Recipe Dry Dog Food, 3-lb bag', 'serp_domain': 'www.chewy.com', 'serp_visible_link': None, 'serp_snippet': 'Blue Buffalo Life Protection Formula Adult Chicken & Brown Rice Recipe Dry Dog Food, 3-lb bagBlue Buffalo Life Protection Formula Adult Chicken & Brown Rice Recipe Dry Dog Food, 3-lb bag$9.98Chewy.comChewy.comSpecial offerCode copied to clipboardAutoship and Save 5%Chewy.comNo code requiredDiscount is automatically applied at checkout.\xa0CONTINUE TO STORECANCELBlue Buffalo Life Protection Formula Adult Chicken & Brown Rice Recipe Dry Dog Food, 3-lb bag$9.98Chewy.comSpecial offerCode copied to clipboardAutoship and Save 5%Chewy.comNo code requiredDiscount is automatically applied at checkout.\xa0CONTINUE TO STORECANCELFor most items:365-day return policy', 'serp_sitelinks': None, 'screenshot': '/tmp/screenshots/2020-10-04/google_best dog food-p1.png'}
I've been using this package for months but this week it starting only returning SERPs with the serp_type value of "shopping". Here is the code I'm running...
config = serpscrap.Config() config.set('scrape_urls', False) config.set('do_caching', False) config.set('num_pages_for_keyword', 1) config.set('clean_cache_after', 0)
scrap = serpscrap.SerpScrap() scrap.init(config=config.get(), keywords="best dog food") results = scrap.run()
results -
{'query_num_results_total': '0', 'query_num_results_page': 3, 'query_page_number': 1, 'query': 'best dog food', 'serp_rank': 1, 'serp_type': 'shopping', 'serp_url': 'https://wildearth.com/products/clean-protein-dog-food?variant=31491525673003', 'serp_rating': None, 'serp_title': 'Wild Earth - Healthy Protein Dog Food, Clean High-Protein Formula, 18lbs', 'serp_domain': 'wildearth.com', 'serp_visible_link': None, 'serp_snippet': 'Wild Earth - Healthy Protein Dog Food, Clean High-Protein Formula, 18lbsWild Earth - Healthy Protein Dog Food, Clean High-Protein Formula, 18lbs$70.00Wild EarthWild Earth(5)Wild Earth - Healthy Protein Dog Food, Clean High-Protein Formula, 18lbs$70.00Wild Earth(5)', 'serp_sitelinks': None, 'screenshot': '/tmp/screenshots/2020-10-04/google_best dog food-p1.png'}
{'query_num_results_total': '0', 'query_num_results_page': 3, 'query_page_number': 1, 'query': 'best dog food', 'serp_rank': 2, 'serp_type': 'shopping', 'serp_url': 'https://www.petco.com/shop/en/petcostore/product/orijen-original-dry-dog-food-25-lbs-2992171', 'serp_rating': None, 'serp_title': 'ORIJEN Original Dry Dog Food, 25 lbs.', 'serp_domain': 'www.petco.com', 'serp_visible_link': None, 'serp_snippet': 'CURBSIDE PICKUPPick up todayORIJEN Original Dry Dog Food, 25 lbs.ORIJEN Original Dry Dog Food, 25 lbs.$90.99Petco.comPetco.com(1k+)CURBSIDE PICKUPPick up todayORIJEN Original Dry Dog Food, 25 lbs.$90.99Petco.com(1k+)', 'serp_sitelinks': None, 'screenshot': '/tmp/screenshots/2020-10-04/google_best dog food-p1.png'}
{'query_num_results_total': '0', 'query_num_results_page': 3, 'query_page_number': 1, 'query': 'best dog food', 'serp_rank': 3, 'serp_type': 'shopping', 'serp_url': 'https://www.chewy.com/blue-buffalo-life-protection-formula/dp/170938?utm_source=google-product&utm_medium=organic&utm_campaign=f&utm_content=Blue%20Buffalo&utm_term=%7Bkeyword%7D', 'serp_rating': None, 'serp_title': 'Blue Buffalo Life Protection Formula Adult Chicken & Brown Rice Recipe Dry Dog Food, 3-lb bag', 'serp_domain': 'www.chewy.com', 'serp_visible_link': None, 'serp_snippet': 'Blue Buffalo Life Protection Formula Adult Chicken & Brown Rice Recipe Dry Dog Food, 3-lb bagBlue Buffalo Life Protection Formula Adult Chicken & Brown Rice Recipe Dry Dog Food, 3-lb bag$9.98Chewy.comChewy.comSpecial offerCode copied to clipboardAutoship and Save 5%Chewy.comNo code requiredDiscount is automatically applied at checkout.\xa0CONTINUE TO STORECANCELBlue Buffalo Life Protection Formula Adult Chicken & Brown Rice Recipe Dry Dog Food, 3-lb bag$9.98Chewy.comSpecial offerCode copied to clipboardAutoship and Save 5%Chewy.comNo code requiredDiscount is automatically applied at checkout.\xa0CONTINUE TO STORECANCELFor most items:365-day return policy', 'serp_sitelinks': None, 'screenshot': '/tmp/screenshots/2020-10-04/google_best dog food-p1.png'}