ecoron / SerpScrap

SEO python scraper to extract data from major searchengine result pages. Extract data like url, title, snippet, richsnippet and the type from searchresults for given keywords. Detect Ads or make automated screenshots. You can also fetch text content of urls provided in searchresults or by your own. It's usefull for SEO and business related research tasks.
https://github.com/ecoron/SerpScrap
MIT License
257 stars 61 forks source link

csv_writer.py csvWriter cannot handle ? in urls #54

Open fcbits opened 5 years ago

fcbits commented 5 years ago

Response Traceback (most recent call last): File "miniconda3/lib/python3.7/site-packages/serpscrap/csv_writer.py", line 14, in write w.writerow(row) File "miniconda3/lib/python3.7/csv.py", line 155, in writerow return self.writer.writerow(self._dict_to_list(rowdict)) File "miniconda3/lib/python3.7/csv.py", line 151, in _dict_to_list

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "example_serp_urls.py", line 12, in scrap.as_csv('/tmp/outputurls') File "miniconda3/lib/python3.7/site-packages/serpscrap/serpscrap.py", line 148, in as_csv writer.write(file_path + '.csv', self.results) File "/miniconda3/lib/python3.7/site-packages/serpscrap/csv_writer.py", line 17, in write raise Exception Exception

MindaugasVaitkus2 commented 4 years ago

I think the problem is Google search result pages which have been changed since last working version. In Lithuanian search scrapper scraps only Youtube results and csv_writer.py writes them down. If there are no Youtube results then there is nothing to write down and code returns error.