bisohns / search-engine-parser

Lightweight package to query popular search engines and scrape for result titles, links and descriptions
https://search-engine-parser.readthedocs.io
460 stars 87 forks source link

Yahoo engine search broken #169

Closed bentsi closed 2 years ago

bentsi commented 2 years ago

Describe the bug

While running Yahoo search using example code from the Readme,

I get exception:

Traceback (most recent call last):
  File "/home/bentsi/.pyenv/versions/cai-backend/lib/python3.10/site-packages/search_engine_parser/core/base.py", line 252, in get_results
    search_results = self.parse_result(results, **kwargs)
  File "/home/bentsi/.pyenv/versions/cai-backend/lib/python3.10/site-packages/search_engine_parser/core/base.py", line 151, in parse_result
    rdict = self.parse_single_result(each, **kwargs)
  File "/home/bentsi/.pyenv/versions/cai-backend/lib/python3.10/site-packages/search_engine_parser/core/engines/yahoo.py", line 61, in parse_single_result
    rdict["descriptions"] = desc.text
AttributeError: 'NoneType' object has no attribute 'text'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/bentsi/.pyenv/versions/cai-backend/lib/python3.10/site-packages/search_engine_parser/core/base.py", line 255, in get_results
    raise NoResultsOrTrafficError(
search_engine_parser.core.exceptions.NoResultsOrTrafficError: The returned results could not be parsed. This might be due to site updates or server errors. Drop an issue at https://github.com/bisoncorps/search-engine-parser if this persists

To Reproduce A bit altered code from Readme:

from search_engine_parser.core.engines.bing import Search as BingSearch
from search_engine_parser.core.engines.google import Search as GoogleSearch
from search_engine_parser.core.engines.yahoo import Search as YahooSearch

company_name = "samsung electronics corp official website"

search_args = {"query": company_name, "page": 1}
gsearch = GoogleSearch()
ysearch = YahooSearch()
bsearch = BingSearch()
gresults = gsearch.search(**search_args)
yresults = ysearch.search(**search_args)
bresults = bsearch.search(**search_args)

Expected behavior Search works

Desktop (please complete the following information):

bentsi commented 2 years ago

fixed in #170