search

bisohns / search-engine-parser

Lightweight package to query popular search engines and scrape for result titles, links and descriptions
https://search-engine-parser.readthedocs.io
459 stars 87 forks source link

DuckDuckGo, end of results link parse failure #156

Closed btreecat closed 2 years ago

btreecat commented 2 years ago

Describe the bug When I do the search site:xkcd.com exploit using DDG, a single page of results is returned. When the duckduckgo.py parse_single_result() method is called, it tries to parse links when it gets a block indicating its the end of the search results. This causes an error with link_tag.get('href') and rdict["descriptions"] = desc.text as they are None.

To Reproduce Steps to reproduce the behavior:

  1. Open python console to
  2. Import search_engine_parser
  3. Search using Duck Duck Go Engine
  4. Use query site:xkcd.com exploit
  5. See NoResultsOrTrafficError (when trying to parse the last result with parse single result)

Expected behavior This should return the list of search_results

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Additional context Working on a PR. I have it fixed but making sure the tests are passing.