Duckduckgo,Brave,Startpage都失效

这三个搜索引擎页面可能改变了.

pipenv install Search-Engines-Scraper-Tasos

duckduckgo报错如下:


from search_engines import Google,Bing,Duckduckgo,Brave,Startpage

>>> proxy="http://127.0.0.1:1080"
>>> engine=Duckduckgo(proxy)
>>> results=engine.search("test")
Searching Duckduckgo                                                           
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
  File "/home/user/.local/share/virtualenvs/scraper/lib/python3.11/site-packages/search_engines/engine.py", line 162, in search
    response = self._get_page(request['url'], request['data'])
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/virtualenvs/scraper/lib/python3.11/site-packages/search_engines/engines/duckduckgo.py", line 44, in _get_page
    response = self._http_client.get(page)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/virtualenvs/scraper/lib/python3.11/site-packages/search_engines/http_client.py", line 21, in get
    page = self._quote(page)
           ^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/virtualenvs/scraper/lib/python3.11/site-packages/search_engines/http_client.py", line 41, in _quote
    if utl.decode_bytes(utl.unquote_url(url)) == utl.decode_bytes(url):
                        ^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.local/share/virtualenvs/scraper/lib/python3.11/site-packages/search_engines/utils.py", line 15, in unquote_url
    return decode_bytes(requests.utils.unquote(url))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/urllib/parse.py", line 657, in unquote
    if '%' not in string:
       ^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable

brave报错::

results=engine.search("test")                                              
Searching Brave                                                                
ERROR Failed to decode response content.

Startpage无论搜索什么都返回空结果:

>>> engine3 = Startpage(proxy)
>>> results3 = engine3.search("test")
Searching Startpage                                                            
>>> links3 = results3.links()                             
>>> print(links3)
[]

tasos-py / Search-Engines-Scraper

Duckduckgo,Brave,Startpage都失效 #82