Search with site: resulting in RateLimitException

[joaopalotti]

Describe the bug Very frequently, maybe all the time, queries for a specific site result in RateLimitException. This problem happens with both API calls and the ddgs script. I am showing the behavior with ddgs script because it is faster to reproduce without pasting any code here:

This works fine and give me two results:

ddgs text -k 'ayrton senna' -m 2

This doest not work and give me the following error:

ddgs text -k 'ayrton senna site:wikipedia.org' -m 2

Traceback (most recent call last):
  File "/Users/joao/miniforge3/envs/test_ddg/bin/ddgs", line 8, in <module>
    sys.exit(cli())
  File "/Users/joao/miniforge3/envs/test_ddg/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/joao/miniforge3/envs/test_ddg/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/joao/miniforge3/envs/test_ddg/lib/python3.10/site-packages/click/core.py", line 1719, in invoke
    rv.append(sub_ctx.command.invoke(sub_ctx))
  File "/Users/joao/miniforge3/envs/test_ddg/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/joao/miniforge3/envs/test_ddg/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/joao/miniforge3/envs/test_ddg/lib/python3.10/site-packages/duckduckgo_search/cli.py", line 150, in text
    for r in DDGS(proxies=proxy).text(
  File "/Users/joao/miniforge3/envs/test_ddg/lib/python3.10/site-packages/duckduckgo_search/duckduckgo_search.py", line 105, in text
    for i, result in enumerate(results, start=1):
  File "/Users/joao/miniforge3/envs/test_ddg/lib/python3.10/site-packages/duckduckgo_search/duckduckgo_search.py", line 156, in _text_api
    resp = self._get_url("GET", "https://links.duckduckgo.com/d.js", params=payload)
  File "/Users/joao/miniforge3/envs/test_ddg/lib/python3.10/site-packages/duckduckgo_search/duckduckgo_search.py", line 48, in _get_url
    raise RateLimitException(f"_get_url() {url}")
duckduckgo_search.exceptions.RateLimitException: _get_url() https://links.duckduckgo.com/d.js

Specify this information

• OS: Apple M1 Pro Macos 14.0

• environment New conda env created with:

  conda create --name test_ddg python=3.10
  pip install duckduckgo_search

Output of pip freeze:

aiofiles==23.2.1
anyio==4.1.0
Brotli==1.1.0
certifi==2023.11.17
click==8.1.7
duckduckgo-search==3.9.9
exceptiongroup==1.2.0
h11==0.14.0
h2==4.1.0
hpack==4.0.0
httpcore==1.0.2
httpx==0.25.2
hyperframe==6.0.1
idna==3.6
lxml==4.9.3
sniffio==1.3.0
socksio==1.0.0

• duckduckgo_search version

  duckduckgo_search-3.9.9-py3-none-any.whl

Thanks in advance for having a look at it and thanks for the amazing package! Joao

[deedy5]

fixed in v3.9.10

[deedy5]

The site blocks requests with tls-fingerprint. I had to switch from httpx to curl-cffi. Now everything works as it should, hopefully for a long time.