deedy5 / duckduckgo_search

Search for words, documents, images, videos, news, maps and text translation using the DuckDuckGo.com search engine. Downloading files and images to a local hard drive.
MIT License
927 stars 117 forks source link

Ratelimit when using ~ and "" #194

Closed PascalStrewe closed 3 months ago

PascalStrewe commented 3 months ago

Hey team,

I encounter a ratelimit error when executing this code, when using the commented out line of results = it works fine, so I assume different limits when using advanced search terms, but I only want 1 result, so don't quite get it. Happens for the 1st iteration already.

for company in companies:
    if company not in ["HTTP Error: 412 - Precondition Failed", "JSON array is empty.", '', "#WAARDE!", "#VERW!"]:
        company_results = []  # Store results for this company here

        for keyword in keywords:
            with DDGS() as ddgs:
                # results = ddgs.text(company + " " + keyword, region='wt-wt', safesearch='moderate', max_results=1)
                results = ddgs.text('~' + '"' + company + ' ' + keyword + '"', region='wt-wt', safesearch='moderate', max_results=1)
                company_results.extend(results)

                if len(company_results) >= 5:
                    break

Error:

DuckDuckGoSearchException Traceback (most recent call last) /usr/local/lib/python3.10/dist-packages/duckduckgo_search/duckduckgo_search_async.py in _aget_url(self, method, url, **kwargs) 67 if _is_500_in_url(str(resp.url)) or resp.status_code == 202: ---> 68 raise DuckDuckGoSearchException("Ratelimit") 69 if resp.status_code == 200:

DuckDuckGoSearchException: Ratelimit

The above exception was the direct cause of the following exception:

DuckDuckGoSearchException Traceback (most recent call last) 8 frames /usr/local/lib/python3.10/dist-packages/duckduckgo_search/duckduckgo_search_async.py in _aget_url(self, method, url, **kwargs) 70 return resp_content 71 except Exception as ex: ---> 72 raise DuckDuckGoSearchException(f"_aget_url() {url} {type(ex).name}: {ex}") from ex 73 74 async def _aget_vqd(self, keywords: str) -> Optional[str]:

DuckDuckGoSearchException: _aget_url() https://links.duckduckgo.com/d.js DuckDuckGoSearchException: Ratelimit

deedy5 commented 3 months ago

The problem isn't the double quotes. Update to v5.1.0 pip install -U duckduckgo_search

maxlorax commented 3 months ago

Just commenting to say how much I love your dedication <3

we need more devs like you, deedy5

PascalStrewe commented 3 months ago

Thanks for the quick reply! I am a bit confused, it worked, but I did install duckduckgo_search that way in the first place, just checked the history in Google Colab again. Major thanks anyways, appreciate the help!

image