Facing the Ratelimit problem

deedy5 / duckduckgo_search

Search for words, documents, images, videos, news, maps and text translation using the DuckDuckGo.com search engine. Downloading files and images to a local hard drive.

MIT License

1.18k stars 132 forks source link

Facing the Ratelimit problem #218

Closed smartliuhw closed 5 months ago

smartliuhw commented 5 months ago

Before you open an issue:

[x] I have the latest version. (Check: ddgs version. Update: pip install -U duckduckgo_search)
[x] I tried reinstalling the library. (pip install -I duckduckgo_search)
[ ] https://duckduckgo.com is accessible in my browser
[ ] I tried using a proxy (the site may block ip for a while)

Describe the bug

I'm using this repo to retrieve a dataset (with about 10k datas) to get the knowledge. But soon I got the Ratelimit error. I could get the results at the beginning. Is it beacause my IP was blocked or there's some bug in the newest version? I have set a 1 second interval between each request, but a multiprocessing was used in my code. Is there any suggestion?

Specify this information

OS AlmaLinux release 9.2(RedHat)
environment python3.10
duckduckgo_search version 6.1.0

deedy5 commented 5 months ago

There's some kind of problem on the site.

smartliuhw commented 5 months ago

There's some kind of problem on the site.

I see, is there anything we can do to fix it?

jonny7737 commented 5 months ago

The issue is with Microsoft aka: Bing, Copilot and ChatGPT.

On May 23, 2024, at 9:24 AM, smartliuhw @.***> wrote:

There's some kind of problem on the site.

I see, is there anything we can do to fix it?

— Reply to this email directly, view it on GitHub https://github.com/deedy5/duckduckgo_search/issues/218#issuecomment-2127258061, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK6S42V6EFNXWY4O2XBFIMLZDX3ZNAVCNFSM6AAAAABIFGINA6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRXGI2TQMBWGE. You are receiving this because you are subscribed to this thread.

gee842 commented 5 months ago

I also am facing the same issue. the website works for me and I am calling the library on its own without the use of other tools.

deedy5 commented 5 months ago

Does the error still arise?

smartliuhw commented 5 months ago

Does the error still arise?

Now single thread can run, thank you very much for your answer. Do you have any suggestions on request frequency? Can I open multiple threads to make requests? What is the appropriate interval between

deedy5 commented 5 months ago

If you send requests too often, the site may block your ip.