search

deedy5 / duckduckgo_search

Search for words, documents, images, videos, news, maps and text translation using the DuckDuckGo.com search engine. Downloading files and images to a local hard drive.
MIT License
937 stars 117 forks source link

418 Client Error #34

Closed shaklev closed 1 year ago

shaklev commented 1 year ago

I'm trying to test the package with multiple IPs(proxies) and different user agents. I'm using multithreading ( although each thread rotates the IP with each new request ). After some time i'm getting: requests.exceptions.HTTPError: 418 Client Error: for url: https://duckduckgo.com/ and WARNING keywords=testword _get_vqd() is None. Refresh session and retry...

Although i'm rotating IPs & UAs on each request after some time all the requests get blocked by that error. I suppose is related to the antibotting system for duckduckgo. Are you aware of some limits related to IPs or UAs ? Do you think they also have limits related to searhing specific sites ( i.e. mrbeast site:instagram.com ) etc ?

deedy5 commented 1 year ago

This implementation is configured for sequential requests in a single thread.