search

deedy5 / duckduckgo_search

Search for words, documents, images, videos, news, maps and text translation using the DuckDuckGo.com search engine. Downloading files and images to a local hard drive.
MIT License
932 stars 117 forks source link

RateLimitException in 3.9.10 & 3.9.11 #150

Closed zimpstar closed 6 months ago

zimpstar commented 6 months ago

Hi there,

First of all. Awesome package. Love it.

Howver, ever since the 3.9.10 & 3.9.11 updates, I'm getting this error:

Search Term #1 : what is seo Error occurred while searching for 'what is seo': _get_url() https://duckduckgo.com URL 1: None Search Term #2 : what is cro Error occurred while searching for 'what is cro': _get_url() https://duckduckgo.com URL 2: None Generic URLs: None

It would seem like it's just not grabbing any urls at all from the results anymore. I've spent about 8 hours trying to fix this now and was hoping to find some help here, hope that is OK :)

This is the part of the code I'm using:

def scraper(main_keyword, secondary_keyword_1, secondary_keyword_2, title):
    search_term = secondary_keyword_1
    print(f"Search Term #1 : {search_term}")

    wikipedia_url = None

    try:
        with DDGS() as ddgs:
            wikipedia_results = [r for r in ddgs.text(search_term, max_results=1)]
            if wikipedia_results:
                wikipedia_url = wikipedia_results[0]['href']
    except Exception as e:
        print(f"Error occurred while searching for '{search_term}': {e}")

    print(f"URL 1: {wikipedia_url}")

    time.sleep(5)

    secondary_search_term = secondary_keyword_2
    print(f"Search Term #2 : {secondary_search_term}")
    generic_url = None

    try:
        with DDGS() as ddgs:
            generic_results = [r for r in ddgs.text(secondary_search_term, max_results=1)]
            if generic_results:
                generic_url = generic_results[0]['href']
    except Exception as e:
        print(f"Error occurred while searching for '{secondary_search_term}': {e}")

    print(f"URL 2: {generic_url}")
    print(f"Generic URLs: {generic_url}")

Please excuse the horrendous use of variable names, I'm new to this!

zimpstar commented 6 months ago

I also tried an example from the documentation (for text) and its throwing this error:

image

"duckduckgo_search.exceptions.VQDExtractionException: Could not extract vqd. keywords='live free or die'"

zimpstar commented 6 months ago

Sorry for the spam but after trying again for quite some time I also get rate limited .. I havent sent any requests in 48 hours +

Traceback (most recent call last): File "C:\Users\Dator\Documents\App\Kantspel Omni\DDGS-test.py", line 4, in for r in ddgs.text('live free or die', region='wt-wt', safesearch='off', timelimit='y', max_results=10): File "C:\Users\Dator\AppData\Local\Programs\Python\Python310\lib\site-packages\duckduckgo_search\duckduckgo_search.py", line 106, in text for i, result in enumerate(results, start=1): File "C:\Users\Dator\AppData\Local\Programs\Python\Python310\lib\site-packages\duckduckgo_search\duckduckgo_search.py", line 134, in _text_api vqd = self._get_vqd(keywords) File "C:\Users\Dator\AppData\Local\Programs\Python\Python310\lib\site-packages\duckduckgo_search\duckduckgo_search.py", line 64, in _get_vqd resp = self._get_url("POST", "https://duckduckgo.com", data={"q": keywords}) File "C:\Users\Dator\AppData\Local\Programs\Python\Python310\lib\site-packages\duckduckgo_search\duckduckgo_search.py", line 49, in _get_url raise RateLimitException(f"_get_url() {url}") duckduckgo_search.exceptions.RateLimitException: _get_url() https://duckduckgo.com [Finished in 1.4s]

EDIT: I've tried a VPN, a proxy and another wifi (at work), no difference.

deedy5 commented 6 months ago

update to v3.9.11

zimpstar commented 6 months ago

@deedy5 Sorry if I was unclear, this is indeed v3.9.11

Here is the full error log. Please note that I have tried this on multiple proxies, machines, residential wi-fis and now at a local café as well. I called a friend and had him run it, and he got the same error.

import logging
from duckduckgo_search import DDGS

logging.basicConfig(level=logging.DEBUG)

logger = logging.getLogger(__name__)

with DDGS() as ddgs:
    try:
        for r in ddgs.text('live free or die', region='wt-wt', safesearch='off', timelimit='y', max_results=10):
            print(r)
    except Exception as e:
        logger.error("An error occurred: %s", str(e), exc_info=True)

LOG

DEBUG:httpx:load_ssl_context verify=False cert=None trust_env=True http2=False DEBUG:httpcore.connection:connect_tcp.started host='duckduckgo.com' port=443 local_address=None timeout=10 socket_options=None DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x0000013A26EA4F70> DEBUG:httpcore.connection:start_tls.started ssl_context=<ssl.SSLContext object at 0x0000013A26DCB1C0> server_hostname='duckduckgo.com' timeout=10 DEBUG:httpcore.connection:start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x0000013A26EA4F40> DEBUG:httpcore.http2:send_connection_init.started request=<Request [b'POST']> DEBUG:httpcore.http2:send_connection_init.complete DEBUG:httpcore.http2:send_request_headers.started request=<Request [b'POST']> stream_id=1 DEBUG:hpack.hpack:Adding (b':method', b'POST') to the header table, sensitive:False, huffman:True DEBUG:hpack.hpack:Encoding 3 with 7 bits DEBUG:hpack.hpack:Adding (b':authority', b'duckduckgo.com') to the header table, sensitive:False, huffman:True DEBUG:hpack.hpack:Encoding 1 with 6 bits DEBUG:hpack.hpack:Encoding 11 with 7 bits DEBUG:hpack.hpack:Adding (b':scheme', b'https') to the header table, sensitive:False, huffman:True DEBUG:hpack.hpack:Encoding 7 with 7 bits DEBUG:hpack.hpack:Adding (b':path', b'/') to the header table, sensitive:False, huffman:True DEBUG:hpack.hpack:Encoding 4 with 7 bits DEBUG:hpack.hpack:Adding (b'accept', b'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7') to the header table, sensitive:False, huffman:True DEBUG:hpack.hpack:Encoding 19 with 6 bits DEBUG:hpack.hpack:Encoding 101 with 7 bits DEBUG:hpack.hpack:Adding (b'accept-language', b'en-US,en;q=0.9') to the header table, sensitive:False, huffman:True DEBUG:hpack.hpack:Encoding 17 with 6 bits DEBUG:hpack.hpack:Encoding 11 with 7 bits DEBUG:hpack.hpack:Adding (b'accept-encoding', b'gzip, deflate, br') to the header table, sensitive:False, huffman:True DEBUG:hpack.hpack:Encoding 16 with 6 bits DEBUG:hpack.hpack:Encoding 13 with 7 bits DEBUG:hpack.hpack:Adding (b'referer', b'https://duckduckgo.com/') to the header table, sensitive:False, huffman:True DEBUG:hpack.hpack:Encoding 51 with 6 bits DEBUG:hpack.hpack:Encoding 17 with 7 bits DEBUG:hpack.hpack:Adding (b'user-agent', b'Mozilla/5.0 (Macintosh; Intel Mac OS X 140) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36') to the header table, sensitive:False, huffman:True DEBUG:hpack.hpack:Encoding 58 with 6 bits DEBUG:hpack.hpack:Encoding 87 with 7 bits DEBUG:hpack.hpack:Adding (b'content-length', b'18') to the header table, sensitive:False, huffman:True DEBUG:hpack.hpack:Encoding 28 with 6 bits DEBUG:hpack.hpack:Encoding 2 with 7 bits DEBUG:hpack.hpack:Adding (b'content-type', b'application/x-www-form-urlencoded') to the header table, sensitive:False, huffman:True DEBUG:hpack.hpack:Encoding 31 with 6 bits DEBUG:hpack.hpack:Encoding 24 with 7 bits DEBUG:hpack.hpack:Encoded header block to b'\x83A\x8b\x92\xd2u\x92\xd2u\x98\xeb\x90\xf4\xff\x87\x84S\xe5I|\xa5\x89\xd3M\x1fC\xae\xba\x0cA\xa4\xc7\xa9\x8f3\xa6\x9a?\xdf\x9ah\xfa\x1du\xd0b\r&=Ly\xa6\x8f\xbe\xd0\x01w\xfe\x8dH\xe6+\x03\xeei~\x8dH\xe6+\x1e\x0b\x1d\x7fF\xa4s\x15\x81\xd7T\xdf,|\xfd\xf6\x80\x0b\xbd\xf4:\xeb\xa0\xc4\x1aLz\x98A\xa6\xa8\xb2,$\x9cuL\xbe\xf0F\xcf\xdfh\x00\xbb\xbfQ\x8b-Kp\xdd\xf4Z\xbe\xfb@\x05\xdfP\x8d\x9b\xd9\xab\xfaRB\xcb@\xd2\xa5#\xb3s\x91\x9d)\xad\x17\x18bKI\xd6KI\xd6c\xaeC\xd2\xc7z\xd7\xd0\x7ff\xa2\x81\xb0\xda\xe0S\xfa\xd02\x1a\xa4\x9d\x13\xfd\xa9\x92\xa4\x96\x854\x0c\x8aj\xdc\xa7\xe2\x81j \x7fjC]t\x17\x91c\xccd\xb0\xdb.\xae\xcb\x8a\x7fY\xb1\xef\xd1\x9f\xe9J\r\xd4\xaab):\x9f\xfbR\xf4\xf6\x1e\x92\xb0\x10\xbe\xb8\x17\x02\xe0Sp\xe5\x1d\x86a\xb6]]\x97?\\x82\x0b\xdf\x98\x1du\xd0b\r&=Ly[\xc7\x8f\x0bJ{)Z\xdb(-D<\x85\x93' DEBUG:httpcore.http2:send_request_headers.complete DEBUG:httpcore.http2:send_request_body.started request=<Request [b'POST']> stream_id=1 DEBUG:httpcore.http2:send_request_body.complete DEBUG:httpcore.http2:receive_response_headers.started request=<Request [b'POST']> stream_id=1 DEBUG:httpcore.http2:receive_remote_settings.started DEBUG:httpcore.http2:receive_remote_settings.complete return_value=<RemoteSettingsChanged changed_settings:{ChangedSetting(setting=SettingCodes.MAX_CONCURRENT_STREAMS, original_value=None, new_value=64), ChangedSetting(setting=SettingCodes.INITIAL_WINDOW_SIZE, original_value=65535, new_value=65536), ChangedSetting(setting=SettingCodes.MAX_FRAME_SIZE, original_value=16384, newvalue=16777215)}> DEBUG:hpack.hpack:Decoding b' H\x03202v\x84\xaacU\xe7a\x96\xdc4\xfd(\x07\xd4\xbeR( \t\x95\x02\x0b\x82f\xe3l\xa9\x8bF\xff\x92I|\xa5\x89\xd3M\x1fj\x12q\xd8\x82\xa6\x0e\x1b\xf0\xac\xf7\x00\x8aAl\xee[\x16I\xa95S\x7f\x99I\xd2:>\xe4\xb6\xc8\x1d\x7f\xeeB\xa0\x90\x7f7FN\xa5\xaaE7\x9d#\xa3\xf9\x00\x91Bl1\x12\xb2l\x1dH\xac\xf6%d\x14\x96\xd8d\xfa\x8c\xa4~V\x1c\xc5\x81\x90\xb6\xcb\x80\x00?\x00\x8d\xac\xb6Rd \xc7\xa9\x0bVz\x0cO\x8e5I-\x85BV!\xe7=\x89\x83\xfa\xfe\xff\x00\x90!\xeaIjJ\xc8)-\xb0\xc9\xf4\xb5g\xa0\xc4\xf5\xff\x9e\t\x90\xb2\x8e\xda\x12\xb2,"\x9f\xea\xa3\xd4\xf4\xa7\xda\x84=U\x14\x89Y\x16\x11E\'JkE\xc6\x18\x92\xd2u\x92\xd2u\x98\xeb\x90\xf4\xa9:SZ.0\xc7\xca\xf2ZN\xb2ZN\xb3\x1dr\x1e\x95\'JkE\xc6\x18\x92\xd2u\x92\xd2u\x98\xf3L\xd0\xbc\xf49\x1d\x17\x96Q\xd0h?\x83\x8e\xc9c\x98\x94\xf7\x94\xd4\x8eT\xa5\xc4\xf8\x1c\xc8\xf1\xec\x9e\xc7"\xe7\xa8\xc7\xa9\x85\'JkE\xc6\x18Ev\x14rWa\xbb\x8c\x9e\x97!\xe9S\xedJGQ\xa5\x12\xb2,"\x8aN\x94\xd6\x8b\x8c1%\xa4\xeb%\xa4\xeb1\xd7!\xe9Rt\xa6\xb4\a\x8f\x95\xe4\xb4\x9dd\xb4\x9df:\xe4=N\x94\xd6\x8b\x8c1%\xa4\xeb%\xa4\xeb1\xe6\x99\xa1y\xe8r:/,\xa3\xa0\xd0\x7f\x07\x1d\x92\xc71)\xef)\xa9\x1c\xa9K\x89\xf09\x91\xe3\xd9=\x8eE\xcfQ\x8fS\nN\x94\xd6\x8b\x8c0\x8a\xec(\xe4\xae\xc3w\x19=.C\xd2\xa7\xda\x94\x96C\rdXE\x14\x9d)\xad\x17\x18bKI\xd6KI\xd6c\xaeC\xd2\xa4\xe9Mh\xb8\xc3\x1f+\xc9i:\xc9i:\xccu\xc8zT\x9d)\xad\x17\x18bKI\xd6KI\xd6c\xcd3B\xf3\xd0\xe4t^YGA\xa0\xfe\x0e;%\x8ebS\xdeSR9R\x97\x13\xe0s#\xc7\xb2{\x1c\x8b\x9e\xa3\x1e\xa6\x14\x9d)\xad\x17\x18a\x15\xd8Q\xc9]\x86\xee2z\\x87\xa5O\xb5\x10K\rZVE\x84R:\x0f\x1d\xc5\x14\x9d)\xad\x17\x18bKI\xd6KI\xd6c\xaeC\xd2\xa4\xe9Mh\xb8\xc3\x1f+\xc9i:\xc9i:\xccu\xc8zT\x9d)\xad\x17\x18bKI\xd6KI\xd6c\xcd3B\xf3\xd0\xe4t^YGA\xa0\xfe\x0e;%\x8ebS\xdeSR9R\x97\x13\xe0s#\xc7\xb2{\x1c\x8b\x9e\xa3\x1e\xa6\x14\x9d)\xad\x17\x18a\x15\xd8Q\xc9]\x86\xee2z\\x87\xa5O\xf5mH\x1c\xa5X\xd5Pj\x8b\xfe\x94\xffV\xd4\x81\xcaU\x8b\xdct\x7f\xa5>\xd4\x94\xf5%dXE$\x1aGqE\'JkE\xc6\x18\x92\xd2u\x92\xd2u\x98\xeb\x90\xf4\xa9:SZ.0\xc7\xca\xf2ZN\xb2ZN\xb3\x1dr\x1e\x95\'JkE\xc6\x18\x92\xd2u\x92\xd2u\x98\xf3L\xd0\xbc\xf49\x1d\x17\x96Q\xd0h?\x83\x8e\xc9c\x98\x94\xf7\x94\xd4\x8eT\xa5\xc4\xf8\x1c\xc8\xf1\xec\x9e\xc7"\xe7\xa8\xc7\xa9\x85\'JkE\xc6\x18Ev\x14rWa\xbb\x8c\x9e\x97!\xe9S\xedCS2\xc8\xb0\x8aH4\x8e\xe2\x8aN\x94\xd6\x8b\x8c1%\xa4\xeb%\xa4\xeb1\xd7!\xe9Rt\xa6\xb4\a\x8f\x95\xe4\xb4\x9dd\xb4\x9df:\xe4=N\x94\xd6\x8b\x8c1%\xa4\xeb%\xa4\xeb1\xe6\x99\xa1y\xe8r:/,\xa3\xa0\xd0\x7f\x07\x1d\x92\xc71)\xef)\xa9\x1c\xa9K\x89\xf09\x91\xe3\xd9=\x8eE\xcfQ\x8fS\nN\x94\xd6\x8b\x8c0\x8a\xec(\xe4\xae\xc3w\x19=.C\xd2\xa7\xda\x88O\xaa\n\xb2,"\x8aN\x94\xd6\x8b\x8c1%\xa4\xeb%\xa4\xeb1\xd7!\xe9Rt\xa6\xb4\a\x8f\x95\xe4\xb4\x9dd\xb4\x9df:\xe4=N\x94\xd6\x8b\x8c1%\xa4\xeb%\xa4\xeb1\xe6\x99\xa1y\xe8r:/,\xa3\xa0\xd0\x7f\x07\x1d\x92\xc71)\xef)\xa9\x1c\xa9K\x89\xf09\x91\xe3\xd9=\x8eE\xcfQ\x8fS\nN\x94\xd6\x8b\x8c0\x8a\xec(\xe4\xae\xc3w\x19=.C\xd2\xa7\xfa\xb6\xa4\x0eR\xacj\xa85E\xffJ}\xa8x\xfa\x14\x89Y\x16\x11O\xf5Q\xea/\xfaS\xedO\x07\xb3\xa9lY\x16\x11H\xe8<w\x14\xfbP\x93\x9a\x89\x16E\x84R:\x0f\x1d\xc5\x14\x9d)\xad\x17\x18bKI\xd6KI\xd6c\xaeC\xd2\xa4\xe9Mh\xb8\xc3\x1f+\xc9i:\xc9i:\xccu\xc8zT\x9d)\xad\x17\x18bKI\xd6KI\xd6c\xcd3B\xf3\xd0\xe4t^YGA\xa0\xfe\x0e;%\x8ebS\xdeSR9R\x97\x13\xe0s#\xc7\xb2{\x1c\x8b\x9e\xa3\x1e\xa6\x14\x9d)\xad\x17\x18a\x15\xd8Q\xc9]\x86\xee2z\\x87\xa5O\xb5%\xb0t\x95dXE#\xa0\xf1\xdcQI\xd2\x9a\xd1q\x86$\xb4\x9dd\xb4\x9df:\xe4=N\x94\xd6\x8b\x8c1\xf2\xbc\x96\x93\xac\x96\x93\xac\xc7\\x87\xa5I\xd2\x9a\xd1q\x86$\xb4\x9dd\xb4\x9df<\xd34/=\x0eGE\xe5\x94t\x1a\x0f\xe0\xe3\xb2X\xe6%=\xe55#\x95)q>\x072<{\'\xb1\xc8\xb9\xea1\xeaaI\xd2\x9a\xd1q\x86\x11]\x85\x1c\x95\xd8n\xe3\'\xa5\xc8zT\xfbRS\xd9J\xc3"Lz\x94Rt\xa6\xb4\a\x89-\'Y-\'Y\x8e\xb9\x0fJ\x93\xa55\xa2\xe3\x0c|\xaf%\xa4\xeb%\xa4\xeb1\xd7!\xe9Rt\xa6\xb4\a\x89-\'Y-\'Y\x8f4\xcd\x0b\xcfC\x91\xd1ye\x1d\x06\x83\xf88\xec\x969\x89OyMH\xe5J\O\x81\xcc\x8f\x1e\xc9\xecr.z\x8cz\x98Rt\xa6\xb4\a\x84WaG%v\x1b\xb8\xc9\xe9r\x1e\x95>\xd4\x96\xc1\xd2U\x87Q\n\x84\x9e\xc4)\xfe\x90Z%\xffJ}\xa9\x18\xd0U\xad\xb0\xca\x7f\xa4\x16\x89\x7f\xd2\x9fjGA\xc9\xd5a\xd1B\xd4\x9b\xc9dX\x87\xa9%\xa9}\xff\x00\x8b\xf2\xb4\xb6\x0e\x92\xacz\xd2c\xd4\x8f\x89\xdd\x0e\x8c\x1a\xb6\xe4\xc5\x93O\x00\x8c\xf2\xb7\x94!j\xec:JD\x98\xf5\x7f\x89\x0f\xdd\'\x90\xb0GA\xc9\xd7\x00\x90\xf2\xb1\x0fRKRVO\xaa\xca\xb1\xebI\x8fR?\x85\xa8\xe8\xa8\xd2\xcb\x00\x8b\xb0\xb2\x96\xcb\x0bb\xd5\x9e\x83\x13\xd7\x85=\x86\x98\xd5\x7f\x00\x87/\x9a\xcaD\xacD\xff\x87\xa4~V\x1c\xc5\x80\x1f' DEBUG:hpack.hpack:Decoded 0, consumed 1 bytes DEBUG:hpack.table:Resizing header table to 0 from 4096 DEBUG:hpack.hpack:Decoded 8, consumed 1 bytes DEBUG:hpack.hpack:Decoded 3, consumed 1 bytes DEBUG:hpack.hpack:Decoded (b':status', <memory at 0x0000013A26D5E740>), total consumed 5 bytes, indexed True DEBUG:hpack.hpack:Decoded 54, consumed 1 bytes DEBUG:hpack.hpack:Decoded 4, consumed 1 bytes DEBUG:hpack.hpack:Decoded (b'server', b'nginx'), total consumed 6 bytes, indexed True DEBUG:hpack.hpack:Decoded 33, consumed 1 bytes DEBUG:hpack.hpack:Decoded 22, consumed 1 bytes DEBUG:hpack.hpack:Decoded (b'date', b'Sat, 09 Dec 2023 10:23:53 GMT'), total consumed 24 bytes, indexed True DEBUG:hpack.hpack:Decoded 31, consumed 1 bytes DEBUG:hpack.hpack:Decoded 18, consumed 1 bytes DEBUG:hpack.hpack:Decoded (b'content-type', b'text/html; charset=UTF-8'), total consumed 20 bytes, indexed True DEBUG:hpack.hpack:Decoded 10, consumed 1 bytes DEBUG:hpack.hpack:Decoded 25, consumed 1 bytes DEBUG:hpack.hpack:Decoded (b'server-timing', b'total;dur=79;desc="Backend Total"'), total consumed 38 bytes, indexed False DEBUG:hpack.hpack:Decoded 17, consumed 1 bytes DEBUG:hpack.hpack:Decoded 12, consumed 1 bytes DEBUG:hpack.hpack:Decoded (b'strict-transport-security', b'max-age=31536000'), total consumed 32 bytes, indexed False DEBUG:hpack.hpack:Decoded 13, consumed 1 bytes DEBUG:hpack.hpack:Decoded 14, consumed 1 bytes DEBUG:hpack.hpack:Decoded (b'permissions-policy', b'interest-cohort=()'), total consumed 30 bytes, indexed False DEBUG:hpack.hpack:Decoded 16, consumed 1 bytes DEBUG:hpack.hpack:Decoded 1309, consumed 3 bytes DEBUG:hpack.hpack:Decoded (b'content-security-policy', b"default-src 'none' ; connect-src https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; manifest-src https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; media-src https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; script-src blob: https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com 'unsafe-inline' 'unsafe-eval' ; font-src data: https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; img-src data: https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; style-src https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com 'unsafe-inline' ; object-src 'none' ; worker-src blob: ; child-src blob: https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; frame-src blob: https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; form-action https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; frame-ancestors 'self' ; base-uri 'self' ; block-all-mixed-content ;"), total consumed 1330 bytes, indexed False DEBUG:hpack.hpack:Decoded 11, consumed 1 bytes DEBUG:hpack.hpack:Decoded 9, consumed 1 bytes DEBUG:hpack.hpack:Decoded (b'x-frame-options', b'SAMEORIGIN'), total consumed 23 bytes, indexed False DEBUG:hpack.hpack:Decoded 12, consumed 1 bytes DEBUG:hpack.hpack:Decoded 9, consumed 1 bytes DEBUG:hpack.hpack:Decoded (b'x-xss-protection', b'1;mode=block'), total consumed 24 bytes, indexed False DEBUG:hpack.hpack:Decoded 16, consumed 1 bytes DEBUG:hpack.hpack:Decoded 5, consumed 1 bytes DEBUG:hpack.hpack:Decoded (b'x-content-type-options', b'nosniff'), total consumed 24 bytes, indexed False DEBUG:hpack.hpack:Decoded 11, consumed 1 bytes DEBUG:hpack.hpack:Decoded 5, consumed 1 bytes DEBUG:hpack.hpack:Decoded (b'referrer-policy', b'origin'), total consumed 19 bytes, indexed False DEBUG:hpack.hpack:Decoded 7, consumed 1 bytes DEBUG:hpack.hpack:Decoded 7, consumed 1 bytes DEBUG:hpack.hpack:Decoded (b'expect-ct', b'max-age=0'), total consumed 17 bytes, indexed False DEBUG:httpcore.http2:receive_response_headers.complete return_value=(202, [(b'server', b'nginx'), (b'date', b'Sat, 09 Dec 2023 10:23:53 GMT'), (b'content-type', b'text/html; charset=UTF-8'), (b'server-timing', b'total;dur=79;desc="Backend Total"'), (b'strict-transport-security', b'max-age=31536000'), (b'permissions-policy', b'interest-cohort=()'), (b'content-security-policy', b"default-src 'none' ; connect-src https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; manifest-src https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; media-src https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; script-src blob: https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com 'unsafe-inline' 'unsafe-eval' ; font-src data: https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; img-src data: https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; style-src https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com 'unsafe-inline' ; object-src 'none' ; worker-src blob: ; child-src blob: https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; frame-src blob: https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; form-action https://duckduckgo.com https://*.duckduckgo.com https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ https://spreadprivacy.com ; frame-ancestors 'self' ; base-uri 'self' ; block-all-mixed-content ;"), (b'x-frame-options', b'SAMEORIGIN'), (b'x-xss-protection', b'1;mode=block'), (b'x-content-type-options', b'nosniff'), (b'referrer-policy', b'origin'), (b'expect-ct', b'max-age=0')]) INFO:httpx:HTTP Request: POST https://duckduckgo.com "HTTP/2 202 Accepted" DEBUG:httpcore.http2:receive_response_body.started request=<Request [b'POST']> stream_id=1 DEBUG:httpcore.http2:receive_response_body.complete DEBUG:httpcore.http2:response_closed.started stream_id=1 DEBUG:httpcore.http2:response_closed.complete ERROR:main:An error occurred: _get_url() https://duckduckgo.com Traceback (most recent call last): File "C:\Users\Dator\Documents\App\Kantspel Omni\DDGS-test.py", line 12, in for r in ddgs.text('live free or die', region='wt-wt', safesearch='off', timelimit='y', max_results=10): File "C:\Users\Dator\AppData\Local\Programs\Python\Python310\lib\site-packages\duckduckgo_search\duckduckgo_search.py", line 106, in text for i, result in enumerate(results, start=1): File "C:\Users\Dator\AppData\Local\Programs\Python\Python310\lib\site-packages\duckduckgo_search\duckduckgo_search.py", line 134, in _text_api vqd = self._get_vqd(keywords) File "C:\Users\Dator\AppData\Local\Programs\Python\Python310\lib\site-packages\duckduckgo_search\duckduckgo_search.py", line 64, in _get_vqd resp = self._get_url("POST", "https://duckduckgo.com", data={"q": keywords}) File "C:\Users\Dator\AppData\Local\Programs\Python\Python310\lib\site-packages\duckduckgo_search\duckduckgo_search.py", line 49, in _get_url raise RateLimitException(f"_get_url() {url}") duckduckgo_search.exceptions.RateLimitException: _get_url() https://duckduckgo.com DEBUG:httpcore.connection:close.started DEBUG:httpcore.connection:close.complete

deedy5 commented 6 months ago

All tests passed successfully. Try to reinstall package pip install -I duckduckgo_search. Try to use proxy.

kostum123 commented 6 months ago

This error continues. When I run the same code on Google Colab, Google Colab’s server IP is not blocked, but when I run it on my home computer, my home IP is not allowed. I really can’t understand this logic. Moreover, even though no request has been made from my home IP, it immediately gives an error message. I guess with the recent changes, you have started to block residential IPs as well. I hope this is a mistake because it’s very absurd to block residential users’ IPs while not blocking datacenter IPs.

deedy5 commented 6 months ago

The site blocks requests with tls-fingerprint. I had to switch from httpx to curl-cffi. Now everything works as it should, hopefully for a long time.

ShubhamNandalwar8 commented 3 months ago

The site blocks requests with tls-fingerprint. I had to switch from httpx to curl-cffi. Now everything works as it should, hopefully for a long time.

can you please tell me how did you do that?

deedy5 commented 3 months ago

I don't understand the point of the question.

ShubhamNandalwar8 commented 3 months ago

can you provide the code to switch from httpx to curl-cffi.

I don't understand the point of the question.

deedy5 commented 3 months ago

Check it out here: https://github.com/yifeikong/curl_cffi