deedy5 / duckduckgo_search

Search for words, documents, images, videos, news, maps and text translation using the DuckDuckGo.com search engine. Downloading files and images to a local hard drive.
MIT License
937 stars 117 forks source link

ddg_images() works only once #60

Closed jonasheschl closed 1 year ago

jonasheschl commented 1 year ago

Could reproduce this issue both locally and on a remote Kaggle notebook.

Library version tested: 2.9.5 Python versions tested: 3.10.6 (native), 3.10.10 (conda)

from duckduckgo_search import ddg_images

result = ddg_images('chess rook', max_results=50)
print(result)

Running this code once returns an array of image URLs as expected. Running the script again returns an empty array. After waiting some time (~10 minutes), the script works again in Kaggle. My best guess is that ddg changed their API policy and blocks IPs upon accessing their API through scripts. This would also explain why the script fixes itself in Kaggle, as the VM might change public IP after some time.

jonasheschl commented 1 year ago

I have also tested the ddg function. It is a similar story. Though I could call it twice before it not returning anymore.

LindezaGrey commented 1 year ago

I think it has to do with the recent change for vqd. I tried to disable the cache here:

https://github.com/deedy5/duckduckgo_search/blob/90a2cd7f5951b4dc5373753d65343b75a75293ba/duckduckgo_search/utils.py#L41

And then this problem was gone. I have the same issue, that i can only do a search with a specific set of keywords once. if i change the search keywords it works (or if i wait long enough)

deedy5 commented 1 year ago

The api on the site has changed, I'm working on the problem

KirillRepinArt commented 1 year ago

Same problem here

onlyrohits commented 1 year ago

Following

Hrushikesh777 commented 1 year ago

Same Here. I have app that using this images_search. As of now i have made some temp changes that can get me images, till the fix happen.

import httpx
import json

def get_images(keywords, max_results):
    url = f"https://duckduckgo.com/?va=f&t=hg&q={keywords}&iax=images&ia=images"
    headers = {
        "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
        "accept-language": "en-US,en;q=0.9,hi;q=0.8",
        "cache-control": "max-age=0",
        "sec-ch-ua": "\"Google Chrome\";v=\"113\", \"Chromium\";v=\"113\", \"Not-A.Brand\";v=\"24\"",
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": "\"Windows\"",
        "sec-fetch-dest": "document",
        "sec-fetch-mode": "navigate",
        "sec-fetch-site": "same-origin",
        "sec-fetch-user": "?1",
        "sec-gpc": "1",
        "upgrade-insecure-requests": "1",
        "cookie": "p=-2; ah=in-en; l=in-en",
        "Referer": "https://duckduckgo.com/",
        "Referrer-Policy": "origin"
    }

    with httpx.Client() as client:
        response = client.get(url, headers=headers)

    print("Response status code:", response.status_code)
    resp = response.text

    vqd_index_start = resp.index('vqd="') + 5
    vqd_index_end = resp.index('"', vqd_index_start)
    vqd_bytes = resp[vqd_index_start:vqd_index_end]
    print(f"vqd_bytes: {vqd_bytes}")

    images_url = f"https://duckduckgo.com/i.js?o=json&q={keywords}&vqd={vqd_bytes}"

    with httpx.Client() as client:
        response = client.get(images_url, headers=headers)

    print("Response status code:", response.status_code)
    # with open("F://Python//playground//temp.json", "w") as f:
    #     json.dump(response.json(), f)

    response = response.json()
    response["results"] = response["results"][:max_results]
    return response

if __name__ == "__main__":
    response = get_images(keywords, 10)
    print(json.dumps(response, indent=4))
    # for result in response["results"]:
    #     print(result["image"])
joshgodsiff commented 1 year ago

Same problem. Following

deedy5 commented 1 year ago

Update to v3.0.2 pip install -U duckduckgo_search

Himanshu-369 commented 3 months ago

Update to v3.0.2 pip install -U duckduckgo_search

try this one, worked for my code after some alteration