limited scopes!!

[amad3us47]

it only gives 400 search urls how can we maximize it

https://support.google.com/websearch/thread/24227169/i-can-t-see-all-search-results-there-are-less-results-than-google-thinks?hl=en

[opsdisk]

@amad3us47 Can you provide more information? What were the values you used for the search? What was the search string query?

[opsdisk]

Is this what you're referring to?

[amad3us47]

Yeah is there any other alternative or some search engine which doesn't have google like seo and could it integrate with it ... For bug bounty script. Wayback machine could do that using Tom's waybackurls tool.. but it's a recorded indexes archive not that good.

[opsdisk]

This returned 420 URLs when I set max_search_result_urls_to_return=600, so not sure how accurate the 400 cap is.

query = "computer"

client = yagooglesearch.SearchClient(
    query,
    tbs="li:1",
    max_search_result_urls_to_return=600,
    http_429_cool_off_time_in_minutes=45,
    http_429_cool_off_factor=1.5,
    # proxy="socks5h://127.0.0.1:9050",
    verbosity=5,
    verbose_output=True,  # False (only URLs) or True (rank, title, description, and URL)
)
client.assign_random_user_agent()

urls = client.search()

len(urls)

[amad3us47]

This returned 420 URLs when I set max_search_result_urls_to_return=600, so not sure how accurate the 400 cap is.

query = "computer"

client = yagooglesearch.SearchClient(
    query,
    tbs="li:1",
    max_search_result_urls_to_return=600,
    http_429_cool_off_time_in_minutes=45,
    http_429_cool_off_factor=1.5,
    # proxy="socks5h://127.0.0.1:9050",
    verbosity=5,
    verbose_output=True,  # False (only URLs) or True (rank, title, description, and URL)
)
client.assign_random_user_agent()

urls = client.search()

len(urls)

Still won't give the max results

Google is limiting the search results (it's a feature). I will need to find new search engine or ways to fix that .

[opsdisk]

What were your search criteria that you ran into that limit?

[amad3us47]

What were your search criteria that you ran into that limit?

I was indexing some countries sites with dork (site:.pk)

[tw-evan]

I’ve also encountered this problem!

[amad3us47]

@tw-even is there any solution for this you might have encountered?

[opsdisk]

Yeah I got 399 when using the site:.pk query...might be a limitation of using the GUI and not the official search API (https://developers.google.com/custom-search/v1/overview). If that's the case, there's not much yagooglesearch can do...I'll just have to add a note in the docs not to expect more than 400. I'll keep this open for the time being.

[opsdisk]

Thanks for bringing this to my attention @amad3us47 I updated the README:

https://github.com/opsdisk/yagooglesearch?tab=readme-ov-file#max-400-results-returned