duckduckgo rate limit exception

[uniqueNY85]

Has happened two times in a row this morning. Here's my terminal output: 🔎 Running research for 'latest AI developments'... 🤖 AI Research Agent 🧠 I will conduct my research based on the following queries: ['latest AI developments 2024', 'recent advancements in artificial intelligence technology 2024', 'current trends in AI research and ethics May 2024', 'latest AI developments']...

🔎 Running research for 'latest AI developments 2024'... Error: 502 Server Error: Bad Gateway for url: https://api.tavily.com/search C:\gpt_researcher\gpt_research_venv\Lib\site-packages\curl_cffi\aio.py:39: RuntimeWarning: Proactor event loop does not implement add_reader family of methods required. Registering an additional selector thread for add_reader support. To avoid this warning use: asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())

warnings.warn(PROACTOR_WARNING, RuntimeWarning)

🔎 Running research for 'recent advancements in artificial intelligence technology 2024'... ✅ Adding source url to research: https://www.forbes.com/sites/forbestechcouncil/2024/01/31/artificial-intelligence-in-2024-what-it-has-in-store-for-us/

✅ Adding source url to research: https://www.technologyreview.com/2024/01/04/1086046/whats-next-for-ai-in-2024/

✅ Adding source url to research: https://www.ibm.com/blog/artificial-intelligence-trends/

✅ Adding source url to research: https://devabit.com/blog/top-11-new-technologies-in-ai-exploring-the-latest-trends/

🤔 Researching for relevant information...

📝 Getting relevant content based on query: recent advancements in artificial intelligence technology 2024... C:\gpt_researcher\gpt_research_venv\Lib\site-packages\langchain_core_api\deprecation.py:119: LangChainDeprecationWarning: The method BaseRetriever.get_relevant_documents was deprecated in langchain-core 0.1.46 and will be removed in 0.3.0. Use invoke instead. warn_deprecated( 📃 Source: https://www.forbes.com/sites/forbestechcouncil/2024/01/31/artificial-intelligence-in-2024-what-it-has-in-store-for-us/ Title: Content: More From Forbes Artificial Intelligence In 2024: What It Has In Store For Us Shahid Hanif is the Chief Technology Officer and Founder of Shufti Pro, a biometric identity verification (IDV) solution. Artificial intelligence (AI) has led to countless technological advancements in 2023. The advent of automation and machine learning will contribute to a more developed tech ecosystem in 2024. In 2022, AI use was twice as likely in larger companies, and 80% of retail executives plan to harness AI to experience automation by 2025. This is primarily because modern AI has recently come out of its development stage, and it is now maturing slowly.

Source: https://www.ibm.com/blog/artificial-intelligence-trends/ Title: Content: The most important AI trends in 2024 2022 was the year that generative artificial intelligence (AI) exploded into the public consciousness, and 2023 was the year it began to take root in the business world. 2024 thus stands to be a pivotal year for the future of AI, as researchers and enterprises seek to establish how this evolutionary leap in technology can be most practically integrated into our everyday lives. The evolution of generative AI has mirrored that of computers, albeit on a dramatically accelerated timeline. Massive, centrally operated mainframe computers from a few players gave way to smaller, more efficient machines accessible to enterprises and research institutions. In the decades that followed, incremental advances yielded home computers that hobbyists could tinker with. In time, powerful personal computers with intuitive no-code interfaces became ubiquitous.

Source: https://www.ibm.com/blog/artificial-intelligence-trends/ Title: Content: As AI systems speed up and incorporate new streams and formats of information, they expand the possibilities for not just communication and instruction following, but also task automation. “2023 was the year of being able to chat with an AI. Multiple companies launched something, but the interaction was always you type something in and it types something back,” says Stanford’s Norvig. “In 2024, we’ll see the ability for agents to get stuff done for you. Make reservations, plan a trip, connect to other services.”

Source: https://www.technologyreview.com/2024/01/04/1086046/whats-next-for-ai-in-2024/ Title: Content: In 2024, generative AI might actually become useful for the regular, non-tech person, and we are going to see more people tinkering with a million little AI models. State-of-the-art AI models, such as GPT-4 and Gemini, are multimodal, meaning they can process not only text but images and even videos. This new capability could unlock a whole bunch of new apps. For example, a real estate agent can upload text from previous listings, fine-tune a powerful model to generate similar text with just a click of a button, upload videos and photos of new listings, and simply ask the customized AI to generate a description of the property.

Source: https://www.technologyreview.com/2024/01/04/1086046/whats-next-for-ai-in-2024/ Title: Content: MIT Technology Review What’s next for AI in 2024 Our writers look at the four hot trends to watch out for this year MIT Technology Review’s What’s Next series looks across industries, trends, and technologies to give you a first look at the future. You can read the rest of our series here. This time last year we did something reckless. In an industry where nothing stands still, we had a go at predicting the future.

Source: https://www.ibm.com/blog/artificial-intelligence-trends/ Title: Content: 6 ”Deepfaking it: America’s 2024 election coincides with AI boom,” Reuters, 30 May 2023 7 ”How organizations can stop skyrocketing AI use from fueling anxiety,” Ernst & Young, December 2023 Tags Redefining clinical trials: Adopting AI for speed, volume and diversity 8 min read - By using AI in your talent acquisition process, you can reduce time-to-hire, improve candidate quality, and increase inclusion and diversity. Revolutionize your talent acquisition strategy: How AI can help you find the right candidates faster 2 min read - Our leading artificial intelligence (AI) solution is designed to help you find the right candidates faster and more efficiently. Enterprises’ best bet for the future: Securing generative AI 4 min read - As AI transforms and redefines how businesses operate and how customers interact with them, trust in technology must be built. More from Artificial intelligence

Source: https://www.forbes.com/sites/forbestechcouncil/2024/01/31/artificial-intelligence-in-2024-what-it-has-in-store-for-us/ Title: Content: Debunking The Myth As discussed above, there are speculations about AI facing a downturn in 2024. First, it needs to be clarified that this opinion has multiple aspects. Looking at the AI market's growth from an investor’s point of view, there will be a downturn, primarily because of the immense investments flowing into the sector. Technology maturation is a time-intensive process that was totally ignored by tech giants in 2023. The potential that 2024 holds is immense. AI will enter early maturation, and the world will see the development of real-world applications. An interesting debate that sparks here is that of AI eliminating significant jobs. By 2025, AI is expected to eliminate 85 million jobs but create 97 million new ones, resulting in a net gain of 12 million, which rejects the myths of this aspect being negative. New FBI Warning As Hackers Strike: Email Senders Must Do This 1 Thing 2 Obvious Signs Of Workplace Gaslighting From A Psychologist

Source: https://www.ibm.com/blog/artificial-intelligence-trends/ Title: Content: Like many ongoing developments, this underscores how the dangers of generative AI rise almost linearly with its capabilities. With great power comes great responsibility. Moving forward As we proceed through a pivotal year in artificial intelligence, understanding and adapting to emerging trends is essential to maximizing potential, minimizing risk and responsibly scaling generative AI adoption. 1 “Gartner Places Generative AI on the Peak of Inflated Expectations on the 2023 Hype Cycle for Emerging Technologies,” Gartner, 16 August 2023 2 ”Deloitte’s State of Generative AI in the Enteprrise Quarter one report,” Deloitte, January 2024 3 ”What to Expect in AI in 2024,” Stanford University, 8 December 2023 4 ”Q&A: UW researcher discusses just how much energy ChatGPT uses,” University of Washington, 27 July 2023 5 “Generative AI in the Enterprise,” O’Reilly, 28 November 2023 6 ”Deepfaking it: America’s 2024 election coincides with AI boom,” Reuters, 30 May 2023

🔎 Running research for 'current trends in AI research and ethics May 2024'... ✅ Adding source url to research: https://sloanreview.mit.edu/article/five-key-trends-in-ai-and-data-science-for-2024/

✅ Adding source url to research: https://www.linkedin.com/pulse/top-5-artificial-intelligence-ai-trends-2024-bernard-marr-xqide

✅ Adding source url to research: https://www.coursera.org/articles/ai-trends

✅ Adding source url to research: https://www.techtarget.com/searchenterpriseai/tip/9-top-AI-and-machine-learning-trends

✅ Adding source url to research: https://unu.edu/macau/blog-post/navigating-latest-trends-ai-ethics

🤔 Researching for relevant information...

📝 Getting relevant content based on query: current trends in AI research and ethics May 2024... 📃 Source: https://unu.edu/macau/blog-post/navigating-latest-trends-ai-ethics Title: Content: Navigating Latest Trends in AI Ethics Insights from keynote speakers shed light on the current trends in AI ethics, spanning philosophical, legal, and business perspectives. Share Navigating Latest Trends in AI Ethics

Source: https://unu.edu/macau/blog-post/navigating-latest-trends-ai-ethics Title: Content: Share Navigating Latest Trends in AI Ethics Artificial Intelligence's (AI) rise in the Fourth Industrial Revolution promises significant economic gains, yet raise ethical dilemmas, from data bias and misinformation to intellectual property concerns. Recognizing the urgency for ethical oversight, UNU-Macau, in collaboration with UNESCO and the University of Macau, organized the international workshop titled "Ethical AI: Pioneering Progress in the Asia-Pacific". The workshop aims to facilitate multi-stakeholder dialogue, uniting experts, policymakers, and stakeholders to explore the ethical dimensions of AI and seeks to catalyze collective efforts toward responsible AI development and deployment in the region. This article synthesizes insights from three keynote speakers of the workshop, exploring the latest trends in AI ethics from philosophical, legal, and business perspectives. AI Ethics from the Ground Up: Toward a New Theory of AI Ethics

Source: https://sloanreview.mit.edu/article/five-key-trends-in-ai-and-data-science-for-2024/ Title: Content: Topics AI in Action Carolyn Geason-Beissel/MIT SMR | Getty Images Artificial intelligence and data science became front-page news in 2023. The rise of generative AI, of course, drove this dramatic surge in visibility. So, what might happen in the field in 2024 that will keep it on the front page? And how will these trends really affect businesses? During the past several months, we’ve conducted three surveys of data and technology executives. Two involved MIT’s Chief Data Officer and Information Quality Symposium attendees — one sponsored by Amazon Web Services (AWS) and another by Thoughtworks. The third survey was conducted by Wavestone, formerly NewVantage Partners, whose annual surveys we’ve written about in the past. In total, the new surveys involved more than 500 senior executives, perhaps with some overlap in participation. Get Updates on Leading With AI and Data

Source: https://www.coursera.org/articles/ai-trends Title: Content: Read more: AI Ethics: What It Is and Why It Matters Start learning AI today Enroll in one of Courseraâs most popular courses AI for Everyone, from DeepLearning.AI. Youâll learn what artificial intelligence is, its impact on our lives, and how it can be applied to your job function. Start today wi th a free 7-day trial of Coursera Plus. Build job-ready skills with a Coursera Plus subscription Article sources PricewaterhouseCoopers. â2024 AI Business Predictions, https://www.pwc.com/us/en/tech-effect/ai-analytics/ai-predictions.html.âThe economic potential of generative AI: The next productivity frontier, https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/the-economic-potential-of-generative-ai-the-next-productivity-frontier# introduction.â3 big AI trends to watch in 2024, https://news.microsoft.com/three-big-ai-trends-to-watch-in-2024/.âERROR: Exception in ASGI application Traceback (most recent call last): File "C:\gpt_researcher\gpt-researcher\gpt_researcher\retrievers\tavily_search\tavily_search.py", line 45, in search results = self.client.search(self.query, search_depth="advanced", max_results=max_results) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\tavily\tavily.py", line 48, in search return self._search(query, search_depth=search_depth, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\tavily\tavily.py", line 42, in _search response.raise_for_status() # Raises a HTTPError if the HTTP request returned an unsuccessful status code ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\requests\models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.tavily.com/search

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\uvicorn\protocols\websockets\websockets_impl.py", line 240, in run_asgi result = await self.app(self.scope, self.asgi_receive, self.asgi_send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\uvicorn\middleware\proxy_headers.py", line 69, in call return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\fastapi\applications.py", line 1054, in call await super().call(scope, receive, send) File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\starlette\applications.py", line 123, in call await self.middleware_stack(scope, receive, send) File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\starlette\middleware\errors.py", line 151, in call await self.app(scope, receive, send) File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\starlette\middleware\exceptions.py", line 65, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\starlette_exception_handler.py", line 64, in wrapped_app raise exc File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\starlette_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\starlette\routing.py", line 756, in call await self.middleware_stack(scope, receive, send) File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\starlette\routing.py", line 776, in app await route.handle(scope, receive, send) File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\starlette\routing.py", line 373, in handle await self.app(scope, receive, send) File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\starlette\routing.py", line 96, in app await wrap_app_handling_exceptions(app, session)(scope, receive, send) File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\starlette_exception_handler.py", line 64, in wrapped_app raise exc File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\starlette_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\starlette\routing.py", line 94, in app await func(session) File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\fastapi\routing.py", line 348, in app await dependant.call(values) File "C:\gpt_researcher\gpt-researcher\backend\server.py", line 52, in websocket_endpoint report = await manager.start_streaming(task, report_type, websocket) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt-researcher\backend\websocket_manager.py", line 57, in start_streaming report = await run_agent(task, report_type, websocket) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt-researcher\backend\websocket_manager.py", line 74, in run_agent report = await researcher.run() ^^^^^^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt-researcher\backend\report_type\basic_report\basic_report.py", line 17, in run await researcher.conduct_research() File "C:\gpt_researcher\gpt-researcher\gpt_researcher\master\agent.py", line 80, in conduct_research self.context = await self.get_context_by_search(self.query) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt-researcher\gpt_researcher\master\agent.py", line 149, in get_context_by_search context = await asyncio.gather([self.process_sub_query(sub_query) for sub_query in sub_queries]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt-researcher\gpt_researcher\master\agent.py", line 163, in process_sub_query scraped_sites = await self.scrape_sites_by_query(sub_query) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt-researcher\gpt_researcher\master\agent.py", line 199, in scrape_sites_by_query search_results = retriever.search( ^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt-researcher\gpt_researcher\retrievers\tavily_search\tavily_search.py", line 51, in search search_response = ddg.text(self.query, region='wt-wt', max_results=max_results) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\duckduckgo_search\duckduckgo_search.py", line 57, in text return self._run_async_in_thread(super().text(args, kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\duckduckgo_search\duckduckgo_search.py", line 53, in _run_async_in_thread result = future.result() ^^^^^^^^^^^^^^^ File "C:\Program Files\Python312\Lib\concurrent\futures_base.py", line 456, in result return self.get_result() ^^^^^^^^^^^^^^^^^^^ File "C:\Program Files\Python312\Lib\concurrent\futures_base.py", line 401, in get_result raise self._exception File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\duckduckgo_search\duckduckgo_search_async.py", line 167, in text results = await self._text_api(keywords, region, safesearch, timelimit, max_results) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\duckduckgo_search\duckduckgo_search_async.py", line 258, in _text_api raise e File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\duckduckgo_search\duckduckgo_search_async.py", line 253, in _text_api await asyncio.gather(*tasks) File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\duckduckgo_search\duckduckgo_search_async.py", line 229, in _text_api_page resp_content = await self._aget_url("GET", "https://links.duckduckgo.com/d.js", params=payload) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\gpt_researcher\gpt_research_venv\Lib\site-packages\duckduckgo_search\duckduckgo_search_async.py", line 124, in _aget_url raise RatelimitException(f"{resp.url} {resp.status_code} Ratelimit") duckduckgo_search.exceptions.RatelimitException: https://links.duckduckgo.com/d.js?q=latest+AI+developments+2024&kl=wt-wt&l=wt-wt&p=&s=0&df=&vqd=4-332229213911005882589792712173234275436&ex=-1 202 Ratelimit INFO: connection closed

[arsaboo]

Relevant discussion in the upstream library.

[0xj7r]

Also coming across these errors today. Would be great to add retries with exponential backoffs (tenacity or something similar)

[0xj7r]

Solved with pip install -U duckduckgo_search to v5.3.1

[assafelovic]

Thanks @0xj7r this fixed it. Also improved the fallback mechanism here: https://github.com/assafelovic/gpt-researcher/commit/d12ee5d3889ba0cd09e236c16f01369ab2fd2151

[0xj7r]

@assafelovic I know this is marked as closed now but still being rate-limited even with bump to 5.3.1. Upstream issue still open: https://github.com/deedy5/duckduckgo_search/issues/213

[oktaydbk54]

I also get the same error. This problem is experienced in all versions of duckduckgo. Unfortunately, since there is so much duckduckgo usage, I think they have introduced something like a daily requests quota. I tried to send it using a proxy, but I couldn't do it properly. This problem can be solved if the request is sent using a proxy.

[assafelovic]

Should i revert back to Tavily as default?

[oktaydbk54]

Frankly, I am in favor of duckduckgo being the first choice. If it encounters an error, such as ratelimiterror, then it may be better to write the same request so that it can be sent to tavily.

[assafelovic]

@oktaydbk54 but then it would require a prequisite of adding a tavily env at the start

[oktaydbk54]

@oktaydbk54 but then it would require a prequisite of adding a tavily env at the start

Honestly, a system like this can be written. If duckduckgo receives an error, it can be redirected to another free resource, yahoo search, but the biggest thing I have noticed lately is that opensource web search engine systems are decreasing and I have tried many resources, none of them are successful. So maybe tavily can be used for the 3rd option, and the first 1000 credits are free, which isn't bad.

[assafelovic]

@oktaydbk54 but again, even if it’s the third option it would require to sign up before use. Which kind of eliminates the point of not using it as a default. But maybe yahoo+ddgo can work

[oktaydbk54]

@assafelovic Then the best choice may be to use duckduckgo and yandex search. I'm currently using duckduckgo-search==5.3.0b4 version. Many forums say there is no problem with this version. If I encounter any problems, I will let you know.

[assafelovic]

Perfect lmk if that version works

[oktaydbk54]

@assafelovic

I have been using duckduckgo-search==5.3.0b4 version for about 2 days and I am using a multi-agent system. I have not encountered any errors yet, but I use it as follows when sending requests.

DDGS().text(f"What is the annual revenue of {company_name}?",backend='api', max_results=5)

Here I chose the backend structure as the API. I recommend you try this version.

[assafelovic]

Thanks I just pushed a revert to Tavily but will give it a go

[danieldekay]

Second that: Tavily is nice, but not always the best. I have played with it for a while, and compared to other search engines (in my case bing) - and they aren't always the best. Options are good, and even necessary.

[assafelovic]

@danieldekay there are many search engine options here: https://docs.gptr.dev/docs/gpt-researcher/retrievers What would you recommend should be the default?

[danieldekay]

@assafelovic, yes, thank you! I only discovered the web interface last night, and only after digging into the code I realized how many more options there are, unexposed to the web front end.

[danieldekay]

I kind of agree with your move to Tavily as default, as it always provides a few relevant hits, often more relevant than a pure google search.

I played with google yesterday, and found it has quite some false hits (compared to Tavily), but this might be approachable with a better query builder and a pre-screener agent...