unstable api.tavily.com/search

jeremi commented 10 months ago

Over the last hour I have been trying
gpt-researcher but it seems the tavily is quite unstable. it sometimes return results, but so far, I have not been able to finish a research. There is always a search that fail. Should retry be implemented? Is there is a way to restart a research that has been interrupted?

Here is an example when it fail:

  File "/Users/xx/.virtualenvs/gpt-researcher/lib/python3.11/site-packages/tavily/tavily.py", line 47, in search
    return self._search(query, search_depth=search_depth, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xx/.virtualenvs/gpt-researcher/lib/python3.11/site-packages/tavily/tavily.py", line 41, in _search
    response.raise_for_status()  # Raises a HTTPError if the HTTP request returned an unsuccessful status code
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xx/.virtualenvs/gpt-researcher/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.tavily.com/search

assafelovic commented 10 months ago

Hey @jeremi due to high volumes we had some slow response times. After scaling up it should be better now, can you check?

jeremi commented 10 months ago

It seems good today, no issue. Thanks!

Tristandiscorder commented 10 months ago

Hmm looks like its recurring on my end though

🔎 Running research for 'Current market opinion on Samsung Electronics stock December 2023'... ERROR: Exception in ASGI application Traceback (most recent call last): File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/uvicorn/protocols/websockets/wsproto_impl.py", line 234, in run_asgi result = await self.app(self.scope, self.receive, self.send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in call return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/fastapi/applications.py", line 1106, in call await super().call(scope, receive, send) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/applications.py", line 122, in call await self.middleware_stack(scope, receive, send) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 149, in call await self.app(scope, receive, send) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in call raise exc File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in call await self.app(scope, receive, sender) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in call raise e File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in call await self.app(scope, receive, send) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 718, in call await route.handle(scope, receive, send) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 341, in handle await self.app(scope, receive, send) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 82, in app await func(session) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/fastapi/routing.py", line 325, in app await dependant.call(values) File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/backend/server.py", line 50, in websocket_endpoint report = await manager.start_streaming(task, report_type, websocket) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/gpt_researcher/utils/websocket_manager.py", line 51, in start_streaming report = await run_agent(task, report_type, websocket) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/gpt_researcher/utils/websocket_manager.py", line 63, in run_agent report = await researcher.run() ^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/gpt_researcher/master/agent.py", line 52, in run scraped_sites = await self.scrape_sites_by_query(sub_query) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/gpt_researcher/master/agent.py", line 91, in scrape_sites_by_query search_results = retriever.search(max_results=self.cfg.max_search_results_per_query) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/gpt_researcher/retrievers/tavily_search/tavily_search.py", line 43, in search results = self.client.search(self.query, search_depth="advanced", max_results=max_results) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/tavily/tavily.py", line 47, in search return self._search(query, search_depth=search_depth, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/tavily/tavily.py", line 41, in _search response.raise_for_status() # Raises a HTTPError if the HTTP request returned an unsuccessful status code ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.tavily.com/search

samgoodman commented 10 months ago

EchoingTristandiscorder, this seems to be persistent

garmanwu commented 9 months ago

File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.tavily.com/search

I have the same problem.

sameigen commented 9 months ago

This is consistently happening to me as well, @assafelovic maybe worth reopening? Not sure if there is another issue open for this

SimonB97 commented 8 months ago

Yep, same. @assafelovic this should be re-opened.

Not usable at current state.

seanmavley commented 8 months ago

Can't even ping the API endpoint at the moment. Not sure what's happening with tavily. Yet it was working earlier the other day. 😓😮‍💨

It's kinda unstable since it's failure breaks the AI chain

🤦🏾‍♂️

AryanPadmanabhan commented 7 months ago

Happening to me as well. It works once and then breaks the next time. It also uses one of my API call requests when it doesn't work @assafelovic

evangriffiths commented 7 months ago

Also getting requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.tavily.com/search. Please re-open. I get this when using both a free and paid-for API key.