QuivrHQ / quivr

Opiniated RAG for integrating GenAI in your apps 🧠 Focus on your product rather than the RAG. Easy integration in existing products with customisation! Any LLM: GPT4, Groq, Llama. Any Vectorstore: PGVector, Faiss. Any Files. Anyway you want.
https://core.quivr.com
Other
36.6k stars 3.57k forks source link

[Bug]: rpc 'match_brain' timeout #2375

Closed zhaojunfeng closed 4 months ago

zhaojunfeng commented 7 months ago

What happened?

input any question, self._client.rpc 'match_brain', timeout exption. I deployed quivr on a host with ip '192.168.1.84', not in my local pc. But I can create new brain and upload document successfullly.

Relevant log output

2024-03-23T14:47:57.681814699Z INFO:     192.168.1.201:51499 - "GET /user HTTP/1.1" 200 OK
2024-03-23T14:47:57.941795835Z 2024-03-23 14:47:57,941:INFO - HTTP Request: POST http://103.143.248.115:8080/v1/embeddings "HTTP/1.1 200 OK"
2024-03-23T14:48:02.988374575Z INFO:     192.168.1.201:51491 - "POST /chat/6d52cbfe-e3a4-4047-a76a-674839fefac1/question/stream HTTP/1.1" 500 Internal Server Error
2024-03-23T14:48:03.000733780Z ERROR:    Exception in ASGI application
2024-03-23T14:48:03.000779832Z Traceback (most recent call last):
2024-03-23T14:48:03.000794786Z   File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
2024-03-23T14:48:03.000807672Z     yield
2024-03-23T14:48:03.000818612Z   File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 228, in handle_request
2024-03-23T14:48:03.000830752Z     resp = self._pool.handle_request(req)
2024-03-23T14:48:03.000841792Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.000852700Z   File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 216, in handle_request
2024-03-23T14:48:03.000873735Z     raise exc from None
2024-03-23T14:48:03.000908946Z   File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
2024-03-23T14:48:03.000922561Z     response = connection.handle_request(
2024-03-23T14:48:03.000933606Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.000944900Z   File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
2024-03-23T14:48:03.000957189Z     return self._connection.handle_request(request)
2024-03-23T14:48:03.000968140Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.000979158Z   File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 143, in handle_request
2024-03-23T14:48:03.000990906Z     raise exc
2024-03-23T14:48:03.001001681Z   File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 113, in handle_request
2024-03-23T14:48:03.001013278Z     ) = self._receive_response_headers(**kwargs)
2024-03-23T14:48:03.001024089Z         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.001035029Z   File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 186, in _receive_response_headers
2024-03-23T14:48:03.001046761Z     event = self._receive_event(timeout=timeout)
2024-03-23T14:48:03.001057860Z             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.001068715Z   File "/usr/local/lib/python3.11/site-packages/httpcore/_sync/http11.py", line 224, in _receive_event
2024-03-23T14:48:03.001080218Z     data = self._network_stream.read(
2024-03-23T14:48:03.001091066Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.001101929Z   File "/usr/local/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 124, in read
2024-03-23T14:48:03.001113258Z     with map_exceptions(exc_map):
2024-03-23T14:48:03.001124261Z   File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
2024-03-23T14:48:03.001163281Z     self.gen.throw(typ, value, traceback)
2024-03-23T14:48:03.001176992Z   File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
2024-03-23T14:48:03.001188812Z     raise to_exc(exc) from exc
2024-03-23T14:48:03.001201646Z httpcore.ReadTimeout: timed out
2024-03-23T14:48:03.001213749Z 
2024-03-23T14:48:03.001247423Z The above exception was the direct cause of the following exception:
2024-03-23T14:48:03.001259401Z 
2024-03-23T14:48:03.001269649Z Traceback (most recent call last):
2024-03-23T14:48:03.001280161Z   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
2024-03-23T14:48:03.001291381Z     result = await app(  # type: ignore[func-returns-value]
2024-03-23T14:48:03.001318784Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.001330398Z   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
2024-03-23T14:48:03.001341715Z     return await self.app(scope, receive, send)
2024-03-23T14:48:03.001352272Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.001376281Z   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
2024-03-23T14:48:03.001388169Z     await super().__call__(scope, receive, send)
2024-03-23T14:48:03.001398992Z   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
2024-03-23T14:48:03.001410104Z     await self.middleware_stack(scope, receive, send)
2024-03-23T14:48:03.001421138Z   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
2024-03-23T14:48:03.001432515Z     raise exc
2024-03-23T14:48:03.001457269Z   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
2024-03-23T14:48:03.001469338Z     await self.app(scope, receive, _send)
2024-03-23T14:48:03.001480078Z   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 91, in __call__
2024-03-23T14:48:03.001491195Z     await self.simple_response(scope, receive, send, request_headers=headers)
2024-03-23T14:48:03.001502298Z   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 146, in simple_response
2024-03-23T14:48:03.001513429Z     await self.app(scope, receive, send)
2024-03-23T14:48:03.001523907Z   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
2024-03-23T14:48:03.001535064Z     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2024-03-23T14:48:03.001546121Z   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
2024-03-23T14:48:03.001557507Z     raise exc
2024-03-23T14:48:03.001584449Z   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2024-03-23T14:48:03.001596615Z     await app(scope, receive, sender)
2024-03-23T14:48:03.001607326Z   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
2024-03-23T14:48:03.001618398Z     await self.middleware_stack(scope, receive, send)
2024-03-23T14:48:03.001630255Z   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
2024-03-23T14:48:03.001656778Z     await route.handle(scope, receive, send)
2024-03-23T14:48:03.001670378Z   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle
2024-03-23T14:48:03.001681821Z     await self.app(scope, receive, send)
2024-03-23T14:48:03.001692361Z   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 79, in app
2024-03-23T14:48:03.001733686Z     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2024-03-23T14:48:03.001746446Z   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
2024-03-23T14:48:03.001785612Z     raise exc
2024-03-23T14:48:03.001797238Z   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2024-03-23T14:48:03.001819895Z     await app(scope, receive, sender)
2024-03-23T14:48:03.001854535Z   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
2024-03-23T14:48:03.001867021Z     response = await func(request)
2024-03-23T14:48:03.001877938Z                ^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.001888638Z   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
2024-03-23T14:48:03.001899875Z     raw_response = await run_endpoint_function(
2024-03-23T14:48:03.001911067Z                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.001921953Z   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
2024-03-23T14:48:03.001933726Z     return await dependant.call(**values)
2024-03-23T14:48:03.001944518Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.001956361Z   File "/code/modules/chat/controller/chat_routes.py", line 259, in create_stream_question_handler
2024-03-23T14:48:03.001967830Z     gpt_answer_generator = get_answer_generator(
2024-03-23T14:48:03.001978781Z                            ^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.001989538Z   File "/code/modules/chat/controller/chat_routes.py", line 77, in get_answer_generator
2024-03-23T14:48:03.002000901Z     brain, metadata_brain = brain_service.find_brain_from_question(
2024-03-23T14:48:03.002011867Z                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.002044938Z   File "/code/modules/brain/service/brain_service.py", line 109, in find_brain_from_question
2024-03-23T14:48:03.002083344Z     list_brains = vector_store.find_brain_closest_query(user.id, question)
2024-03-23T14:48:03.002096150Z                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.002107224Z   File "/code/vectorstore/supabase.py", line 54, in find_brain_closest_query
2024-03-23T14:48:03.002118307Z     ).execute()
2024-03-23T14:48:03.002128664Z       ^^^^^^^^^
2024-03-23T14:48:03.002140615Z   File "/usr/local/lib/python3.11/site-packages/postgrest/_sync/request_builder.py", line 114, in execute
2024-03-23T14:48:03.002153273Z     r = self.session.request(
2024-03-23T14:48:03.002169852Z         ^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.002195639Z   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 814, in request
2024-03-23T14:48:03.002207467Z     return self.send(request, auth=auth, follow_redirects=follow_redirects)
2024-03-23T14:48:03.002218439Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.002229338Z   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 901, in send
2024-03-23T14:48:03.002251870Z     response = self._send_handling_auth(
2024-03-23T14:48:03.002263190Z                ^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.002273809Z   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 929, in _send_handling_auth
2024-03-23T14:48:03.002285049Z     response = self._send_handling_redirects(
2024-03-23T14:48:03.002295612Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.002306138Z   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 966, in _send_handling_redirects
2024-03-23T14:48:03.002317159Z     response = self._send_single_request(request)
2024-03-23T14:48:03.002327686Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.002338396Z   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1002, in _send_single_request
2024-03-23T14:48:03.002349362Z     response = transport.handle_request(request)
2024-03-23T14:48:03.002360167Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-23T14:48:03.002371353Z   File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 227, in handle_request
2024-03-23T14:48:03.002382892Z     with map_httpcore_exceptions():
2024-03-23T14:48:03.002393384Z   File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
2024-03-23T14:48:03.002404207Z     self.gen.throw(typ, value, traceback)
2024-03-23T14:48:03.002414793Z   File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
2024-03-23T14:48:03.002425962Z     raise mapped_exc(message) from exc
2024-03-23T14:48:03.002436353Z httpx.ReadTimeout: timed out
2024-03-23T14:48:07.592753147Z INFO:     127.0.0.1:36372 - "GET /healthz HTTP/1.1" 200 OK
^C

Twitter / LinkedIn details

No response

dosubot[bot] commented 7 months ago

Hey there @zhaojunfeng! Great to see you diving into another challenge with quivr. Let's tackle this together! πŸš€

The "httpx.ReadTimeout: timed out" error you're experiencing when trying to use the 'match_brain' rpc on quivr deployed on a host with IP '192.168.1.84' suggests a timeout issue during a network request. This error is not directly related to the ability to create a new brain or upload documents successfully, as those operations might not require the same amount of time or resources.

Given the context provided, there's no explicit mention of a 'match_brain' rpc call or its implementation details within the provided code snippet. However, the error message indicates that the issue lies in a network request that takes too long to complete, possibly due to:

  1. The 'match_brain' rpc call attempting to communicate with a service or endpoint that is slow to respond or not responding at all.
  2. The request timeout settings for the httpx client being too low for the operation's required completion time.

To address this issue, consider the following steps, although specific code changes cannot be provided due to the lack of direct references in the provided context:

Without specific details on the implementation of the 'match_brain' rpc call or the configuration of the httpx client within your application, these are general recommendations to address the timeout issue.

Regarding the similar issues in the quivr repository, there was a closed issue [Issue] Networking Issues with self-hosted Supabase docker-compose and self-hosted docker-compose Quivr on host. which was last updated on January 04, 2024. This issue was referenced by:

I finally got it working, but the networking was non-intuitive for self-hosting Supabase & Quivr on the same box.

Although the user did not provide the exact steps they took to resolve the issue, this comment indicates that they were able to solve the networking issues encountered when self-hosting Supabase and Quivr on the same machine. Other users experiencing similar problems may need to reach out directly to the user for a detailed solution or explore networking configurations that allow docker containers to communicate with the host machine or with each other.

I hope this information helps. If you need further assistance, please provide more details about your setup and the steps you've taken so far.

Sources

#### About Dosu This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a πŸ‘ if this is helpful and πŸ‘Ž if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.
zhaojunfeng commented 7 months ago

after add the below code in settings.py: supabase_client: Client = create_client( settings.supabase_url, settings.supabase_service_key, ClientOptions(storage_client_timeout=60, postgrest_client_timeout=60) )

 new error messages:

 2024-03-24T09:02:05.752157654Z                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2024-03-24T09:02:05.752166189Z File "/code/modules/brain/service/brain_service.py", line 109, in find_brain_from_question 2024-03-24T09:02:05.752175101Z list_brains = vector_store.find_brain_closest_query(user.id, question) 2024-03-24T09:02:05.752183658Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-03-24T09:02:05.752192054Z File "/code/vectorstore/supabase.py", line 54, in find_brain_closest_query 2024-03-24T09:02:05.752200523Z ).execute() 2024-03-24T09:02:05.752209029Z ^^^^^^^^^ 2024-03-24T09:02:05.752216912Z File "/usr/local/lib/python3.11/site-packages/postgrest/_sync/request_builder.py", line 127, in execute 2024-03-24T09:02:05.752227087Z raise APIError(r.json()) 2024-03-24T09:02:05.752240378Z postgrest.exceptions.APIError: {'code': 'PGRST001', 'details': 'no connection to the server\n', 'hint': None, 'message': 'Database client error. Retrying the connection.'}

github-actions[bot] commented 4 months ago

Thanks for your contributions, we'll be closing this issue as it has gone stale. Feel free to reopen if you'd like to continue the discussion.