search

uptrain-ai / uptrain

UpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.
https://uptrain.ai/
Apache License 2.0
2.18k stars 192 forks source link

Event loop is closed when running factual accuracy evaluations #714

Open Breamer opened 2 months ago

Breamer commented 2 months ago

I'm using a python AWS lambda function to run evaluations using Bedrock, with model Claude 3.5 sonnet and Uptrain version 0.7.1.

I'm getting an "event loop is closed" error when trying to run Factual Accuracy evaluations, other types of evaluations work fine with the same data.

Steps to reproduce the behavior:

  1. Create a lambda function using python 3.11
  2. Install uptrain
  3. Use bedrock to run a factual accuracy evaluation

  4. See error

    ERROR | uptrain.operators.language.llm:async_process_payload:103 - Error when sending request to LLM API: litellm.APIConnectionError: Event loop is closed Traceback (most recent call last): File \"/var/lang/lib/python3.11/site-packages/litellm/main.py\", line 381, in acompletion response = await init_response ^^^^^^^^^^^^^^^^^^^ File \"/var/lang/lib/python3.11/site-packages/litellm/llms/bedrock_httpx.py\", line 1669, in async_completion response = await client.post(api_base, headers=headers, data=data) # type: ignore ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/var/lang/lib/python3.11/site-packages/litellm/llms/custom_httpx/http_handler.py\", line 124, in post raise e File \"/var/lang/lib/python3.11/site-packages/litellm/llms/custom_httpx/http_handler.py\", line 98, in post response = await self.client.send(req, stream=stream) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/var/lang/lib/python3.11/site-packages/httpx/_client.py\", line 1661, in send response = await self._send_handling_auth( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/var/lang/lib/python3.11/site-packages/httpx/_client.py\", line 1689, in _send_handling_auth response = await self._send_handling_redirects( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/var/lang/lib/python3.11/site-packages/httpx/_client.py\", line 1726, in _send_handling_redirects response = await self._send_single_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/var/lang/lib/python3.11/site-packages/httpx/_client.py\", line 1763, in _send_single_request response = await transport.handle_async_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/var/lang/lib/python3.11/site-packages/httpx/_transports/default.py\", line 373, in handle_async_request resp = await self._pool.handle_async_request(req) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/var/lang/lib/python3.11/site-packages/httpcore/_async/connection_pool.py\", line 216, in handle_async_request raise exc from None File \"/var/lang/lib/python3.11/site-packages/httpcore/_async/connection_pool.py\", line 196, in handle_async_request response = await connection.handle_async_request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/var/lang/lib/python3.11/site-packages/httpcore/_async/connection.py\", line 101, in handle_async_request return await self._connection.handle_async_request(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File \"/var/lang/lib/python3.11/site-packages/httpcore/_async/http11.py\", line 142, in handle_async_request await self._response_closed() File \"/var/lang/lib/python3.11/site-packages/httpcore/_async/http11.py\", line 257, in _response_closed await self.aclose() File \"/var/lang/lib/python3.11/site-packages/httpcore/_async/http11.py\", line 265, in aclose await self._network_stream.aclose() File \"/var/lang/lib/python3.11/site-packages/httpcore/_backends/anyio.py\", line 55, in aclose await self._stream.aclose() File \"/var/lang/lib/python3.11/site-packages/anyio/streams/tls.py\", line 202, in aclose await self.transport_stream.aclose() File \"/var/lang/lib/python3.11/site-packages/anyio/_backends/_asyncio.py\", line 1202, in aclose self._transport.close() File \"/var/lang/lib/python3.11/asyncio/selector_events.py\", line 864, in close self._loop.call_soon(self._call_connection_lost, None) File \"/var/lang/lib/python3.11/asyncio/base_events.py\", line 762, in call_soon self._check_closed() File \"/var/lang/lib/python3.11/asyncio/base_events.py\", line 520, in _check_closed raise RuntimeError('Event loop is closed') RuntimeError: Event loop is closed

    2024-07-04 16:02:21.619 | ERROR | uptrain.operators.language.factual_accuracy:evaluate_local:238 - Error when processing payload at index 0: litellm.APIConnectionError: Event loop is closed

Any idea or help will be much appreciated

lazyhope commented 1 month ago

It also occurs to me when using instructor together with litellm