search

explodinggradients / ragas

Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines
https://docs.ragas.io
Apache License 2.0
5.83k stars 551 forks source link

Timeout Error When Evaluating answer_similarity and answer_correctness in Ragas #447

Closed CZT0 closed 1 month ago

CZT0 commented 6 months ago

Describe the bug When attempting to evaluate answer_similarity and answer_correctness using the Ragas framework, I encounter a timeout error. While I can successfully retrieve metrics for context_relevancy and context_recall, the progress bar stalls for a long time and eventually times out when evaluating answer_similarity and answer_correctness.

Ragas version: ragas-0.0.23.dev37+g041b20c Python version: 3.11 DATA: 63521_1.json

Code to Reproduce

import os
import json
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_similarity

from config import OPENAI_API_KEY, OPENAI_BASE_URL

os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY
os.environ["OPENAI_API_BASE"] = OPENAI_BASE_URL

def evaluate_ragas(file_path):
    with open(file_path, 'r', encoding='utf-8') as file:
        data = json.load(file)

    dataset = Dataset.from_dict(data)

    result = evaluate(
            dataset,
            metrics=[answer_similarity],
    )
    print(result)

if __name__ == '__main__':
    file_path = "63521_1.json"
    evaluate_ragas(file_path)

Error trace

/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import ChatOpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import AzureChatOpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import ChatVertexAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import AzureOpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import OpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import VertexAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.embeddings import AzureOpenAIEmbeddings`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.embeddings import FastEmbedEmbeddings`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.embeddings import OpenAIEmbeddings`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain_core/_api/deprecation.py:115: LangChainDeprecationWarning: The class `ChatOpenAI` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use langchain_openai.ChatOpenAI instead.
  warn_deprecated(
Evaluating:   0%|          | 0/5 [07:32<?, ?it/s]
Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 206, in connect_tcp
    sock = socket.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 851, in create_connection
    raise exceptions[0]
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 836, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
    yield
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 228, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request
    raise exc
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request
    response = connection.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectTimeout: [Errno 60] Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 877, in _request
    response = self._client.send(
               ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 901, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 929, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 966, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 1002, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 227, in handle_request
    with map_httpcore_exceptions():
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 206, in connect_tcp
    sock = socket.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 851, in create_connection
    raise exceptions[0]
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 836, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
    yield
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 228, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request
    raise exc
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request
    response = connection.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectTimeout: [Errno 60] Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 877, in _request
    response = self._client.send(
               ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 901, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 929, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 966, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 1002, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 227, in handle_request
    with map_httpcore_exceptions():
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 206, in connect_tcp
    sock = socket.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 851, in create_connection
    raise exceptions[0]
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 836, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
    yield
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 228, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request
    raise exc
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request
    response = connection.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectTimeout: [Errno 60] Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 877, in _request
    response = self._client.send(
               ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 901, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 929, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 966, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 1002, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 227, in handle_request
    with map_httpcore_exceptions():
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout: [Errno 60] Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/evaluation.py", line 176, in evaluate
    raise e
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/evaluation.py", line 159, in evaluate
    results = executor.results()
              ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/executor.py", line 118, in results
    raise e
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/executor.py", line 114, in results
    r = future.result()
        ^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/executor.py", line 36, in wrapped_callable
    return counter, callable(*args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/metrics/base.py", line 75, in score
    raise e
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/metrics/base.py", line 71, in score
    score = self._score(row=row, callbacks=group_cm)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/metrics/_answer_similarity.py", line 70, in _score
    embeddings_1 = np.array(self.embeddings.embed_documents(ground_truths))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain_community/embeddings/openai.py", line 668, in embed_documents
    return self._get_len_safe_embeddings(texts, engine=engine)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain_community/embeddings/openai.py", line 494, in _get_len_safe_embeddings
    response = embed_with_retry(
               ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain_community/embeddings/openai.py", line 116, in embed_with_retry
    return embeddings.client.create(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/resources/embeddings.py", line 106, in create
    return self._post(
           ^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 1088, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 853, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 884, in _request
    return self._retry_request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 958, in _retry_request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 884, in _request
    return self._retry_request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 958, in _retry_request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 893, in _request
    raise APITimeoutError(request=request) from err
openai.APITimeoutError: Request timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jellow/code/python/summary_tree/data_process/SQuALITY/计算ragas.py", line 40, in <module>
    evaluate_ragas("63521", result_id="1")
  File "/Users/jellow/code/python/summary_tree/data_process/SQuALITY/计算ragas.py", line 24, in evaluate_ragas
    result = evaluate(
             ^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/evaluation.py", line 178, in evaluate
    result = Result(
             ^^^^^^^
  File "<string>", line 6, in __init__
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/evaluation.py", line 207, in __post_init__
    for cn in self.scores[0].keys():
              ~~~~~~~~~~~^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 2795, in __getitem__
    return self._getitem(key)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 2779, in _getitem
    pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/datasets/formatting/formatting.py", line 583, in query_table
    _check_valid_index_key(key, size)
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/datasets/formatting/formatting.py", line 526, in _check_valid_index_key
    raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 0 is out of bounds for size 0

Expected behavior I expected the answer_similarity and answer_correctness evaluations to succeed without a timeout error, similar to the other metrics.

Additional context In my use of Ragas for evaluation, other metrics like context_relevancy and context_recall work normally. The issue arises solely when evaluating answer_similarity and answer_correctness.

violaluciano commented 6 months ago

I'm also having the same issue

dylanjcastillo commented 4 months ago

I'm also having the same issue for answer_correctness and answer_similarity.

I'm getting the following error:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/executor.py", line 93, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/executor.py", line 81, in _aresults
    raise e
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/executor.py", line 76, in _aresults
    r = await future
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/executor.py", line 36, in sema_coro
    return await coro
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/executor.py", line 109, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/metrics/base.py", line 91, in ascore
    raise e
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/metrics/base.py", line 87, in ascore
    score = await self._ascore(row=row, callbacks=group_cm, is_async=is_async)
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/metrics/_answer_similarity.py", line 65, in _ascore
    embedding_1 = np.array(await self.embeddings.embed_text(ground_truth))
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/embeddings/base.py", line 23, in embed_text
    embs = await self.embed_texts([text], is_async=is_async)
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/embeddings/base.py", line 33, in embed_texts
    return await aembed_documents_with_retry(texts)
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped
    return await fn(*args, **kwargs)
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 47, in __call__
    do = self.iter(retry_state=retry_state)
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 325, in iter
    raise retry_exc.reraise()
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/tenacity/__init__.py", line 158, in reraise
    raise self.last_attempt.result()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 50, in __call__
    result = await fn(*args, **kwargs)
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/embeddings/base.py", line 66, in aembed_documents
    return await self.embeddings.aembed_documents(texts)
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/langchain_openai/embeddings/base.py", line 526, in aembed_documents
    return await self._aget_len_safe_embeddings(texts, engine=engine)
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/langchain_openai/embeddings/base.py", line 435, in _aget_len_safe_embeddings
    token = encoding.encode(
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/tiktoken/core.py", line 116, in encode
    if match := _special_token_regex(disallowed_special).search(text):
TypeError: expected string or buffer
Traceback (most recent call last):
  File "/Users/dcast/Documents/GitHub/all-about-rag/src/scripts/evaluate_auto_merging_ragas.py", line 67, in <module>
    result = evaluate(
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/evaluation.py", line 228, in evaluate
    raise e
  File "/Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/evaluation.py", line 210, in evaluate
    raise ExceptionInRunner()
ragas.exceptions.ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
2024-03-10 19:28:51,571 ERROR asyncio: Task was destroyed but it is pending!
task: <Task pending name='Task-48' coro=<as_completed.<locals>.sema_coro() running at /Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/executor.py:36> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py:558]>
2024-03-10 19:28:51,571 ERROR asyncio: Task was destroyed but it is pending!
task: <Task pending name='Task-43' coro=<as_completed.<locals>.sema_coro() running at /Users/dcast/Documents/GitHub/all-about-rag/.venv/lib/python3.10/site-packages/ragas/executor.py:36> wait_for=<Future pending cb=[Task.task_wakeup()]> cb=[as_completed.<locals>._on_completion() at /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/tasks.py:558]>
dylanjcastillo commented 4 months ago

Just found the issue. This was due to having null values in the ground_truth column. Interestingly, those NAs are from synthetically generated questions using generate_with_llamaindex_docs.

@shahules786 might be good to raise a warning in these cases. What do you think?

Happy to help with a PR, if that makes sense.

shahules786 commented 4 months ago

Hey @dylanjcastillo makes sense. Would love to see the PR bro