Testset generation error on Bedrock

[subha-aws]

Describe the bug A clear and concise description of what the bug is.

Ragas version: 0.1.0rc2.dev26+g6a88465 Python version: 3.10.13

Code to Reproduce

###Add custom llms and embeddings
generator_llm = LangchainLLMWrapper(candidate_llm1)
critic_llm = LangchainLLMWrapper(judge_llm)
embeddings_model = LangchainEmbeddingsWrapper(titan_embeddings,RunConfig(max_retries=2, max_wait=30))

Change resulting question type distribution
testset_distribution = {
"simple": 0.25,
"reasoning": 0.5,
"multi_context": 0.0,
"conditional": 0.25,
}

splitter = TokenTextSplitter(chunk_size=1000, chunk_overlap=100)
keyphrase_extractor = keyphraseExtractor(llm=generator_llm)

docstore = InMemoryDocumentStore(
splitter=splitter,
embeddings=embeddings_model,
extractor=keyphrase_extractor,
)
test_generator = TestsetGenerator(
generator_llm=generator_llm,
critic_llm=critic_llm,
embeddings=titan_embeddings,
docstore=docstore
)

testset = test_generator.generate_with_langchain_docs(documents=documents,
test_size=10,
distributions=testset_distribution)

Error trace

/ragas/src/ragas/testset/docstore.py:250, in InMemoryDocumentStore.add_nodes(self, nodes, show_progress, desc)
248 for i, n in enumerate(nodes):
249 if i in nodes_to_embed.keys():
--> 250 n.embedding = results[nodes_to_embed[i]]
251 if i in nodes_to_extract.keys():
252 keyphrases = results[nodes_to_extract[i]]

IndexError: list index out of range

Expected behavior A clear and concise description of what you expected to happen.

Additional context langchain documents. Tried with diffrent chunk sizes and overlapps, different embedding configs and different bedrock models(cohere and titan text) I am getting with any model on Bedrock "IndexError: list index out of range" happens exactly at 50% point all the time

[shahules786]

Hey @subha-aws , thanks for sharing this bug. I have also noticed this, and will fix is ASAP

[jjmachan]

hey @subha-aws can you update to v0.1. I don't think that fixes this issue but the error trace you gave is a bit misleading I think. Need the full trace for v0.1. If you could pass that that would be really helpful for us to reproduce :)

[subha-aws]

A little more descriptive error with v0.1

Exception in thread Thread-5:
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/executor.py", line 75, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/nest_asyncio.py", line 99, in run_until_complete
    return f.result()
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/executor.py", line 63, in _aresults
    raise e
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/executor.py", line 58, in _aresults
    r = await future
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/executor.py", line 91, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/testset/extractor.py", line 49, in extract
    results = await self.llm.generate(prompt=prompt, is_async=is_async)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/llms/base.py", line 92, in generate
    return await agenerate_text_with_retry(
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped
    return await fn(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/tenacity/_asyncio.py", line 47, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/tenacity/__init__.py", line 325, in iter
    raise retry_exc.reraise()
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/tenacity/__init__.py", line 158, in reraise
    raise self.last_attempt.result()
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/tenacity/_asyncio.py", line 50, in __call__
    result = await fn(*args, **kwargs)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/llms/base.py", line 177, in agenerate_text
    result = await self.langchain_llm.agenerate_prompt(
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 540, in agenerate_prompt
    return await self.agenerate(
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 815, in agenerate
    isinstance(callbacks[0], (list, BaseCallbackManager))
IndexError: list index out of range
---------------------------------------------------------------------------
ExceptionInRunner                         Traceback (most recent call last)
Cell In[10], line 54
     41 docstore = InMemoryDocumentStore(
     42     splitter=splitter,
     43     embeddings=embeddings_model,
     44     extractor=KeyphraseExtractor(generator_llm),
     45 )
     47 test_generator = TestsetGenerator(
     48     generator_llm=generator_llm,
     49     critic_llm=critic_llm,
     50     embeddings=titan_embeddings,
     51     docstore=docstore
     52 )
---> 54 testset = test_generator.generate_with_langchain_docs(documents=documents,
     55                                                       test_size=10,
     56                                                       distributions=testset_distribution)

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/testset/generator.py:150, in TestsetGenerator.generate_with_langchain_docs(self, documents, test_size, distributions, with_debugging_logs, is_async, raise_exceptions, run_config)
    139 def generate_with_langchain_docs(
    140     self,
    141     documents: t.Sequence[LCDocument],
   (...)
    148 ):
    149     # chunk documents and add to docstore
--> 150     self.docstore.add_documents(
    151         [Document.from_langchain_document(doc) for doc in documents]
    152     )
    154     return self.generate(
    155         test_size=test_size,
    156         distributions=distributions,
   (...)
    160         run_config=run_config,
    161     )

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/testset/docstore.py:210, in InMemoryDocumentStore.add_documents(self, docs, show_progress)
    204 # split documents with self.splitter into smaller nodes
    205 nodes = [
    206     Node.from_langchain_document(d)
    207     for d in self.splitter.transform_documents(docs)
    208 ]
--> 210 self.add_nodes(nodes, show_progress=show_progress)

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/testset/docstore.py:250, in InMemoryDocumentStore.add_nodes(self, nodes, show_progress, desc)
    248 results = executor.results()
    249 if results == []:
--> 250     raise ExceptionInRunner()
    252 for i, n in enumerate(nodes):
    253     if i in nodes_to_embed.keys():

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exception=False` incase you want to show only a warning message instead.

[jjmachan]

wow it seems like the error is coming from langchain because there are not callbacks used - very strange. could you add a callback to test it? please refer https://docs.ragas.io/en/stable/howtos/applications/tracing.html

[shahules786]

Hey @subha-aws Is this a new error? I have raised a fix for the earlier one you posted and was able to run test gen on bedrock

from langchain_community.chat_models import BedrockChat
from langchain_community.embeddings import BedrockEmbeddings
from ragas.embeddings.base import BaseRagasEmbeddings, LangchainEmbeddingsWrapper
from ragas.llms import BaseRagasLLM, LangchainLLMWrapper

config = {
   your-config
}

bedrock_model = BedrockChat(
    credentials_profile_name=config["credentials_profile_name"],
    region_name=config["region_name"],
    endpoint_url=f"https://bedrock-runtime.{config['region_name']}.amazonaws.com",
    model_id=config["model_id"],
    model_kwargs=config["model_kwargs"],
)
bedrock_model = LangchainLLMWrapper(bedrock_model)
# init the embeddings
bedrock_embeddings = BedrockEmbeddings(
    credentials_profile_name=config["credentials_profile_name"],
    region_name=config["region_name"],
)

bedrock_embeddings = LangchainEmbeddingsWrapper(bedrock_embeddings)

###Add custom llms and embeddings
generator_llm = bedrock_model
critic_llm = bedrock_model

from ragas.testset.extractor import KeyphraseExtractor
from langchain.text_splitter import TokenTextSplitter
from ragas.testset.docstore import InMemoryDocumentStore

splitter = TokenTextSplitter(chunk_size=1000, chunk_overlap=100)
keyphrase_extractor = KeyphraseExtractor(llm=generator_llm)
docstore = InMemoryDocumentStore(
splitter=splitter,
embeddings=bedrock_embeddings,
extractor=keyphrase_extractor,
)
from ragas.testset import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context

test_generator = TestsetGenerator(
generator_llm=generator_llm,
critic_llm=critic_llm,
embeddings=bedrock_embeddings,
docstore=docstore,
)

testset = test_generator.generate_with_llamaindex_docs(documents=documents[:5],
test_size=10,distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})

[shahules786]

Hey @subha-aws once the PR is merged can you reinstall from source and try again?

[subha-aws]

Hey @subha-aws once the PR is merged can you reinstall from source and try again?

Sure will test this out today. btw, I was using langchain documents not llama index. I will check them out either way! Thank you!

[shahules786]

cool, feel free to close this issue if it works out for you. @subha-aws

[subha-aws]

@shahules786 , Can I add this to the documentation for Bedrock? I also have a notebook for bedrock for LLM comparison.

[shahules786]

Yes please @subha-aws . I think it would be best to add it here Like I have added for language adaptation, create a section for metrics and test generation. If you need clarity, happy to hoping a call

[shahules786]

Let me know if you're working on it so that I can create an issue and assign it to you and make sure non one else is doing the same. @subha-aws

[subha-aws]

@shahules786 Yes please do.

[subha-aws]

cool, feel free to close this issue if it works out for you. @subha-aws

This works! Thank you

[Xiaoyuan-xyz]

Has this issue been closed? I had the similar error and I was using the latest version of ragas, and I used LangchainLLMWrapper.

File "/home/admin1/miniconda3/envs/lccc/lib/python3.11/site-packages/langchain_core/language_models/llms.py", line 853, in agenerate isinstance(callbacks[0], (list, BaseCallbackManager)) IndexError: list index out of range

[dabincicode]

@subha-aws Same issue. Please let me know if the issue has been resolved!

[Totoro8697]

@shahules786 hi, I am getting same error as well others. I am using generate_with_langchain_docs by the way.

[vecorro]

please see #748

[sona-16]

I was trying to do "Synthetic test data generation" using RAGAS framework with the help of the below document.

"https://docs.ragas.io/en/stable/concepts/testset_generation.html"

I'm facing error. Please have a look on the below error.

ragas.exceptions.ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass raise_exceptions=False incase you want to show only a warning message instead. Task was destroyed but it is pending! task: <Task pending name='Task-3' coro=<as_completed..sema_coro() running at C:\Users\sonaganesh.g\Desktop\RAGAS syhthetic data generation\syn\Lib\sit e-packages\ragas\executor.py:38> wait_for= cb=[as_completed.._on_completion() at C:\Users\sonaganesh.g\AppData\Local\Programs\Python\Python312\Lib\asyncio\tasks.py:618]> Task was destroyed but it is pending! task: <Task pending name='Task-5' coro=<as_completed..sema_coro() running at C:\Users\sonaganesh.g\Desktop\RAGAS syhthetic data generation\syn\Lib\sit e-packages\ragas\executor.py:38> wait_for=<_GatheringFuture pending cb=[Task.task_wakeup()]> cb=[as_completed.._on_completion() at C:\Users\sonaganesh.g\AppData\Local\Programs\Python\Python312\Lib\asyncio\tasks.py:618]> Task was destroyed but it is pending! task: <Task pending name='Task-2' coro=<as_completed..sema_coro() running at C:\Users\sonaganesh.g\Desktop\RAGAS syhthetic data generation\syn\Lib\site-packages\ragas\executor.py:38> wait_for= cb=[as_completed.._on_completion() at C:\Users\sonaganesh.g\AppData\Local\Programs\Python\Python312\Lib\asyncio\tasks.py:618]>

[Grantsydney2020]

@sona-16 ,I have the same question,have you already solve it ?

[sadaf0714]

I have a related error, I am using VertexAI with ragas, followed this documentation https://docs.ragas.io/en/latest/howtos/customisations/gcp-vertexai.html ,below is the traceback: WARNING:ragas.validation:passing column names as 'ground_truths' is deprecated and will be removed in the next version, please use 'ground_truth' instead. Note that ground_truth should be of type string and not Sequence[string] like ground_truths Evaluating:  25%  3/12 [00:05<00:11,  1.25s/it] WARNING:ragas.llms.output_parser:Failed to parse output. Returning None. WARNING:ragas.llms.output_parser:Failed to parse output. Returning None. Exception in thread Thread-355: Traceback (most recent call last): File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 96, in run results = self.loop.run_until_complete(self._aresults()) File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 84, in _aresults raise e File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 79, in _aresults r = await future File "/usr/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one return f.result() # May raise f.exception(). File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 38, in sema_coro return await coro File "/usr/local/lib/python3.10/dist-packages/ragas/executor.py", line 112, in wrapped_callable_async return counter, await callable(*args, *kwargs) File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/base.py", line 116, in ascore raise e File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/base.py", line 112, in ascore score = await self._ascore(row=row, callbacks=group_cm, is_async=is_async) File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/_answer_relevance.py", line 167, in _ascore return self._calculate_score(answers, row) File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/_answer_relevance.py", line 139, in _calculate_score cosine_sim = self.calculate_similarity(question, gen_questions) File "/usr/local/lib/python3.10/dist-packages/ragas/metrics/_answer_relevance.py", line 113, in calculate_similarity question_vec = np.asarray(self.embeddings.embed_query(question)).reshape(1, -1) File "/usr/local/lib/python3.10/dist-packages/langchain_google_vertexai/embeddings.py", line 391, in embed_query return self.embed([text], 1, "RETRIEVAL_QUERY")[0] File "/usr/local/lib/python3.10/dist-packages/langchain_google_vertexai/embeddings.py", line 363, in embed embeddings.extend(t.result()) File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result return self.get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(self.args, **self.kwargs) File "/usr/local/lib/python3.10/dist-packages/langchain_google_vertexai/embeddings.py", line 194, in _get_embeddings_with_retry with telemetry.tool_context_manager(self._user_agent): File "/usr/lib/python3.10/contextlib.py", line 142, in exit next(self.gen) File "/root/.local/lib/python3.10/site-packages/google/cloud/aiplatform/telemetry.py", line 48, in tool_context_manager _pop_tool_name(tool_name) File "/root/.local/lib/python3.10/site-packages/google/cloud/aiplatform/telemetry.py", line 57, in _pop_tool_name raise RuntimeError( RuntimeError: Tool context error detected. This can occur due to parallelization.

ExceptionInRunner Traceback (most recent call last)

in () ----> 1 metrices = getMetrices(dataset) 2 frames /usr/local/lib/python3.10/dist-packages/ragas/evaluation.py in evaluate(dataset, metrics, llm, embeddings, callbacks, is_async, run_config, raise_exceptions, column_map) 211 results = executor.results() 212 if results == []: --> 213 raise ExceptionInRunner() 214 215 # convert results to dataset_like ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead. Please help me with this asap! @subha-aws

[damlitos]

same error persists ...

252 results = executor.results() 253 if not results: --> 254 raise ExceptionInRunner() 256 for i, n in enumerate(nodes): 257 if i in nodes_to_embed.keys():

ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass raise_exceptions=False incase you want to show only a warning message instead.