Getting RecursionError: maximum recursion depth exceeded while calling a Python object while creating tru_recorder.

[sadaf0714]

from trulens_eval import Feedback, LiteLLM, TruLlama from trulens_eval.feedback import Groundedness from trulens_eval.feedback.provider.hugs import Huggingface

import numpy as np os.environ["HUGGINGFACE_API_KEY"] = "MY_HUGGINGFACE_API_KEY"

LiteLLM.set_verbose = True provider = LiteLLM(model_engine="huggingface/mistralai/Mistral-7B-Instruct-v0.1")

f_qa_relevance = Feedback( provider.relevance_with_cot_reasons, name="Answer Relevance" ).on_input_output() context_selection = TruLlama.select_source_nodes().node.text

import numpy as np

f_qs_relevance = ( Feedback(provider.qs_relevance, name="Context Relevance") .on_input() .on(context_selection) .aggregate(np.mean) ) GROUNDED = Groundedness(groundedness_provider=provider)

f_groundedness = ( Feedback(GROUNDED.groundedness_measure_with_cot_reasons, name="Groundedness") .on(context_selection.collect()) .on_output() .aggregate(GROUNDED.grounded_statements_aggregator) )

from trulens_eval import TruLlama from trulens_eval import FeedbackMode

tru_recorder = TruLlama( sentence_window_engine, app_id="App_1", feedbacks=[ f_qa_relevance, f_qs_relevance, f_groundedness ] )

This is my code, i am unable to create a tru_recorder with it. I am using a huggingface pipeline for RAG.

[dosubot[bot]]

Hey there, @sadaf0714! 👋 I'm Dosu, your friendly neighborhood bot, here to lend a hand with bugs, answer your questions, and guide you through contributing while we wait for a human maintainer to swing by. I'm diving into your issue and will get back to you with a more detailed response soon!

[sadaf0714]

This is the error snapshot.

[dosubot[bot]]

The RecursionError you're encountering, especially with the traceback pointing to matplotlib, suggests a complex issue possibly related to infinite recursion in your code or interactions with dependencies. Here's a focused approach to troubleshoot and potentially resolve the problem:

1. Check for Infinite Recursion: Ensure there's no logical error in your custom code that could cause infinite recursion, particularly in parts of the code that involve text processing or feedback aggregation.

2. Review Matplotlib Integration: Since matplotlib is involved in the error, review any plotting or visualization logic in your project. An inadvertent recursive call in this part of your code could be the culprit.

3. Examine Huggingface Pipeline Interactions: Investigate how the Huggingface pipeline's output is used, especially if it's part of a complex workflow that includes data transformation or visualization with matplotlib. There might be unexpected recursive calls triggered by these interactions.

4. Environment and Dependency Check: Look into your project's dependencies for any known issues or incompatibilities between matplotlib, the Huggingface pipeline, and other libraries you're using. Updating or adjusting these dependencies might resolve the issue.

5. Isolate and Diagnose: Try isolating the problematic section by systematically commenting out parts of the code related to matplotlib and feedback processing. This can help identify the exact source of the recursion error.

6. Temporary Increase in Recursion Limit: As a diagnostic step, you can temporarily increase the Python recursion limit with sys.setrecursionlimit(), but be cautious as this doesn't solve the underlying problem and could lead to a stack overflow.

By focusing on these areas, you should be able to narrow down the cause of the RecursionError and find a path towards resolving it.

#### Details

**Sources** - [trulens_eval/examples/expositional/use_cases/iterate_on_rag/3_harmless_eval.ipynb](https://github.com/truera/trulens/blob/main/trulens_eval/examples/expositional/use_cases/iterate_on_rag/3_harmless_eval.ipynb) - [trulens_eval/trulens_eval/tru_llama.py](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/tru_llama.py) - [trulens_eval/trulens_eval/feedback/feedback.py](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/feedback.py) - [trulens_explain/trulens/utils/tru_logger.py](https://github.com/truera/trulens/blob/main/trulens_explain/trulens/utils/tru_logger.py) - [trulens_explain/trulens/visualizations.py](https://github.com/truera/trulens/blob/main/trulens_explain/trulens/visualizations.py) - [trulens_eval/trulens_eval/feedback/provider/hugs.py](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/provider/hugs.py) - [trulens_eval/trulens_eval/feedback/provider/endpoint/hugs.py](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/provider/endpoint/hugs.py) - [trulens_explain/trulens/nn/models/_model_base.py](https://github.com/truera/trulens/blob/main/trulens_explain/trulens/nn/models/_model_base.py) - [trulens_explain/trulens/nn/backend/__init__.py](https://github.com/truera/trulens/blob/main/trulens_explain/trulens/nn/backend/__init__.py)
**About Dosu** This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[joshreini1]

@sadaf0714 can you share your huggingface rag pipeline code so we can reproduce/help? Assuming this is the same application as your other open issue?

[guangyao-quan]

[guangyao-quan]

[guangyao-quan]

Same error here.

[wahidur028]

[sadaf0714]

this is how i created feedback metrices f_qa_relevance = Feedback( provider.relevance_with_cot_reasons, name="Answer Relevance" ).on_input_output() query = Select.Record.app.retriever._get_relevant_documents.args.query context = Select.Record.app.retriever.get_relevant_documents.rets[:].page_content

f_context_relevance = ( Feedback( provider.qs_relevance_with_cot_reasons, name="Context Relevance", ) .on(query) .on(context) .aggregate(np.mean) ) grounded = Groundedness(groundedness_provider=provider)

f_groundedness = ( Feedback( grounded.groundedness_measure_with_cot_reasons, name="Groundedness", ) .on(context.collect()) .on_output() .aggregate(grounded.grounded_statements_aggregator) ) and tru_recorder: tru_recorder = TruChain( qa, app_id="App_1", feedbacks=[ f_qa_relevance, f_context_relevance, f_groundedness, f_lang_match ] ) @wahidur028 @guangyao-quan let me know if this helps?

[wahidur028]

@sadaf0714 could you please tell me how can I fix it.

also plz check

[sadaf0714]

@sadaf0714 could you please tell me how can I fix it.

also plz check

import it like this: from trulens_eval import Feedback, LiteLLM, TruLlama, Select, TruChain, FeedbackMode from trulens_eval.feedback import Groundedness

[wahidur028]

[wahidur028]

tru_recorder = TruLlama() whenever I call this function, I get the "maximum recursion depth exceeded" error. But I use custom RAG, following link the code works fine. Is there any issue in TruLlama()?

[wahidur028]

Here is my code. Please suggest how to handle this error

import os
from llama_index.core import  VectorStoreIndex, get_response_synthesizer, Settings, StorageContext, PromptTemplate, load_index_from_storage, SimpleDirectoryReader
from llama_index.llms.huggingface import HuggingFaceLLM
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from trulens_eval import Feedback, Select

# HuggingFace API key
os.environ["HUGGINGFACE_TOKEN"] = "hf_RyUYYSrF----------VStqeKYsj"

#Openai API key
os.environ["OPENAI_API_KEY"] = "sk-7N1V337g-----7jQGT3BlbkFJVo-----enMlKbnHhW"

# Configure custom embedding model
embed_model = HuggingFaceEmbedding(model_name="wahid028/GIST-Law-Embed")
Settings.embed_model = embed_model

hf_token = "hf_YCZNvLuVr-----ZQcdbNVQhWdvb"

from transformers import AutoTokenizer

HF_model = "meta-llama/Meta-Llama-3-8B"

tokenizer = AutoTokenizer.from_pretrained(
    HF_model,
    token=hf_token,
)

stopping_ids = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|eot_id|>"),
]

llm = HuggingFaceLLM(
    model_name=HF_model,
    model_kwargs={
        "token": hf_token,
        "torch_dtype": torch.bfloat16,  # comment this line and uncomment below to use 4bit
    },
    generate_kwargs={
        "do_sample": True,
        "temperature": 0.7,
        "top_p": 0.9,
    },

    tokenizer_name= HF_model,
    tokenizer_kwargs={"token": hf_token},
    stopping_ids=stopping_ids,
)

Settings.llm = llm

documents = SimpleDirectoryReader("../RAG/data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(similarity_top_k=3)

response = query_engine.query("What are the .....?")
print(str(response))

eval_questions = []
with open('eval_questions.txt', 'r') as file:
    for line in file:
        # Remove newline character and convert to integer
        item = line.strip()
        print(item)
        eval_questions.append(item)

        # You can try your own question:
new_question = "What are the primary objectives ..?"
eval_questions.append(new_question)

from trulens_eval import Tru
tru = Tru()

tru.reset_database()

from trulens_eval.feedback.provider.openai import OpenAI
import numpy as np

# Initialize provider class
provider = OpenAI()

# select context to be used in feedback. the location of context is app specific.
from trulens_eval.app import App
context = App.select_context(query_engine)

# Define a groundedness feedback function
f_groundedness = (
    Feedback(provider.groundedness_measure_with_cot_reasons)
    .on(context.collect()) # collect context chunks into a list
    .on_output()
)

# Question/answer relevance between overall question and answer.
f_answer_relevance = (
    Feedback(provider.relevance)
    .on_input_output()
)
# Question/statement relevance between question and each context chunk.
f_context_relevance = (
    Feedback(provider.context_relevance_with_cot_reasons)
    .on_input()
    .on(context)
    .aggregate(np.mean)
)

from trulens_eval import TruLlama
tru_query_engine_recorder = TruLlama(query_engine,
    app_id='LlamaIndex_App1',
    feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance])

RecursionError: maximum recursion depth exceeded

[OrionStar25]

I'm getting the exact same error. Is there a fix yet?

[joshreini1]

Hi folks - there is a fix here: https://github.com/truera/trulens/pull/1176

You can try it out by installing from the branch:

pip install git+https://github.com/piotrm0/trulens.git@piotrm/jsonify-stack-size-protect#subdirectory=trulens_eval

[wahidur028]

@joshreini1 Now it's working fine for me. Thanks for your support.