[R-237] ValueError: Unknown format code 'f' for object of type 'str'

[satadda]

[X] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug ValueError: Unknown format code 'f' for object of type 'str'

Ragas version: 0.1.7 Python version: 3.10

Code to Reproduce

    metrics = [
            answer_relevancy,
            answer_correctness,
            answer_similarity,
            context_recall,
            context_precision,
            context_relevancy, 
        ]

       for question in test_questions:
            response = rag_chain.invoke({"question" : question})
            answers.append(response["answer"].content)
            contexts.append([context.page_content for context in response['docs']])

        # create hf dataset
        response_dataset = Dataset.from_dict({
            "question" : test_questions,
            "answer" : answers,
            "contexts" : contexts,
            "ground_truth" : test_groundtruths
        })

        print("sample response: ")
        print(response_dataset[0])

        results = evaluate(response_dataset, metrics, 
                           llm=llm, embeddings = openai_embed,
                           raise_exceptions=False,
                           )
        print(results)

Error trace

  File "/home/asrst/.cache/pypoetry/virtualenvs/llm-NDDKoWLm-py3.10/lib/python3.10/site-packages/ragas/evaluation.py", line 292, in __repr__
    score_strs = [f"'{k}': {v:0.4f}" for k, v in scores.items()]
  File "/home/asrst/.cache/pypoetry/virtualenvs/llm-NDDKoWLm-py3.10/lib/python3.10/site-packages/ragas/evaluation.py", line 292, in <listcomp>
    score_strs = [f"'{k}': {v:0.4f}" for k, v in scores.items()]
ValueError: Unknown format code 'f' for object of type 'str'

Expected behavior

output metrics (python dictonary)

Additional context this usually can happen when v is a string instead of float. But in what cases ragas returns a string value for v in scores.items() ? Ideally it should be numeric for any metric.

_R-237

[jjmachan]

thank you for sharing this @satadda - I don't know why it's spitting a string too - will try to reproduce

thanks a lot for reporting 🙂