TonicAI / tonic_validate

Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.
https://docs.tonic.ai/validate/
MIT License
253 stars 27 forks source link

Integration with Ragas in not working #222

Open Jeerhz opened 2 months ago

Jeerhz commented 2 months ago

Question I am just asking if Ragas is still working with Tonic Validate since the code provided in the documentation is not working: https://validate.tonic.ai/

Below is a code that is supposed to calculate the score on a benchmark composed of one question.

Code Examples

from ragas import evaluate
import json
from datasets import Dataset
from tonic_ragas_logger import RagasValidateApi

#Case of no restriction
data_samples = {"question": [], "answer": [], "contexts": [], "ground_truth": []}

with open("reponse_rag_testset_one_question.json", "r", encoding="utf-8") as file:
    reponse_data = json.load(file)

for item in reponse_data:
    data_samples["answer"].append(item["answer"][0])
    data_samples["question"].append(item["question"])
    data_samples["contexts"].append(item["context"])
    data_samples["ground_truth"].append(item["ground_truth"])

dataset = Dataset.from_dict(data_samples)
scores = evaluate(
        dataset, metrics=metrics, llm=azure_model, embeddings=azure_embeddings
    )

#send the score to Tonic Validate
validate_api = RagasValidateApi("G6ph3yz5hk47bQIUtRKMqfF7y8GgiJ-uBLrv6-GNmtU")
validate_api.upload_results("f5aa069e-924b-4723-a5b5-ced06f9b8ab7", scores)

Error I also provide here the error message when running the code:

Traceback (most recent call last):
  File "c:\Users\adle.bensalem\Downloads\RAG-HES-SO\ragas_evaluate.py", line 90, in <module>
    validate_api.upload_results("f5aa069e-924b-4723-a5b5-ced06f9b8ab7", scores)
  File "C:\Users\adle.bensalem\AppData\Local\Programs\Python\Python312\Lib\site-packages\tonic_ragas_logger\ragas_validate_api.py", line 56, in upload_results
    run = self.__convert_to_run(results)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\adle.bensalem\AppData\Local\Programs\Python\Python312\Lib\site-packages\tonic_ragas_logger\ragas_validate_api.py", line 124, in __convert_to_run
    return Run(
           ^^^^
  File "C:\Users\adle.bensalem\AppData\Local\Programs\Python\Python312\Lib\site-packages\pydantic\_internal\_dataclasses.py", line 141, in __init__
    s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
pydantic_core._pydantic_core.ValidationError: 1 validation error for Run
llm_evaluator
  Field required [type=missing, input_value=ArgsKwargs((), {'overall_...onner.'])], 'id': None}), input_type=ArgsKwargs]