dosubot[bot]
commented
1 month ago The error you're encountering indicates that the CustomRagasLLM class is missing implementations for the abstract methods generate_text and agenerate_text from the BaseRagasLLM class. However, your code does include these methods, so the issue might be related to the method signatures not matching the expected ones in the BaseRagasLLM class.
Here's what you need to do:
-
Ensure Method Signatures Match: The method signatures in your subclass must match those defined in the
BaseRagasLLMclass. Here are the correct signatures based on theBaseRagasLLMclass:def generate_text( self, prompt: PromptValue, n: int = 1, temperature: float = 1e-8, stop: t.Optional[t.List[str]] = None, callbacks: Callbacks = None, ) -> LLMResult: # Your implementation here async def agenerate_text( self, prompt: PromptValue, n: int = 1, temperature: t.Optional[float] = None, stop: t.Optional[t.List[str]] = None, callbacks: Callbacks = None, ) -> LLMResult: # Your implementation here -
Adjust Your Implementations: Update your
generate_textandagenerate_textmethods to match these signatures. You will need to adjust the logic inside these methods to handle the additional parameters and return anLLMResultobject.
Here's an example of how you might adjust your generate_text method:
def generate_text(
self,
prompt: PromptValue,
n: int = 1,
temperature: float = 1e-8,
stop: t.Optional[t.List[str]] = None,
callbacks: Callbacks = None,
) -> LLMResult:
# Example implementation
text = self._call(prompt)
return LLMResult(generations=[Generation(text=text)])And for agenerate_text:
async def agenerate_text(
self,
prompt: PromptValue,
n: int = 1,
temperature: t.Optional[float] = None,
stop: t.Optional[t.List[str]] = None,
callbacks: Callbacks = None,
) -> LLMResult:
# Example implementation
text = self._call(prompt)
return LLMResult(generations=[Generation(text=text)])Make sure to replace PromptValue, LLMResult, and Generation with the appropriate types or classes used in your project. This should resolve the error and allow you to instantiate your CustomRagasLLM class.
To continue talking to Dosu, mention @dosu.
amin-kh96
I wrote this code and I am using a subclass of LLM and BASERAGASEMBEDDINGS. I already have the embeddings and I face the error to set the API key.to solve that since I have all the requirements I created a sub class of baseragasLLM, but I face this error, and the thing is that I already defined what it needed in my subclass, please take a look at my code and my error and help me to fix it.
the code: import json from transformers import AutoModel, AutoTokenizer import torch import numpy as np from datasets import Dataset from ragas.embeddings import BaseRagasEmbeddings from ragas.metrics import context_utilization,ContextUtilization from ragas.llms import BaseRagasLLM import asyncio from ragas import evaluate
Load the ground truth data
file_path = 'C:\Users\Amin\OneDrive - unige.it\Documenti\projectss\ragas-prototype\src\assets\GT.json' with open(file_path) as f: ground_truth_data = json.load(f)
Load the question and the answer and the chunks
file_path = 'C:\Users\Amin\OneDrive - unige.it\Documenti\projectss\ragas-prototype\src\assets\user_llm_interaction_embeddings_c1521dd5_b819_4241_b3a4_3e5c1388037c.json' with open(file_path) as f: llm = json.load(f)
Initialize an empty list to hold the new dataset
data_set = []
Iterate through the list and combine every two dictionaries
for i in range(0, len(llm), 2): combined_dict = { "text_vector_1": llm[i].get("text_vector", []), "text_vector_2": llm[i + 1].get("text_vector", []), 'chunks': llm[i + 1].get('chunks', []) } data_set.append(combined_dict)
def map_chunks(data_set, ground_truth_data): for item in data_set: # Iterate over each dictionary in data_set c = [] # Reset c for each item for chunk_id in item['chunks']: # Loop through 'chunks' in the current dictionary for element in ground_truth_data: # Loop through ground_truth_data if element['id'] == chunk_id: # Match chunk_id with element's id c.append(element['text_vector']) # Append the matching text_vector to c item['chunks'] = c # Replace the original 'chunks' (ids) with the mapped text_vector values
data_set = map_chunks(data_set, ground_truth_data)
Assuming data_set is a list of dictionaries
ragas_data = [ { "question": entry["text_vector_1"], # Assuming this is a list of strings "answer": entry["text_vector_2"], # Assuming this is a list of strings "contexts": entry["chunks"] # Assuming this is a list of lists of strings } for entry in data_set ]
Create the required structure for Dataset
formatted_data = { "question": [entry["question"] for entry in ragas_data], "contexts": [entry["contexts"] for entry in ragas_data], "answer": [entry["answer"] for entry in ragas_data] }
model_name = 'distilbert-base-uncased'
class CustomHuggingFaceRagasEmbeddings(BaseRagasEmbeddings): def init(self, model_name: str, custom_embeddings: list = None): """ Initialize the Custom Hugging Face Ragas Embeddings with the specified model and custom embeddings.
Initialize the custom embeddings class
custom_embeddings = CustomHuggingFaceRagasEmbeddings(ragas_data)
ragas_embeddings = CustomHuggingFaceRagasEmbeddings(model_name=model_name, custom_embeddings=custom_embeddings)
Define the custom LLM class
class CustomRagasLLM(BaseRagasLLM): def init(self, api_key: str = None): """ Initialize the custom LLM, optionally using an API key if necessary. """ self.api_key = api_key
Initialize the custom LLM class (set api_key=None if you don't need it)
custom_llm = CustomRagasLLM(api_key=None)
Define the evaluation metrics
metrics = [context_utilization]
lets define a custom function of evaluate function
def custom_evaluate(ragas_data, metrics, embeddings: BaseRagasEmbeddings):
""" Custom evaluation function that avoids using OpenAI API.
results = {}
return results
Run the evaluation
evaluation_report = evaluate(ragas_data, metrics=metrics, embeddings=ragas_embeddings,llm=custom_llm) # Pass your custom LLM here
Print the evaluation results
print("RAGAS Evaluation Report:") print(evaluation_report)
the error: Exception has occurred: TypeError Can't instantiate abstract class CustomRagasLLM without an implementation for abstract methods 'agenerate_text', 'generate_text' File "C:\Users\Amin\OneDrive - unige.it\Documenti\projectss\ragas-prototype\src\rag_eva.py", line 158, in
custom_llm = CustomRagasLLM(api_key=None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Can't instantiate abstract class CustomRagasLLM without an implementation for abstract methods 'agenerate_text', 'generate_text'