dosubot[bot]
commented
1 month ago Hello, @Jeerhz! I'm here to assist you with any bugs, questions, or contributions. Let's work together to solve the issue you're facing.
To resolve the JSON format error and the non-existing method calls (generate_text and generate_prompt) when using the adapt function in the Ragas library, you need to ensure that the JSON format is valid and replace the non-existing method calls with the correct ones.
-
JSON Format Error: Ensure that the examples provided in the
Promptclass are correctly formatted as JSON. Thevalidate_promptmethod already includes a check for valid JSON format. -
Non-existing Method Calls: Replace
generate_textwith the correct method from theBaseRagasLLMclass.
Here is the corrected adapt function:
def adapt(
self, language: str, llm: BaseRagasLLM, cache_dir: t.Optional[str] = None
) -> Prompt:
def get_all_keys(nested_json):
keys = set()
for key, value in nested_json.items():
keys.add(key)
if isinstance(value, dict):
keys = keys.union(get_all_keys(value))
return keys
if self.language == language:
return self
# TODO: Add callbacks
cache_dir = cache_dir if cache_dir else get_cache_dir()
if os.path.exists(os.path.join(cache_dir, language, f"{self.name}.json")):
return self._load(language, self.name, cache_dir)
logger.info("Adapting %s to %s", self.name, language)
prompts = []
output_keys = []
for example in self.examples:
prompts.extend(
[
str_translation.format(
translate_to=language, input=example.get(key)
)
for key in self.input_keys
]
)
prompts.append(
json_translatation.format(
translate_to=language, input=example.get(self.output_key)
)
if self.output_type.lower() == "json"
else str_translation.format(
translate_to=language, input=example.get(self.output_key)
)
)
if self.output_type.lower() == "json":
output = example.get(self.output_key)
if isinstance(output, str):
output = json.loads(output)
if isinstance(output, dict):
output_keys.append(get_all_keys(output))
elif isinstance(output, list) and all(
isinstance(item, dict) for item in output
):
output_keys.append([get_all_keys(item) for item in output])
# NOTE: this is a slow loop, consider Executor to fasten this
results = []
for p in prompts:
results.append(llm.generate(p).generations[0][0].text) # Correct method call
per_example_items = len(self.input_keys) + 1
grouped_results = [
results[i : i + per_example_items]
for i in range(0, len(results), per_example_items)
]
assert len(grouped_results) == len(
self.examples
), "examples and adapted examples must be of equal length"
for i, example in enumerate(grouped_results):
example_dict = {}
example_dict.update(
{k: v for k, v in zip(self.input_keys, example[: len(self.input_keys)])}
)
example_dict[self.output_key] = (
json_loader._safe_load(example[-1], llm)
if self.output_type.lower() == "json"
else example[-1]
)
if self.output_type.lower() == "json":
output = example_dict[self.output_key]
if isinstance(output, dict):
assert (
set(output.keys()) == output_keys[i]
), f"Adapted output keys {set(output.keys())=} do not match with the original output keys: {output_keys[i]=}"
elif isinstance(output, list) and all(
isinstance(item, dict) for item in output
):
assert all(
set(item.keys()) in output_keys[i] for item in output
), "Adapted output keys do not match with the original output keys"
self.examples[i] = example_dict
self.language = language
# TODO:Validate the prompt after adaptation
return selfReplace llm.generate_text(p).generations[0][0].text with llm.generate(p).generations[0][0].text to use the correct method from the BaseRagasLLM class. Additionally, ensure that the Prompt class and its examples are correctly formatted as JSON [1][2][3].
To continue talking to Dosu, mention @dosu.
Jeerhz
heuristicwave
Mervyn-1
jjmachan
PanicButtonPressed
IvanPerezMelendez
I have followed the steps of the "How-To-Guide" about Automatic Language Adaptation (https://docs.ragas.io/en/v0.1.9/howtos/applications/use_prompt_adaptation.html) and tried to modify the already implemented classes but I could not resolve my problem.
Describe the bug
Ragas version:0.1.10 Python version:3.12.4
Code to Reproduce
Error trace Traceback (most recent call last): File "c:\Users\adles\Downloads\base.py", line 21, in
adapt(metrics=[faithfulness,answer_correctness], language="hindi", llm=openai_model)
File "C:\Users\adles\AppData\Local\Programs\Python\Python312\Lib\site-packages\ragas\adaptation.py", line 36, in adapt
metric.adapt(language, cache_dir=cache_dir)
File "C:\Users\adles\AppData\Local\Programs\Python\Python312\Lib\site-packages\ragas\metrics_faithfulness.py", line 306, in adapt
self.nli_statements_message = self.nli_statements_message.adapt( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\adles\AppData\Local\Programs\Python\Python312\Lib\site-packages\ragas\llms\prompt.py", line 181, in adapt return self._load(language, self.name, cache_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\adles\AppData\Local\Programs\Python\Python312\Lib\site-packages\ragas\llms\prompt.py", line 275, in _load return cls(json.load(open(path))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\adles\AppData\Local\Programs\Python\Python312\Lib\site-packages\pydantic\v1\main.py", line 341, in init raise validation_error pydantic.v1.error_wrappers.ValidationError: 1 validation error for Prompt root answer in example 1 is not in valid json format: Expecting value: line 1 column 1 (char 0) (type=value_error) Expected behavior** To inspect the adapted prompt belonging to the answer correctness metric
Additional context