A bug happened!
When launching the "generate_testset" after defining the bedrock Client for Claud03, I get an error saying: "MainThread giskard.rag ERROR Encountered error in question generation: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row. Skipping.
2024-05-14 08:10:06,421 pid:12762 MainThread giskard.rag ERROR An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row"
And just under it a "KeyError: "None of ['id'] are in the columns""
PS: I don't use giskard hub
Standalone code OR list down the steps to reproduce the issue
import giskard
from giskard.llm.client.bedrock import ClaudeBedrockClient
import boto3
bedrock_runtime= boto3.client("bedrock-runtime",
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
region_name=region_name)
bedrock_runtime_embed= boto3.client("bedrock-runtime",
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
region_name=region_name)
claud_client=ClaudeBedrockClient(bedrock_runtime, model="anthropic.claude-3-haiku-20240307-v1:0")#"anthropic.claude-3-sonnet-20240229-v1:0"
giskard.llm.set_default_client(claud_client)
from giskard.llm.embeddings.bedrock import BedrockEmbedding
embed_client=BedrockEmbedding(bedrock_runtime_embed, model="amazon.titan-embed-text-v1")
giskard.llm.embeddings.set_default_embedding(embed_client)
#Question generation
from giskard.rag import KnowledgeBase, generate_testset, QATestset
import pandas as pd
text_nodes = splitter(documents)
knowledge_base_df = pd.DataFrame([node.text for node in text_nodes], columns=["text"])
#knowledge_base_df["id"]=knowledge_base_df.index
knowledge_base = KnowledgeBase(knowledge_base_df)
testset = generate_testset(knowledge_base,
num_questions=3,
agent_description="TEST")#here the error
Relevant log output
2024-05-07 08:27:45,145 pid:12138 MainThread giskard.rag INFO Finding topics in the knowledge base.
2024-05-07 08:27:53,602 pid:12138 MainThread giskard.rag INFO Found 1 topics in the knowledge base.
Generating questions: 0%| | 0/3 [00:00<?, ?it/s]
2024-05-07 08:27:54,321 pid:12138 MainThread giskard.rag ERROR Encountered error in question generation: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row. Skipping.
2024-05-07 08:27:54,330 pid:12138 MainThread giskard.rag ERROR An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 57, in generate_questions
yield self.generate_single_question(knowledge_base, *args, **kwargs)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/simple_questions.py", line 96, in generate_single_question
generated_qa = self._llm_complete(messages=messages)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 42, in _llm_complete
out = self._llm_client.complete(
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/llm/client/bedrock.py", line 72, in complete
response = self._client.invoke_model(body=body, modelId=self.model, accept=accept, contentType=contentType)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 565, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 1021, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
2024-05-07 08:27:54,530 pid:12138 MainThread giskard.rag ERROR Encountered error in question generation: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row. Skipping.
2024-05-07 08:27:54,538 pid:12138 MainThread giskard.rag ERROR An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 57, in generate_questions
yield self.generate_single_question(knowledge_base, *args, **kwargs)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/simple_questions.py", line 96, in generate_single_question
generated_qa = self._llm_complete(messages=messages)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 42, in _llm_complete
out = self._llm_client.complete(
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/llm/client/bedrock.py", line 72, in complete
response = self._client.invoke_model(body=body, modelId=self.model, accept=accept, contentType=contentType)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 565, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 1021, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
2024-05-07 08:27:54,722 pid:12138 MainThread giskard.rag ERROR Encountered error in question generation: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row. Skipping.
2024-05-07 08:27:54,730 pid:12138 MainThread giskard.rag ERROR An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
Traceback (most recent call last):
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 57, in generate_questions
yield self.generate_single_question(knowledge_base, *args, **kwargs)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/simple_questions.py", line 96, in generate_single_question
generated_qa = self._llm_complete(messages=messages)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/question_generators/base.py", line 42, in _llm_complete
out = self._llm_client.complete(
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/llm/client/bedrock.py", line 72, in complete
response = self._client.invoke_model(body=body, modelId=self.model, accept=accept, contentType=contentType)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 565, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/botocore/client.py", line 1021, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: messages: roles must alternate between "user" and "assistant", but found multiple "user" roles in a row
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
/tmp/ipykernel_12138/3725230573.py in ?()
----> 1 testset = generate_testset(knowledge_base,
2 num_questions=tuning_parameters["number_questions"],
3 agent_description="TEST")
~/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/testset_generation.py in ?(knowledge_base, num_questions, question_generators, language, agent_description)
97 "question_generators": [qg.__class__.__name__ for qg in question_generators],
98 "knowledge_base_size": len(knowledge_base._documents),
99 },
100 )
--> 101 return QATestset(questions)
~/anaconda3/envs/python3/lib/python3.10/site-packages/giskard/rag/testset.py in ?(self, question)
43 def __init__(self, question: Sequence[QuestionSample]):
44 self._questions = question
---> 45 self._dataframe = pd.DataFrame.from_records([question.to_dict() for question in self._questions]).set_index(
46 "id"
47 )
~/anaconda3/envs/python3/lib/python3.10/site-packages/pandas/core/frame.py in ?(self, keys, drop, append, inplace, verify_integrity)
6105 if not found:
6106 missing.append(col)
6107
6108 if missing:
-> 6109 raise KeyError(f"None of {missing} are in the columns")
6110
6111 if inplace:
6112 frame = self
KeyError: "None of ['id'] are in the columns"
Issue Type
Bug
Source
source
Giskard Library Version
2.11.0
Giskard Hub Version
1.3.0
OS Platform and Distribution
Linux 5.10.215-203.850.amzn2.x86_64 x86_64
Python version
3.10.14
Installed python packages
Current Behaviour?
Standalone code OR list down the steps to reproduce the issue
Relevant log output