dosubot[bot]
commented
4 days ago To ensure that the feedback provider can be called normally and fix the RuntimeError related to the expected_response KeyError, you need to ensure that the expected_response key is properly handled in your code. Here is how you can modify your code to handle this key appropriately:
-
Modify the
golden_setdictionary: Ensure that each item in thegolden_setdictionary includes theexpected_responsekey. If your dataset does not have this key, you can set it toNone. -
Update the
_create_chat_completionmethod: Add a check to handle theexpected_responsekey.
Here is an example of how you can modify your code:
Step 1: Modify the golden_set Dictionary
Ensure that each item in the golden_set dictionary includes the expected_response key:
test_dataset = pd.read_csv("test_output.csv")
test_dataset.rename(columns={"Input": "query", "GT Response": "response"}, inplace=True)
test_dataset["expected_response"] = None # Add this line to include the expected_response key
golden_set = test_dataset[["query", "response", "expected_response"]].to_dict(orient="records")Step 2: Update the _create_chat_completion Method
Modify the _create_chat_completion method to handle the expected_response key:
def _create_chat_completion(
self,
prompt: Optional[str] = None,
messages: Optional[Sequence[Dict]] = None,
**kwargs
) -> str:
if 'model' not in kwargs:
kwargs['model'] = self.model_engine
if 'temperature' not in kwargs:
kwargs['temperature'] = 0.0
if 'seed' not in kwargs:
kwargs['seed'] = 123
# Handle the 'expected_response' key
if 'expected_response' not in kwargs:
kwargs['expected_response'] = None
if messages is not None:
completion = self.endpoint.client.chat.completions.create(
messages=messages, **kwargs
)
elif prompt is not None:
completion = self.endpoint.client.chat.completions.create(
messages=[{
"role": "system",
"content": prompt
}], **kwargs
)
else:
raise ValueError("`prompt` or `messages` must be specified.")
return completion.choices[0].message.contentStep 3: Ensure GroundTruthAgreement Handles expected_response
Ensure that the GroundTruthAgreement class handles the expected_response key properly. The expected_response key might be missing (i.e., set to None) in ground truth datasets, as explicitly mentioned in the class definition.
expected_response: Optional[str] = (
None # expected response can be empty in GT datasets
)The _find_response method in the GroundTruthAgreement class should handle cases where the expected_response key might be missing:
def _find_response(self, prompt: str) -> Optional[str]:
if self.ground_truth_imp is not None:
return self.ground_truth_imp(prompt)
responses = [
qr["response"]
for qr in self.ground_truth
if qr["query"] == prompt
]
if responses:
return responses[0]
else:
return NoneBy adding the check for the expected_response key and ensuring that your golden_set dictionary includes this key, you can ensure that the feedback provider can be called normally and fix the RuntimeError related to the expected_response KeyError [1][2][3].
To continue talking to Dosu, mention @dosu.
2500035435
ERROR AS FOLLOWS:
I try to use local deployment llm (base and finetuned), feedback provider use openai llm with endpoint and api_key, and evaluate in my dataset.
I want to know how to make sure that the feedback provider can be called normally, and how to fix the error with my part of the code causing an error(reference ) to build my code.
Any helpful help would be greatly appreciated.