mikhailocampo
commented
1 month ago @bkb-Git Quick question, so what is the expected incoming format for the choices data structure? I know it is a list type but are you expecting the keys to be explicitly key or the actual letter like
{
"key": "A",
"value": "<answer>"
}OR
{
"<letter_key>": "<choice>"
}Let us know so we can validate
This Pull Request is done for solving #30. As @mikhailocampo mentioned, I had to monitor the LLM responses in order to evaluate the problem. What I have discovered was that the actual problem was not the ```json string, but indeed it was this specific response format that was not appropriately managed by the validate_response function:
As you can see, the format changes for the choices, organizing them differently as how it is normally accepted:
For that reason, I have implemented a function to pre-process the provided dict from the chain. As a result, we can decrease the latency for performing less requests, decreasing the token costs and achieving the requested amount of questions.
This was the result of requesting 3 questions (With multiple attempts as a result of the unaccepted format) before of the implementation of the function:
And this is the result of requesting 3 questions after implementing the function:
