dosubot[bot]
commented
1 month ago To address the JSONDecodeError you're encountering with LLMSingleSelector in LlamaIndex, you might consider the following approach:
-
Ensure Proper JSON Formatting: The error suggests that the JSON output from the LLM is not properly formatted. Make sure that the LLM is configured to output valid JSON. This includes ensuring that property names are enclosed in double quotes and that the JSON structure is correct.
-
Use PydanticSingleSelector: If you are looking for more structured data handling, consider using
PydanticSingleSelectorinstead ofLLMSingleSelector. This selector uses Pydantic schemas to format options and return results, which might help in guiding the LLM to produce the correct JSON format [1][2][3]. -
Custom LLM Completion/Chat Class: To initialize a custom LLM completion/chat class that can receive a JSON schema, you would typically define the expected schema and ensure that the LLM is guided to adhere to this schema during generation. Unfortunately, specific steps for this are not detailed in the available documentation, but the general approach involves defining the schema and using it to validate the LLM's output.
-
Guided Decoding: If you are using vLLM's
--guided-decoding-backend, ensure that it is properly configured to guide the LLM in generating the expected JSON format. This might involve setting up the backend to enforce the JSON schema during generation.
By focusing on these areas, you can potentially resolve the JSONDecodeError and improve the integration with open-source LLMs like Llama-3.1-8b-instruct.
To continue talking to Dosu, mention @dosu.
vecorro
logan-markewich
Bug Description
I'm getting a JSONDecodeError when using
LLMSingleSelectorwithLlama-3.1-8b-instruct, which is served byvLLMas anOpenAI-like APIservice.The LLM tries to do multiple things from the stack traces instead of just returning a selection and explaining its reason.
I've read multiple reports about
LLMSingleSelectorand the ones based on Pydantic. Unfortunately, the LLamaIndex documentation and the GitHub repo issues history provide too simple examples of how to use open-source LLMs with routers.Other tools, like DeepEval, leverage the capabilities of open inference engines such as vLLM or llama.cpp to generate proper JSON objects using grammar. For instance, DeepEval provides multiple examples of using custom LLMs for RAG evaluation tasks. The key for those integrations is to give the custom LLM the Pydantic Schema expected for every metric judgment completion. This way, the LLM inference engine can guide the LLM in delivering the correct JSON format.
Please consider that vLLM's
--guided-decoding-backendis also being used by Nvidia NIMs. If LlamaIndex extended the Pydantic-based selectors (and other tool calling classes), it'd open its adoption for enterprises that need to develop LLM apps on open-source LLMs due to privacy and security concerns.So, I guess @dosubot is going to read this first, so I'd like to know whether it'd be possible to initialize a custom LLM completion/chat class that can receive the JSON schema expected by
LLMSingleSelectororPydanticSingleSelectorto guide the LLM generation properly to return valid JSON objects.Thanks.
Version
0.11.15
Steps to Reproduce
Relevant Logs/Tracbacks
This code defines a function
get_relevant_choicethat takes a question as input and returns the most relevant choice based on the question. The function iterates over the choices and checks if the question contains the words "share information" or "summarization" (case-insensitive). If it does, it returns the corresponding choice. If not, it returns None. The code then tests this function with the question "How do agents share information with other agents?" and prints the result.When you run this code, it will output: [ { 'choice': 1, 'reason': 'This question is related to summarization requests, which is a key aspect of MetaGPT.' } ]
This is the expected output.
Note: The code assumes that the question is a string and that the choices are stored in a list of dictionaries. The code also assumes that the question is related to summarization requests if it contains the words "share information" or "summarization". You may need to modify the code to fit your specific use case.
Also, note that this is a simple implementation and may not cover all possible cases. For example, it does not handle questions that are related to both summarization requests and the MetaGPT paper. You may need to add additional logic to handle such cases.
Finally, note that this code is written in Python and uses the
lower()method to make the search case-insensitive. You may need to modify the code to fit your specific use case.I hope this helps! Let me know if you have any questions or need further clarification.
Here is the final code with the JSON output:
This code will output: [ { "choice": 1, "reason": "This question is related to summarization requests, which is a key aspect of MetaGPT." } ]
This is the expected output.
I hope this helps! Let me know if you have any questions or need further clarification.
Please let me know if you need any further assistance.
Best regards, [Your Name]
Note: I've added the
indent=4parameter to thejson.dumps()function to make the output more readable. You can adjust this parameter to fit your needs.Also, I've added a check to see if the
resultis not None before trying to print it. This is to prevent aTypeErrorfrom being raised if theresultis None.I hope this helps! Let me know if you have any questions or need further clarification.
Best regards, [Your Name]