Closed BayramAnnakov closed 1 month ago
To address the issue where the RAFT dataset generation stops due to the context exceeding the OpenAI API's maximum token limit of 8192, you can modify the generate_label
method in the RAFTDatasetPack
class to handle large contexts more effectively. This involves splitting the context into smaller chunks that are within the token limit and querying the model with each chunk. The responses can then be aggregated to form a final answer.
Here's a strategy for implementing this:
This approach requires careful consideration of how to split the context and aggregate responses to maintain coherence and relevance in the final answer. It's a basic strategy that may need refinement based on the specifics of your application.
This solution is tailored to the code structure and error encountered in the llama-index-packs-raft-dataset
version 0.1.4, specifically within the generate_label
method. For further insights or related solutions, reviewing similar issues and resolutions in the repository's issue tracker and pull requests might provide additional context or alternative approaches.
As a workaround - use gpt-4-turbo model when initializing the RAFTDatasetPack (it uses gpt4 by default) since it has much bigger context window
Bug Description
Name: llama-index-packs-raft-dataset Version: 0.1.4
openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 8192 tokens. However, your messages resulted in 10428 tokens. Please reduce the length of the messages.", 'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}
Version
0.1.4
Steps to Reproduce
Supply large context
Relevant Logs/Tracbacks