search

stanfordnlp / dspy

DSPy: The framework for programming—not prompting—language models
https://dspy.ai
MIT License
19.35k stars 1.47k forks source link

Addressing Context Length Limitations in DSPy #381

Open fangyuan-ksgk opened 9 months ago

fangyuan-ksgk commented 9 months ago

I've recently attempted to utilize DSPy for the BigBench Hard dataset, specifically focusing on the Casual Judgment scenarios. These scenarios are characterized by lengthy descriptions, which pose a significant challenge due to the context length limitations inherent in current language models like GPT-3.5 (4097 tokens), GPT-4 (8192 tokens), and Mistral (8000+ tokens). This limitation often results in errors during the compilation process when attempting few-shot learning approaches.

To address this, two potential solutions are possible:

Prompt Compression: Implementing a mechanism for condensing prompts could enable the inclusion of longer scenarios within the model's token limitations. This would involve summarizing or distilling the essence of the scenario while maintaining the crucial elements necessary for the model to understand and respond accurately.

Principle-Based Few-Shot Learning: Instead of trying to include every detail of a scenario in the few-shot demonstration, we could focus on capturing the underlying strategies or principles that are key to success. This approach would involve identifying and leveraging the most critical aspects of the examples to guide the model's learning and application in new situations.

I am currently exploring ways to extend the DSPy compiler to incorporate these ideas. If the DSPy team is already working on similar solutions or has plans in this direction, I'd be keen to know and possibly collaborate :> Thanks in advance!

ee-lang commented 8 months ago

Any news on this? I ran into the same problem.