This pull request introduces a new DspyMiproTrainer and refactors the Evaluate class along with several other trainers. The main changes include:
Refactoring of the Evaluate class in libs/ape-common/ape/common/evaluate/evaluate.py:
Simplified constructor and removed dependency on EvaluationConfig
Added generate parameter to the constructor
Modified __call__ method to use trainset instead of valset
Updated evaluation logic to use the new generate method
Addition of DspyMiproTrainer in libs/ape-core/ape/core/v2/trainer/dspy_mipro_trainer.py:
Implements a new training strategy using MIPRO (Minimum Prompt Optimization)
Includes methods for generating instruction candidates and fewshot demo sets
Updates to existing trainers (ExpelTrainer, FewShotTrainer, OptunaTrainer, TextGradientTrainer):
Modified to use trainset instead of valset for evaluation
Updated to use the refactored Evaluate class
Addition of a new prompt template gen-fewshot-placeholder.prompt for generating fewshot placeholders in prompts
Minor version bumps in pyproject.toml files for both ape-common and ape-core
These changes aim to improve the flexibility and performance of the training process by introducing a new training strategy and refactoring existing components to work with the updated Evaluate class.
Changes that Break Backward Compatibility
The refactoring of the Evaluate class and changes in trainer implementations may break backward compatibility for code relying on the previous implementations. Specifically:
The Evaluate class constructor now requires a generate parameter
Trainers now use trainset instead of valset for evaluation
The __call__ method of Evaluate has changed its return type and parameter list
aschung01
commented
1 week ago
Description
This pull request introduces a new
DspyMiproTrainerand refactors theEvaluateclass along with several other trainers. The main changes include:Refactoring of the
Evaluateclass inlibs/ape-common/ape/common/evaluate/evaluate.py:EvaluationConfiggenerateparameter to the constructor__call__method to use trainset instead of valsetgeneratemethodAddition of
DspyMiproTrainerinlibs/ape-core/ape/core/v2/trainer/dspy_mipro_trainer.py:Updates to existing trainers (
ExpelTrainer,FewShotTrainer,OptunaTrainer,TextGradientTrainer):EvaluateclassAddition of a new prompt template
gen-fewshot-placeholder.promptfor generating fewshot placeholders in promptsMinor version bumps in
pyproject.tomlfiles for bothape-commonandape-coreThese changes aim to improve the flexibility and performance of the training process by introducing a new training strategy and refactoring existing components to work with the updated
Evaluateclass.Changes that Break Backward Compatibility
The refactoring of the
Evaluateclass and changes in trainer implementations may break backward compatibility for code relying on the previous implementations. Specifically:Evaluateclass constructor now requires agenerateparameter__call__method ofEvaluatehas changed its return type and parameter listDocumentation
N/A
Created with Palmier