Adding MetricHandler class for handling and generating metrics
A new MetricHandler class is now available to generate a list of static metrics on-the-fly for optimizing the agent for the provided task description. This class is now instantiated inside the agent optimization pipeline. See /metric_generator/ for more. Besides requiring a task description, the user can also edit the num_metrics parameter during configuration to request a specific number of generated metrics. Shout-out to Elad for helping with these!
Added meta prompts for handling the metric generation
A new meta prompt is now available under /prompts/meta_prompts_generation/ for creating helpful metrics.
Added JSON Schema for generated metrics
A new JSON schema has been added to provide a format for metric generation. The generated metrics currently have the following general format:
Adding
MetricHandler
class for handling and generating metricsA new
MetricHandler
class is now available to generate a list of static metrics on-the-fly for optimizing the agent for the provided task description. This class is now instantiated inside the agent optimization pipeline. See/metric_generator/
for more. Besides requiring a task description, the user can also edit thenum_metrics
parameter during configuration to request a specific number of generated metrics. Shout-out to Elad for helping with these!Added meta prompts for handling the metric generation
A new meta prompt is now available under
/prompts/meta_prompts_generation/
for creating helpful metrics.Added JSON Schema for generated metrics
A new JSON schema has been added to provide a format for metric generation. The generated metrics currently have the following general format:
Shout-out to Elad and Michael for feedback and helpful discussions here!