Eladlev / AutoPrompt

A framework for prompt tuning using Intent-based Prompt Calibration
Apache License 2.0
2.22k stars 196 forks source link

Added Static Metric Generation Wokflow #82

Closed akashest41 closed 3 months ago

akashest41 commented 3 months ago

Adding MetricHandler class for handling and generating metrics

A new MetricHandler class is now available to generate a list of static metrics on-the-fly for optimizing the agent for the provided task description. This class is now instantiated inside the agent optimization pipeline. See /metric_generator/ for more. Besides requiring a task description, the user can also edit the num_metrics parameter during configuration to request a specific number of generated metrics. Shout-out to Elad for helping with these!

Added meta prompts for handling the metric generation

A new meta prompt is now available under /prompts/meta_prompts_generation/ for creating helpful metrics.

Added JSON Schema for generated metrics

A new JSON schema has been added to provide a format for metric generation. The generated metrics currently have the following general format:

{
"metrics_list": [
        {
            "metric_name": "...",
            "metric_desc": "...",
            "metric_prompt": "...",
            "metric_scale": [
                {
                    "deficient_desc": "...",
                    "adequate_desc": "...",
                    "competent_desc": "...",
                    "proficient_desc": "...",
                    "exemplary_desc": "..."
                }
            ]
        }, ...
    ]
}

Shout-out to Elad and Michael for feedback and helpful discussions here!