Evaluation script added

[pilot-j]

Evaluation Script: Prompt Quality Assessment

Use eval_script.py to evaluate the quality of responses based on custom prompts.

Arguments:

• --base_path: Directory for output CSV reports.
• --eval_csv: Path to the evaluation metrics CSV.
• --prompt_file: Path to a .txt file containing a list of prompts.

Prompt Class Description

The Prompt class is designed to encapsulate and structure the information needed for generating and evaluating language model prompts. It consists of three key attributes:

• translate_to: (String) Specifies the target language or action for the prompt. For example, "Hindi" or "Spanish".

• preamble: (String) Provides the introductory text or instructions that guide the interpretation of the message. For example, "Translate" or "Translate to Spanish".

• message: (String) Contains the main content or input text that needs to be processed or translated as per the translate_to instruction.

Example Usage:

Given a prompt like:

prompt = Prompt(translate_to="Hindi", preamble="Translate", message="Input")

This creates a prompt where the input text "Input" is instructed to be translated to Hindi. The Prompt class structures this data to be used effectively within the evaluation script.

Prompt Format: Each prompt is a list of three strings: translate_to, preamble, and message, wrapped into a Prompt class. Example prompt file content:

[['Hindi', 'Translate', 'Input'], ['Spanish', 'Translate to Spanish', 'Input Text']]

Output: Generates evaluation reports named as:

prompt_<prompt_no>_<language>_eval_report.csv

Example: prompt_0_Hindi_eval_report.csv

Usage:

python3 eval_script.py --base_path ./output_reports --eval_csv ./evaluation_dataset.csv --prompt_file ./prompts.txt

[pilot-j]

Great Work. The following are the minor changes that needs to be done:

1. Update the requirements.txt file.
2. Remove unused import statements.
3. Remove unneccessary arguments.
4. If possible, write the datatypes of the arguments like other files translation/verification.py file.

Necessary changes done. Please review and let me know.