Evaluation Script

[SuryaKrishna02]

The Issue discusses the Creation of the Evaluation Script.

The following needes to done as part of the Evaluation Script.

• Use the language prompts that are present in the prompt.
• Use the data of the evaluation dataset with similar format of pretraining dataset. (We are in the process of creating this dataset but you can create a sample dataset or directly use the pretraining dataset to run the evaluation script.)
• Fetch the values of the English Sentence in the evaluation dataset and call the Cohere API (c4ai-aya-23) using language prompts to get the translation of the English Sentence.
• Calculate the score either BLEU (My preference) or METEOR or using LLM of how good the translation is using English Sentence and Aya Output.
• Finally, the script should give detailed report of individual sentence translations and overall translation score.

Also, Evaluation script should be able to take input as either single language or multiple languages or all languages and it should be able to give evaluation report on how good the translations are for the prompts.

I am envisioning the script to more flexible to run and easy usable for doing the prompt engineering and figuring out the prompt that works better.

[rematchka]

@SuryaKrishna02 Could you explain more about the task?

[SuryaKrishna02]

@rematchka Updated the Issue description. If you have any more doubts, we can discuss here or you can directly DM on discord and discuss over a short call.

If you feel something can be done in a more better way, Please let me know :)

[Karthikreddyk99]

Yeah lets take the discussion of this topic to Discord and comeback with it.