Complete e2e Evaluation pipeline

[ArthurMinovsky]

Check list for evaluation pipeline

1. Automatic Evaluation Thai Tasks:

Sequence Classification

• [x] Wisesight Sentiment
• [ ] XNLI (Multilingual)

QA

• [x] XSquad (Multilingual)
• [x] Hellaswaq (Translated to Thai)
• [ ] SuperGlue (Translated to Thai)
  • [ ] copa
  • [x] Record
  • [x] MultiRC
  • [x] RTE (Added)

Classification

• [ ] SuperGlue
  • [ ] RTE (Recognizing Textual Entailment) (Translated to Thai)

Summarization

• [ ] ThaiSum

Translation

• [ ] Please consult with AJ Peerachet to select task for evaluation

Token Classification

• [ ] LST20 (POS, NER, Sentence, Clause boundary)

Expected outcome

1. Convert all tasks above into huggingface dataset format and save to Lanta
2. Translate all English tasks to Thai. Use mUSE or another encoder to improve translation results if necessary
3. Create a prompt template using Prompt Source for all tasks listed above
4. Write a script to evaluate all tasks (Huggingface evaluate library should help) and open PR
   • [ ] Zero Shot
   • [ ] Few Shot The Script should support input as 4.1 Huggingface PretrainedCasualModel name or path 4.2 Dataset name (from the list) to be evaluated Reference: https://github.com/EleutherAI/lm-evaluation-harness

2. Automatic Evaluation English Tasks

• Use lm-evaluation-harness and try to evaluate model on Huggingface dashboard
• Make sure lm-evaluation-harness can run with private Huggingface model (the model in huggingface format but not yet uploaded into huggingface hub)

[pitikorn32]

Colab: https://colab.research.google.com/drive/13LkwsUX6bomzUdHB-LhLQXPp9qHdLgzv?usp=sharing