Refactoring

[malteos]

Model/tokenizer support

• [ ] BERT
• [ ] GPT

Languages: en, de, fr, it, es

Tasks: Verify scores with reported scores in the literature

• Verify with bert-base-multilingual-cased , xlm-roberta-base

• [x] XNLI: en, de, fr, es (missing: it)

  • [x] mBERT acc_en = 80.8; acc_de=70

• [ ] XQUAD: en, de, es (missing: it, fr) no train splitt

• [x] PAWSX: en, de, es, fr (missing: it)

  • [x] mBERT acc_en=94; acc_de=85.7

• [x] XStance: de, fr, it (missing: en, es)

  • [x] mBERT (table 3): macro f1_de = 76; it = 70

• [ ] UDPOS: en, de, fr, it (missing: es)

• [ ] XTREME MLQA:

  • [ ] mBERT: f1_en=80; f1_de=59

• [x] XTREME PANX NER: en, de, es, fr, it

  • [x] mBERT: f1_en=85; f1_es=77.4

• [x] multi_eurlex: en, de, fr, it, es

  • [x] xlm-roberta (table 18; all level - fine tune only in English and test in other languages; scores will be different): en=43; fr=30

[malteos]

Valid runs

#MODEL=xlm-roberta-base
#MODEL=bert-base-multilingual-cased
MODELS_DIR=${DATASETS_DIR}/huggingface_transformers/pytorch

# xnli
python src/finetune_eval_harness/main.py --tasks xnli_en --model_name_or_path ${MODELS_DIR}/bert-base-multilingual-cased \
     --max_train_samples 10 --max_eval_samples 10 \
     --output_dir ./output/verify --results_log_path ./output/verify/metrics --report_to none --overwrite_cache

# pawsx
python src/finetune_eval_harness/main.py --tasks pawsx_en --model_name_or_path ${MODELS_DIR}/bert-base-multilingual-cased \
     --max_train_samples 10 --max_eval_samples 10 \
     --output_dir ./output/verify --results_log_path ./output/verify/metrics --report_to none --overwrite_cache