Claim level Chinese

[ruixing76]

Claim-level UE for Chinese (Jupyter notebook)

• Claim-level utils adaptation for Chinese
  • greedily match algorithm for Chinese in lm_polygraph.stat_calculators.extract_claims
  • create claim_level_prompts.py that contains all prompts (extract prompts, 2-step match prompts) for English and Chinese
  • add multi-lingual parameters in extract_claims.py and openai_fact_check.py
  • add multi-lingual Deberta implementation in deberta.py
• Support of all claim-level UE methods for Chinese
• Benchmark avalible using following cmd:

  OPENAI_KEY=YOUR_KEY HYDRA_CONFIG=../examples/configs/polygraph_eval_person_bio_zh.yaml python3.10 ./polygraph_eval +model.device=cuda:0 ++model.load_model_args.device_map=0 model.path=01-ai/Yi-6B batch_size=1