Add Voyage multilingual datasets

[Muennighoff]

Lots of multilingual datasets listed here https://docs.google.com/spreadsheets/d/1qf0iYejG-9RgEEi13qB_SK_178-eNaeJDmSDNSj260A/edit?gid=1875159366#gid=1875159366 from https://blog.voyageai.com/2024/06/10/voyage-multilingual-2-multilingual-embedding-model/ ; I imagine some of them are not in MTEB yet; would be great to have them 🙌

[KennethEnevoldsen]

I know some of these are already covered, and some of them I can't seem to find (dan_news_summ_test). Do we have more references on these?

For convenience here is a list (I have not checked all in this list):

FRENCH

(@imenelydiaker can you have a look at these)

• [ ] french-open-fiscal-texts
• [ ] french_boolq
• [ ] bank-es-2401
• [ ] hc3_french_ood_faq_fr_gouv
• [ ] french_trivia_qa_with_wikicontext

GERMAN

• [ ] german-court-decisions
• [ ] germanrag
• [ ] Biology_German_DHBW
• [ ] german_legal_sentences
• [ ] Dialogsum-german
• [ ] german_seahorse_dataset_with_articles
• [ ] health_care_german
• [ ] German_Poems
• [ ] German_Songs
• [ ] job_listing_german_cleaned_bert
• [ ] german_OpenOrca
• [ ] german_press_releases_100

JAPANESE

• [ ] mmarco-japanese-hard-negatives
• [ ] japanese-conala
• [ ] sakura_japanese_dataset
• [ ] japan-law
• [ ] llm-japanese-dataset_wikinews
• [ ] fujiki_49k_japanese_dataset
• [ ] JapaneseSummalization_task
• [ ] japan_diet_q_and_a_sessions_20k

KOREAN

• [ ] Korean-Human-Judgements
• [ ] korean_dialog_summary
• [ ] KoreanSummarizeAiHub
• [ ] alpaca-korean
• [ ] grade_school_math_korean
• [ ] Korean_QA_gen_datasets
• [ ] korean_rlhf_dataset
• [ ] korean_law_open_data_precedents

SPANISH

• [ ] mlsum-spanish-truncated-512
• [ ] spanish-alpaca
• [ ] colmbian_spanish_news
• [ ] orca-math-word-problems-0_10002-spanish
• [ ] guanaco-spanish-dataset
• [ ] emotional_response_spanish_dataset
• [ ] alpaca-spanish
• [ ] Curated-Spanish

OTHER

• [ ] bengali_summ_test
• [ ] dan_news_summ_test
• [ ] dutch_policy_qa_test
• [ ] georgian_faq
• [ ] greek_civics_qa
• [ ] hungarian_summ
• [ ] ilpost_test
• [x] norwegian_snl_summ_test
• [ ] polish_summ_test
• [ ] rojtvheadlines
• [ ] russian_reviews
• [x] slovak_summ_test
• [x] swedish_swefaq
• [ ] thai_summ_test
• [ ] turkish_HistQuAD_test
• [ ] urdu_summ_test
• [ ] viet_quad_test
• [ ] arabic_news
• [ ] czech_summ
• [ ] persian_qa
• [ ] portuguese_xlsum_test

ENGLISH

• [ ] OneSignal
• [ ] PyTorch1024
• [ ] 5GEdge
• [ ] Cohere
• [ ] Doordash
• [ ] Healthforcalifornia
• [ ] Langchain
• [ ] Huffpostscience
• [ ] Huffpostsports
• [ ] gov_report
• [ ] SummScreen
• [ ] qasper
• [ ] qasper_abstract_doc
• [ ] LeetCodeCpp
• [ ] LeetCodeJava
• [ ] LeetCodePython
• [ ] humaneval
• [ ] mbpp
• [x] LegalQuAD
• [ ] AILA_casedocs
• [ ] AILA_statutes
• [ ] rag-benchmark-finance-apple-10K-2022
• [ ] financebench
• [ ] TAT-QA
• [ ] ConvFinQA