Integrate JMTEB

[Muennighoff]

Would be great to have the datasets in JMTEB (https://github.com/sbintuitions/JMTEB) integrated into MTEB for those which aren't yet already, so we can also add a Japanese leaderboard sometime 😊 cc @lsz05 @ryokan0123 @masaya-ohagi & anyone who may be interested

[awinml]

I would like to help integrate JMTEB. The following datasets from JMTEB are already present in MTEB:

• AmazonCounterfactualClassification
• AmazonReviewClassification
• MassiveIntentClassification
• MassiveScenarioClassification
• JSTS
• PawsX PairClassification

The following ones would need to be added:

• Clustering: (#768)

  • [x] Livedoor News
  • [x] MewsC-16-ja

• STS: (#769)

  • [x] JSICK

• Retrieval: (#808)

  • [x] NLP Journal title-abs
  • [x] NLP Journal title-intro
  • [x] NLP Journal abs-intro
  • [x] JaGovFaqs-22k
  • [ ] JAQKET
  • [ ] Mr.TyDi-ja

• Reranking:

  • [ ] ESCI

[AlexeyVatolin]

@Muennighoff, I can take over the integration of ESCI. I have already looked at the contents of this dataset. In the original dataset https://huggingface.co/datasets/tasksource/esci there are three languages: English, Spanish and Japanese. Do we want to add all three languages or just Japanese?

[Muennighoff]

@Muennighoff, I can take over the integration of ESCI. I have already looked at the contents of this dataset. In the original dataset https://huggingface.co/datasets/tasksource/esci there are three languages: English, Spanish and Japanese. Do we want to add all three languages or just Japanese?

I think if we can add all 3 that'd be even better! 🙌