Open achibb opened 4 months ago
Hi @achibb we very much welcome contributions.
There is a guide to adding a dataset here. Generally, we want a new dataset to cover something previously not covered by the benchmark. This indeed sound like such a case.
To me, it seems like a pair classification task where the pairs would be as follows:
(input sentence, true_synonym, label=positive) (input sentence, false_friend, label=negative)
I'm interested and would like to offer any help with this. Happy to collaborate or take on any tasks needed. Let me know!
Wonderful @KranthiGV! I believe there are some potential good additions in the Aya and OpenAssistant datasets (large multilingual). If you start working on them please create an issue so that others can see that you are working on it. Alternatively, if you are less interested in dataset creation and more interested in improving the codebase then we also have a few tasks there.
@achibb do you have the time to add this dataset?
Yes it is on the plan for today or latest next week. Is this fine or does this intervene with some release ?
Gesendet von Outlook für iOShttps://aka.ms/o0ukef
Von: Kenneth Enevoldsen @.> Gesendet: Friday, April 12, 2024 11:32:45 AM An: embeddings-benchmark/mteb @.> Cc: achibb @.>; Mention @.> Betreff: Re: [embeddings-benchmark/mteb] Adding False Friends Dataset to German MTEB (Issue #331)
Wonderful @KranthiGVhttps://github.com/KranthiGV! I believe there are some potential good additions in the Aya and OpenAssistant datasets (large multilingual). If you start working on them please create an issue so that others can see that you are working on it. Alternatively, if you are less interested in dataset creation and more interested in improving the codebase then we also have a few tasks there.
@achibbhttps://github.com/achibb do you have the time to add this dataset?
— Reply to this email directly, view it on GitHubhttps://github.com/embeddings-benchmark/mteb/issues/331#issuecomment-2051397283, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKBF2KS6THNV43AZSHHALTDY46S33AVCNFSM6AAAAABF6SFYGSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJRGM4TOMRYGM. You are receiving this because you were mentioned.Message ID: @.***>
The PR for false friends as pair classification model:
https://github.com/embeddings-benchmark/mteb/pull/349
tests so far look aligned with the research, even though the method is slightly edited. What do you think? Anything i need to do else? :) Thinking about doing such tests for other languages as well.
Thanks @achibb - Will respond to the above in the PR
Nice thanks @KennethEnevoldsen I adjusted it except for two points where I have not found a use of the metadata in any file I looked. So if given some feedback I would work it in but would also be fine for me to go with the current status. Also excited to see what @Muennighoff thinks in including it on run_mteb_german.py and benchmarks as I see this as a quite nice opportunity for adversarial testing :) Enjoy the weekend guys!
Yeah metadata in the old datasets is not great (we are working on it!), but the documentation for adding a new dataset have some examples.
Love the adversarial testing part! Would love to see more datasets like this one
Yeah no worries :).
Yeah totally. I am also trying to contribute and am currently forming a group for multilingual stuff at hf.co/multilingual
feel free to join anyone who wants and has time and would love to see some more languages creating false friends tests.
Ahh that is great. Is it targeted at adversarial-style tests?
Hi all,
dataset: https://huggingface.co/datasets/aari1995/false_friends_en_de
associated paper
this dataset can be used to test german and especially multilingual models for their understanding capabilities of "false-friends". false friends are words that sound the same / spelled similarly but mean different in different languages.
Example false friend: german: boot (boat) english: boot (shoe)
The evaluation is the following:
input sentence: ich lebe auf einem boot (i live on a boat) true_synonym: ich lebe auf einem schiff (i live on a ship) false_friend: ich lebe auf einem schuh (i live on a shoe)
multilingual models mostly perform worse, bad ones even worse than chance, which indicates they are primarily good for english and prefer english representations.
May I add it to the german MTEB tasks and if so, what should I consider ? Is it a reranking task?
Thanks and all the best Aaron