Extending the dataset to other languages - Githubissues

KennethEnevoldsen / scandinavian-embedding-benchmark

A Scandinavian Benchmark for sentence embeddings

https://kennethenevoldsen.github.io/scandinavian-embedding-benchmark/

MIT License

27 stars 3 forks source link

Extending the dataset to other languages #168

Open KennethEnevoldsen opened 6 months ago

KennethEnevoldsen commented 6 months ago

Extending the dataset to other Scandinavian languages

These resources should be checked before implementing on whether they are translated or not:

Greenlandic
- Danish-Greenlandic
- Greenlandic news
Icelandic
- QA: https://huggingface.co/datasets/vesteinn/icelandic-qa-NQiI
- ScaLA
- News: https://huggingface.co/datasets/thors/RRN
- https://huggingface.co/datasets/mideind/icelandic-error-corpus-IceEC
- https://huggingface.co/datasets/vesteinn/icelandic-parallel-abstracts-corpus-IPAC
- Potentially: https://huggingface.co/datasets/mideind/icelandic-english-translation and its reverse: https://huggingface.co/datasets/mideind/english-icelandic-translation
- translation with localization: https://huggingface.co/datasets/mideind/icelandic-winogrande
- Unsure what this is: https://huggingface.co/datasets/mideind/icelandic-sentences-gec
Faroese
- ScaLA
- https://huggingface.co/datasets/strombergnlp/itu_faroese_danish
- (translation): https://huggingface.co/datasets/vesteinn/faroese-sts
- https://huggingface.co/datasets/vesteinn/faroese-parallel-bible
- potentially some structure in: https://www.openslr.org/125/
Potentially Finnish as well:
https://github.com/TurkuNLP/FIN-bench/tree/main/benchmark_tasks/emotions (the remainder of the dataset seems to be translated)
Potentially also the open Assistant datasets: https://huggingface.co/datasets/mkayhko/oasst2-finnish-threads

Anything beyond this the benchmark should probably be renamed.