LSDC The Low Saxon Dialect Classification dataset (VarDial 2020)

[ulf1]

Info

• Paper: https://aclanthology.org/2020.vardial-1.3
• Daten: https://github.com/Helsinki-NLP/LSDC

Setup:

• A language detection algorithm for lang:de should cover German dialects and regional varieties, e.g. lang:nds (Lower German), lang:gsw (Swiss German), lang:de-AT (Austrian German).
• The corpus contains 100% lang:nds text examples from regions in North-West Germany and North-East Netherlands. Thus, we add 25% Standard German and 25% Standard Dutch sentence examples with label=0 to the dataset. The result is a 50/50 balance binary classification dataset.

Class labels

• Achterhoek (ACH),
• Drenthe (DRE),
• Groningen (GRO) (not included; i guess copyright issues)
• Hamburg (HAM),
• Holstein (HOL),
• Mark Brandenburg (MAR),
• Mecklenburg-Vorpommern (MKB),
• Münsterland (MON),
• Northern Lower Saxony (NNI),
• Lower Prussia (NPR) (dead dialect)
• Eastphalia (OFL),
• East Frisia (OFR),
• Western Overijssel (OVY),
• Eastern Westphalia (OWL),
• Sauerland (SUD)
• Twente (TWE)

Add random examples for two further class labels

• Standard German (DE)
• Dutch (NL)