Create dataset AOC

[albertvillanova]

Source: Masader Project

• uid: arabic_online_commentary
• entry: https://arbml.github.io/masader/card.html?39
• Link: https://github.com/sjeblee/AOC
• License : unknown
• Year: 2011
• Language: ar
• Dialect: other
• Domain: news articles
• Form: text
• Collection Style: crawling and annotation(other)
• Description: a 52M-word monolingual dataset rich in dialectal content
• Volume: 108,000
• Unit: sentences
• Ethical Risks: Low
• Provider: Johns Hopkins University
• Derived From:
• Paper Title: The Arabic Online Commentary Dataset: an Annotated Dataset of Informal Arabic with High Dialectal Content
• Paper Link: https://aclanthology.org/P11-2007.pdf
• Script: Arab
• Tokenized: No
• Host: GitHub
• Access: Free
• Cost:
• Test Split: No
• Tasks: dialect identification
• Evaluation Set?:
• Venue Title: ACL
• Citations: 147
• Venue Type: conference
• Venue Name: Assofications of computation linguisitcs
• authors: Omar Zaidan,Chris Callison-Burch
• affiliations: ,
• abstract: The written form of Arabic, Modern Standard Arabic (MSA), differs quite a bit from the spoken dialects of Arabic, which are the true "native" languages of Arabic speakers used in daily life. However, due to MSA's prevalence in written form, almost all Arabic datasets have predominantly MSA content. We present the Arabic Online Commentary Dataset, a 52M-word monolingual dataset rich in dialectal content, and we describe our long-term annotation effort to identify the dialect level (and dialect itself) in each sentence of the dataset. So far, we have labeled 108K sentences, 41% of which as having dialectal content. We also present experimental results on the task of automatic dialect identification, using the collected labels for training and evaluation.
• Added by :
• Notes:

[cakiki]

@albertvillanova

From a footnote in the paper: Data URL: http://cs.jhu.edu/˜ozaidan/RCLMT/ Link leads to a 404; not sure if the data is available anywhere.

The linked repo (does not belong to any of the authors) does not seem to be complete (paper mentions three data sources, I can only see two of those) and the data itself is just raw HTML; I don't think it's the data this issue mentions. Maybe someone else can corroborate.

[apergo-ai]

self-assign

[apergo-ai]

I contacted the author

[albertvillanova]

Thanks @cakiki and @apergo-ai. Any update from the author?