Create dataset OSIAN

[albertvillanova]

Source: Masader Project

• uid: osian
• entry: https://arbml.github.io/masader/card.html?25
• Link: http://oujda-nlp-team.net/en/corpora/osian-corpus/
• License : CC BY-NC 4.0
• Year: 2019
• Language: ar
• Dialect: other
• Domain: news articles
• Form: text
• Collection Style: crawling
• Description: The corpus data was collected from international Arabic news websites,
• Volume: 3,500,000
• Unit: documents
• Ethical Risks: Low
• Provider: Multiple institutions
• Derived From:
• Paper Title: OSIAN: Open Source International Arabic News Corpus -Preparation and Integration into the CLARIN-infrastructure
• Paper Link: https://aclanthology.org/W19-4619.pdf
• Script: Arab
• Tokenized: No
• Host: other
• Access: Free
• Cost:
• Test Split: No
• Tasks: text generation
• Evaluation Set?:
• Venue Title: WANLP
• Citations: 15
• Venue Type: workshop
• Venue Name: Arabic Natural Language Processing Workshop
• authors: Imad Zeroual,Dirk Goldhahn,Thomas Eckart,A. Lakhouaja
• affiliations: ,,,
• abstract: The World Wide Web has become a fundamental resource for building large text corpora. Broadcasting platforms such as news websites are rich sources of data regarding diverse topics and form a valuable foundation for research. The Arabic language is extensively utilized on the Web. Still, Arabic is relatively an under-resourced language in terms of availability of freely annotated corpora. This paper presents the first version of the Open Source International Arabic News (OSIAN) corpus. The corpus data was collected from international Arabic news websites, all being freely available on the Web. The corpus consists of about 3.5 million articles comprising more than 37 million sentences and roughly 1 billion tokens. It is encoded in XML; each article is annotated with metadata information. Moreover, each word is annotated with lemma and part-of-speech. the described corpus is processed, archived and published into the CLARIN infrastructure. This publication includes descriptive metadata via OAI-PMH, direct access to the plain text material (available under Creative Commons Attribution-Non-Commercial 4.0 International License - CC BY-NC 4.0), and integration into the WebLicht annotation platform and CLARIN’s Federated Content Search FCS.
• Added by :
• Notes: link doesn't work ?

[cakiki]

@albertvillanova : Seems that the data was never available for download (see this snapshot from last January: https://web.archive.org/web/20210125033548/http://oujda-nlp-team.net/en/corpora/osian-corpus/)

[apergo-ai]

self-assign

[apergo-ai]

The data is available here: https://clarin.informatik.uni-leipzig.de/de?corpusId=ara-international_newscrawl-OSIAN_2018

But it is part of a much larger corpus, which I will try to get into the project.

[albertvillanova]

This needs to be clarified: whether this dataset is accessible and which are the corresponding files.