Create dataset KALIMAT

[albertvillanova]

Source: Masader Project

• uid: kalimat
• entry: https://arbml.github.io/masader/card.html?42
• Link: https://sourceforge.net/projects/kalimat/
• License : custom
• Year: 2013
• Language: ar
• Dialect: ar-MSA: (Arabic (Modern Standard Arabic))
• Domain: news articles
• Form: text
• Collection Style: crawling
• Description: 20,291 Arabic articles collected from the Omani newspaper Alwatan
• Volume: 20,291
• Unit: documents
• Ethical Risks: Low
• Provider: Multiple institutions
• Derived From:
• Paper Title: KALIMAT a Multipurpose Arabic Corpus
• Paper Link: https://eprints.lancs.ac.uk/id/eprint/71282/1/KALIMAT_ELHAJ_KOULALI.pdf
• Script: Arab
• Tokenized: No
• Host: sourceforge
• Access: Free
• Cost:
• Test Split: Yes
• Tasks: topic classification,summarization,named entity recognition,part of speech tagging,morphological analysis
• Evaluation Set?:
• Venue Title: other
• Citations: 30
• Venue Type: preprint
• Venue Name:
• authors: Mahmoud El-Haj,R. Koulali
• affiliations: Lancaster University,
• abstract: Resources, such as corpora, are important for researchers working on Arabic Natural Language Processing (NLP) (Al-Sulaiti et al. 2006). For this reason we came up with the idea of generating an Arabic multipurpose corpus, which we call KALIMAT (Arabic transliteration of “WORDS”). The automatically created corpus could benefit researchers working on different Arabic NLP areas. In our work on Arabic we developed, enhanced and tested many Arabic NLP tools. We tuned these tools to provide high quality results. The tools include auto-summarisers, Part of Speech Tagger, Morphological Analyser and Named Entity Recognition (NER). We ran these tools using the same document collection. We provide the output corpus freely for researchers to evaluate their work and to run experiments for different Arabic NLP purposes using one corpus.
• Added by :
• Notes:

[albertvillanova]

DONE: https://huggingface.co/datasets/bigscience-catalogue-data/kalimat

ds = load_dataset("bigscience-catalogue-data/kalimat", split="train", streaming=True, use_auth_token=True)
item = next(iter(ds))
item

output:

{'text': 'كتب'}

[mariosasko]

self-assign

[mariosasko]

Done! LM repo: https://huggingface.co/datasets/bigscience-catalogue-lm-data/lm_ar_kalimat

[albertvillanova]

Thanks @mariosasko