Open hugolpz opened 3 years ago
Discussion engaged with the Wikimedia Foundation's Dumb-Generation project. See : phabricator.wikimedia.org > T276723
Python processing :
Wikicompiler is a fully extensible python library that compile and evaluate text from Wikipedia dump. You can extract text, do text analysis or even evaluate the AST(Abstract Syntax Tree) yourself. Topics: python, compiler, mediawiki, wikipedia, wikitext, wikipedia-dump, wikitext-parser.
One presentation is in italian but has some interesting nugets: here. The gist:
@hugolpz That's impressive.
@GTOqaz there are some upcoming Google crawling on 2000 languages, I hope they will make some data available, especially frequency lists.
There are ready-to-download open licence Wikipedia corpora available.
Project introduction | Type | Languages (2024) | Portal all | Language specific | Download link | Comments |
---|---|---|---|---|---|---|
OpenSubtitles 2016/2018 |
Subtitles Parallel sentences Monolingual sentences |
75 | Portal | br&en |
bre (mono) | '''Source:''' * P. Lison and J. Tiedemann (2016), ''"OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles"'', http://stp.lingfil.uu.se/~joerg/paper/opensubs2016.pdf . '''Licence:''' unclear, "The corpora is made freely available to the research community on the OPUS website" − Lison and Tiedemann (2016). |
Wortschatz by Leipzig | Sentences Monolingual |
290+ | - | bre | bre 100k sentences (2021) | List of sentences corpora : API reference > https://api.wortschatz-leipzig.de/ws/corpora |
CC-100 | Sentences Monolingual |
115 | Portal | n.a. | br (mono) | « No claims of intellectual property are made on the work of preparation of the corpus. » |
A quick search shows you that CorpusCrawler does not crawl or use Wikipedia. I don't know Python but it seems feasible, either from scratch on Wikipedia API (1) or using existing server-side tools (2).
Assess interest
Crawling via API
By using and loading available list of articles per wikipedia, then scrap the sites. If too large, could be limited to
max=n
articles.Given an iso code such as Ndonga's
ng
:Wikipedia API provides text
Various formats available:
format
: The format of the output.jsont
: Output data in JSON format.jsonfmt
: Output data in JSON format (pretty-print in HTML).nonet
: Output nothing.phpt
: Output data in serialised PHP format.phpfmt
: Output data in serialised PHP format (pretty-print in HTML).rawfmt
: Output data, including debugging elements, in JSON format (pretty-print in HTML).xmlt
: Output data in XML format.xmlfmt
: Output data in XML format (pretty-print in HTML).List of Wikipedia (~300)
List of articles per Wikipedia
For convenience, I use the tiny Ndonga (
ng
) Wikipedia (8 articles), easier to explore by hand.For larger demo, you could also inspect similar URLs with the iso of :
Namespaces
On all wikis. See also here
0
: (main)1
: Talk:2
: User:3
: User_talk:Dumps' & paths
Using Wikipedia extractors ?
Hybrid approach
util.py
, code a simple crawler which get just that .zip, convert back to txt content, add to the corpora.cc: @brawer