Json Wikipedia contains code to convert the Wikipedia XML dump into a JSON dump.
What's different about this fork:
mvn assembly:assembly
the command will produce a JAR file containing all the dependencies the target folder.do:
SPARKFOLDER/bin/spark-submit --driver-memory 10G --class it.cnr.isti.hpc.wikipedia.cli.MediawikiToJsonCLI json-wikipedia-
this produces in <PATHTONEWJSONPEDIA>
the JSON version of the dump
You can also call Jsonpedia the usual way but it will use a single thread to process the wiki:
java -cp target/json-wikipedia-1.0.0-jar-with-dependencies.jar it.cnr.isti.hpc.wikipedia.cli.MediawikiToJsonCLI -input wikipedia-dump.xml.bz -output wikipedia-dump.json[.gz] -lang [en|it] -action export
(here you can find an example). Each line of the file contains an article of dump encoded in JSON. Each JSON line can be deserialized in an Article object,which represents an enriched version of the wikitext page. The Article object contains:
Once you have created (or downloaded) the JSON dump (say wikipedia.json
), you can iterate over the articles of the collection
easily using this snippet:
RecordReader<Article> reader = new RecordReader<Article>(
"wikipedia.json",new JsonRecordParser<Article>(Article.class)
).filter(TypeFilter.STD_FILTER);
for (Article a : reader) {
// do what you want with your articles
}
You can also add some filters in order to iterate only on certain articles (in the example we used only the standard type filter, which excludes meta pages e.g., Portal: or User: pages.).
The RecordReader and JsonRecordParser are part of the hpc-utils package.
In order to use these classes, you will have to install json-wikipedia
in your maven repository:
mvn install
and import the project in your new maven project adding the dependency:
<dependency>
<groupId>it.cnr.isti.hpc</groupId>
<artifactId>json-wikipedia</artifactId>
<version>1.0.0</version>
</dependency>
|-- categories: array (nullable = true)
| |-- element: struct (containsNull = false)
| | |-- anchor: string (nullable = true)
| | |-- id: string (nullable = true)
|-- externalLinks: array (nullable = true)
| |-- element: struct (containsNull = false)
| | |-- anchor: string (nullable = true)
| | |-- id: string (nullable = true)
|-- highlights: array (nullable = true)
| |-- element: string (containsNull = false)
|-- infobox: struct (nullable = true)
| |-- anchor: array (nullable = true)
| | |-- element: string (containsNull = false)
| |-- name: string (nullable = true)
|-- integerNamespace: integer (nullable = true)
|-- lang: string (nullable = true)
|-- links: array (nullable = true)
| |-- element: struct (containsNull = false)
| | |-- anchor: string (nullable = true)
| | |-- id: string (nullable = true)
|-- lists: array (nullable = true)
| |-- element: array (containsNull = false)
| | |-- element: string (containsNull = false)
|-- namespace: string (nullable = true)
|-- paragraphs: array (nullable = true)
| |-- element: string (containsNull = false)
|-- redirect: string (nullable = true)
|-- sections: array (nullable = true)
| |-- element: string (containsNull = false)
|-- tables: array (nullable = true)
| |-- element: struct (containsNull = false)
| | |-- name: string (nullable = true)
| | |-- numCols: integer (nullable = true)
| | |-- numRows: integer (nullable = true)
| | |-- table: array (nullable = true)
| | | |-- element: array (containsNull = false)
| | | | |-- element: string (containsNull = false)
|-- templates: array (nullable = true)
| |-- element: struct (containsNull = false)
| | |-- anchor: array (nullable = true)
| | | |-- element: string (containsNull = false)
| | |-- name: string (nullable = true)
|-- templatesSchema: array (nullable = true)
| |-- element: string (containsNull = false)
|-- timestamp: string (nullable = true)
|-- title: string (nullable = true)
|-- type: string (nullable = true)
|-- wid: integer (nullable = true)
|-- wikiTitle: string (nullable = true)
Supporting a new language requires creating a new locale file with specific language information. Most of a locale creation process is automated:
localegen.py
. As an example for generating a locale for German: python localegen.py --lang de --o locale-de.properties
resources/lang
disambiguation=XX