eiwa / 英和

Parses two types of Japanese-English dictionaries:

:jmdict_e - JMDict's English-only export of the WWWJDIC online Japanese dictionary.
:kanjidic2 - the KANJIDIC2 dictionary of roughly 13,000 kanji characters

Usage

Install

Install the gem:

gem install eiwa

Or add it to your Gemfile:

gem 'eiwa'

Download a supported dictionary

Get your hands on a supported dictionary. Right now eiwa only parses JMDict, which can be fetched from the EDRDG ftp site or with a script like this, for the Japanese-English export:

# Download JMDICT-E:
$ curl http://ftp.edrdg.org/pub/Nihongo/JMdict_e.gz -o jmdict.xml.gz"
# Unzip to jmdict.xml
$ gunzip jmdict.xml.gz

# Download KANJIDIC2:
$ curl http://www.edrdg.org/kanjidic/kanjidic2.xml.gz -o kanjidic2.xml.gz
# Unzip to kanjidic2.xml
$ gunzip kanjidic2.xml.gz

These files are updated daily, and are essentially an export of all vocabulary and kanji in the WWWJDIC application

Parse the dictionary

The eiwa gem implements an evented SAX parser via nokogiri to efficiently work through the very large XML file, as loading a full DOM into memory is very resource-intensive. In consideration of this, eiwa's parsing method provides two modes, one that will return every dictionary entry in an array and one that will invoke a provided block with each entry, but which won't retain a reference to the entries, allowing Ruby to garbage collect them as it goes.

Passing a block

If you just want to do some processing on each entry, it probably makes sense to invoke the library by passing a block (note that supported types include only :jmdict_e and :kanjidic2)

Eiwa.parse_file("path/to/some.xml", type: :jmdict_e) do |entry|
  # Do something with that entry
end

This approach can parse the entire JMDICT-E dictionary in a 15MB Ruby 2.6 process.

Return the results in an array

If you're just going to add all the entries to an array or otherwise retain them in memory, you can call the same method without a block, and it will return all the entries in an array.

entries = Eiwa.parse_file("path/to/some.xml", type: :jmdict_e)

Note that for the abridged Japanese-English dictionary, this will consume about 500MB of RAM.

searls / eiwa

readme