[Feature request] Request to add Vietnamese kanjidic2 and jmdict

[dreamofi]

Greetings!

I have just compiled a Vietnamese version for kanjidc2 and jmdict and would like to add them to Yomichan repo. The files can be found here: jmdict_vietnamese kanjidic_vietnamese

My repo can be found here With my limited capability, the dictionaries' format is not quite pretty, but I think it will be of some value to Vietnamese people learning Japanese.

Thank you for making a great addon for Japanese learner!

[siikamiika]

Hi! Thanks for the contribution.

The convention with the dictionaries is that they aren't included with Yomichan (in repo or release), but rather the user installs them separately. @FooSoft hosts some free dictionaries at his homepage here and Yomichan Import is used to convert dictionaries to the Yomichan format.

Some comments:

• You seem to have created your own tool that does the same thing as Yomichan Import, which is fine, but it might be easier to handle future database changes in Yomichan if it was ported to Yomichan Import. I'm not necessarily asking you to do it.
• I installed the dictionary and it seems to have "\n" in places where there should be newlines. That should be an easy fix if the source data is consistently in this format.
• Your repository mentions that it's compiled from open source dictionaries, but I can't find a license for them. Yomichan already supports proprietary EPWING J-J dictionaries through Yomichan Import, but it's expected that the user has their own dictionary and does the conversion themselves. I think the license or a link to the original source should be mentioned somewhere when the dictionary data and not just the conversion tool is distributed.

After fixing at least the newline bug and licensing issues, I would approve linking to your repository from the readme.

[FooSoft]

@siikamiika pretty much hit the nail on the head there 😄

• My biggest concern is licensing. Even though a dictionary may be free, there can still be some strings attached; for example, in the case of JMDICT, I have to properly credit the original source, or Yomichan would technically not be in compliance.
• Adding the conversion code to Yomichan Import would be the ideal route, since then only one tool would have to be modified for future changes in the dictionary format. This would probably be pretty easy to do, since Yomichan Import already handles sqlite databases for other dictionary formats.

As @siikamiika points out, probably the best route would be to link your repository from the README, thus putting all the licensing and future maintenance requirements on you. I would definitely love to officially adopt the dictionaries, but in order for that to happen, the licensing situation would have to be clarified, and the conversion code should be added to Yomichan Import.

Funny story: I actually got takedown requests in the past when I was hosting an Italian version of JMDICT =\

[dreamofi]

Thank you @siikamiika and @FooSoft, I will work on the licensing and other issues shortly.

Initially, I wrote the script as I hadn't known that Yomichan-import can handle SQLite (do you know where I can read about the structure of sqlitedb for yomichan-import?).

I'm still pretty much a noob, so to prepare the sqlite db for the script I did the following steps:

• First I gathered some dictionaries in stardict format
• Then converted them to csv with PyGlossary
• Then parsed the csv files manually by running regex after regex in a text editor
• Then imported them into Sqlite using DB Browser for SQLite
• Then run the script to add the terms to Yomichan jmdict_english and kanji_english

Edit: When I read the source code of Yomichan-import, it seems that it support old rikai sqlite format! That will surely make things easier for me now.

[FooSoft]

@dreamofi check out https://github.com/FooSoft/yomichan-import/blob/master/rikai.go#L69, you would simply need to create a new file for parsing your database format. The other option is to import them from CSV; that would reduce the amount of transformations needed from stardict.

A quick search also found https://github.com/dyatlov/gostardict, maybe it is possible to leverage that through Yomichan Import?

[tomtung]

I'm still pretty much a noob, so to prepare the sqlite db for the script I did the following steps:

• First I gathered some dictionaries in stardict format
• Then converted them to csv with PyGlossary
• Then parsed the csv files manually by running regex after regex in a text editor
• Then imported them into Sqlite using DB Browser for SQLite
• Then run the script to add the terms to Yomichan jmdict_english and kanji_english

I followed the link to PyGlossary, and was glad to find that it already supports reading from a lot of different dictionary formats. I contributed a plugin to add support for writing in Yomichan's dictionary format, and have successfully converted a few of my ja-to-ja and ja-to-zh dictionaries that were originally in mdict format. Other people with similar needs (e.g. #1911, #2002) might find it useful too.