Hooks for plugin creation and custom js injection at the various stages of yomitan lifecycle

[ganqqwerty]

It would be great to introduce hooks in yomitan for a plugin creation. Here is what I can think of immediately:

• beforeWordScanned(text, word) - called when a user hovers over a text while holding a hotkey.
• beforeWordStemmed - called before the word is turned into its dictionary form
• beforeWordSentForRequest(word) - called before Yomitan starts searching the word in the database
• beforePopupRendered,

... etc.

Why is this important?

• Allowing developers to create plugins will lower the entry bar for contribution to yomitan.
• It will also allow the crowdsourcing of the new features: a person can quickly make a plugin that modifies a certain behavior of yomitan, and if the plugin is used widely, these new features can be moved to the core code of yomitan.
• Allowing finer tuning of the language-specific support. Some features are not necessary to keep in the core of yomitan if they are specific to a certain language. This behaviour could be a part of a plugin.
• There can be multiple point of view on some features, e.g. on the design, see #564 for example. Encapsulating the code in the plugin will allow people to choose the preferred feature.

We saw on countless examples that plugin infrastructure promotes creativity and increases engagement of the community. Think of anki plugins, vim, emacs and VSCode plugins, IntelliJ IDEA, chrome and Firefox.

[toasted-nutbread]

Unclear what exactly is meant by a "hook" here. Are you imagining that other web extensions interact with Yomitan via something like this? Or are you thinking some more along the lines of web pages connecting to Yomitan (#522)?

It's also not clear what you would expect these callbacks to be able to do, which may affect scope and feasibility of how something like this would even begin to be approached.

This also has the potential to introduce pretty significant timing overhead in the application if there are async API hooks running at high-bandwidth code paths. There are also other challenges with regards to where this code actually runs. For example, beforeWordStemmed and beforeWordSentForRequest would only happen on the backend, but the others are on the frontend.

[ganqqwerty]

I think that it's my bad to include any kind of terminology here without spending enough time with the code. The intention of this ticket is to discuss the possibility of creating plugins to Yomitan. How exactly to organize the extension points, should they be synchronous or asynchronous, etc. – I'm not yet qualified to say that.

I've had in mind a simple system similar to what have been done in Wordpress. The js code of yomitan can have calls like that:

// yomitan code here
runHooks("beforeWordScanned", word, text). 
// yomitan execute the code for scanning the word

When the frontend is initialized, the extensions register their callbacks like that:

addHook("beforeWordScanned", function (word, text, otherExposedObjects)  { 
  //hook does its job
} )

I imagine that callback functions are stored in a map like that:

const hookMap = {
   beforeWordScanned: [function, function, function]
}

and the runHooks function basically execute them in the order they are stored. The addHook function appends the new callback to the hookMap.

I'm not at all married to that particular implementation though, it's just one of the ways I saw people organize plugins.

[toasted-nutbread]

The idea of more extensibility for the extension is very interesting, don't get me wrong. I just want to make it be known that it may not be as simple as extensions for e.g. Wordpress might be, due to the nature of the web extension security model. The biggest challenge I see is being able to do things that actually modify the extension page. As it stands, I'm not sure other extensions have any way of modifying the DOM without proxying commands into the Yomitan extension itself, same for access to the JavaScript state internals. So the hooks are possible, I'm just not sure what people would want to do with them.

[ganqqwerty]

These are two questions.

First: what the plugins would be useful for.

I'm just not sure what people would want to do with them.

I was thinking about the following categories of plugins:

• plugins that modify the way yomitan looks (skins?). Some things are possible now with custom css, but we don't have custom html or custom js. Different people have different opinions on how stuff should look like.
• plugins that include nice-to-have features that core developers don't want to support in the core. For example, calling 3rd party services during the lookup (e.g. calling google images or anki connect, or some cool API, or ChatGPT).
• plugins that modify behavior: for example, using different parser, or stemmer.
• plugins that add additional behavior and store some additional data. For example, stats plugin (how often do I lookup words and which words I looked up more often than other)
• potentially, extended support for specific languages.

Overall, writing a plugin is typically much easier than contributing to the core, where there are all considerations of the scope of the change, accessibility, cross-platform, touch screens, i18n, corner cases, testing and CI, to name a few. Writing a plugin does not require all of that because the the core team is not responsible for the plugins.

I saw a lot of times how plugin developers got more and more familiarized with the code, ending up being core contributors.

[ganqqwerty]

Second question: how to deliver the custom code to yomitan?

The custom css is now delivered to yomitan from within the extension. Where can the user input custom js? I see the following options.

1. In a separate chrome extension. This is a complex way that will not allow doing much because extensions are well isolated. It will also create a need of some kind of async protocol between extensions which make stuff harder and slower. Sounds like a bad option.
2. Delivering custom js from within the yomitan in runtime: the same way custom css is delivered. This would mean creating the plugin loader inside yomitan. The plugin packages could be simple zip files, similar to anki packages. The immediate question is: where will the uploaded code be located and how will it be evaluated? Does the chrome extension security model allows the extension to store arbitrary js files in a file system and execute them normally for both frontend and backend? I think, there's a high possibility that this is not allowed. Another option is to use Function and eval. Overall this option sounds unclear.
3. Delivering custom js in compile time. The plugins and yomitan core are getting built and downloaded on yomitan website side. Some imaginary integration server will combine the code of all plugins and the code of the core yomitan and build a new chrome extension file. This will allow worpress/mediawiki-like simple model of plugins that can easily do whatever they want on both frontend and backend of Yomitan extension. This presents the following challenges:
   • Each time a new plugin is installed, the new Yomitan extension needs to be installed. How will the extension updates work in this case?
   • Who will do the integration? Either the integration server needs to be developed, with the nice frontend on yomitan website, or the whole process takes place on user's machine. If the whole process takes place on the user's machine, is it actually easier than making a proper fork of Yomitan?

[toasted-nutbread]

2. Delivering custom js from within the yomitan in runtime: the same way custom css is delivered. This would mean creating the plugin loader inside yomitan. The plugin packages could be simple zip files, similar to anki packages. The immediate question is: where will the uploaded code be located and how will it be evaluated? Does the chrome extension security model allows the extension to store arbitrary js files in a file system and execute them normally for both frontend and backend? I think, there's a high possibility that this is not allowed. Another option is to use Function and eval. Overall this option sounds unclear.

Just to give a quick feasibility assessment, this option is not possible. Function/eval is the only way to invoke custom JS and web extensions don't allow it.