Use node_modules as plugin source

danielo515 commented 7 years ago

Hello,

This is more like an open discussion about how could we make this possible. Any suggestion and/or instructions about how to make this will be welcome.

The fact is: it is hard to distribute tiddlywiki plugins, especially for development. If you want to use a third party plugin on a wiki under node js there are not many comfortable ways. Cloning the repository of the plugin and making a symlink to the dist folder of the plugin is one of them. Another option is to manually import the plugin, but then you will have to manually do it every time you want to update the plugin, and pack it with your plugin when you distribute it.

Wouldn't be ideal to just declare your plugin dependencies on a package.json? Something like this:

"dependencies": {
   "tiddlywiki":"^5.1.15",
   "tiddlymap":"*",
   "another-plugin":"1.0.0",
   "d3":"*"
}

This will make starting developing tiddlywiki very easy, all nodejs developers will feel right at home, and instructions about how to use x or y third party plugin on your nodejs tiddlywiki will be much simpler. Also the distribution of specific configurations will become a trivial task. Not to mention that having a bunch of packages related to tiddlywiki on npmjs.org would make tiddlywiki feel more alive and will increase discoverability .

Currently the format of plugin folders is very strict and restrictive, making a configuration like the one described above a manual hard task.

I know about the environment variable TIDDLYWIKI_PLUGIN_PATH, but just pointing that variable to node_modules is not enough, because the naming restrictions plugins have:

All the plugins of the same vendor should be under the same folder
No recursion is possible at the moment
You will need to add each plugin path manually to the variable, totally defeating the purpose

Those restrictions would make impossible to distribute individual plugins as node packages. Maybe it would be possible to distribute collections of plugins. This second approach have several disadvantages:

You are forced to download an entire plugin collection just for one plugin
You can not decide which version of certain plugin you want, you are forced to use the one contained at the collection
I would be forced to name my plugin with a quite weird namespace. I don't want to publish to npm a package named danielo515 containing all my tiddlywiki plugins. I should use tw5-danielo515 and then name mu plugins tw5-danielo515/tiddlypouch and so on.

Thoughts? Corrections ? suggestions?

pmario commented 7 years ago

Cloning the repository of the plugin and making a symlink to the dist folder of the plugin is one of them.

wrong way! We have TIDDLYWIKI environment variables. see: http://tiddlywiki.com/#Environment%20Variables%20on%20Node.js It's simple to set them, so tiddlywiki will find them automatically. .. I do it this way and it works flawlessly.

pmario commented 7 years ago

This will make starting developing tiddlywiki very easy, all nodejs developers will feel right at home,

IMO the main problem here is, that the node directory structure and TW directory structure don't match atm. ... IMO the node_modules structure and the existing dependency resolutions sucks. Especially on windows systems, where it causes path names that the os can't handle well.

pmario commented 7 years ago

... don't understand me wrong. ... I'm in favour of having a simple plugin distribution mechanism. ...

I just think that npm isn't the right tool atm. TW plugins can't be used without TW core. ... so they are pretty much useless for existing node devs. .. So they may be more confused, than happy about the stuff. .. just some thoughts.

danielo515 commented 7 years ago

Dear @pmario

IMO the main problem here is, that the node directory structure and TW directory structure don't match atm

The entire issue is about finding a solution to that problem.

wrong way! We have TIDDLYWIKI environment variables. see: I do that too, but it is also cumbersome and very manual . But the way you get the code is the same, clone the repo if it is available.

I just think that npm isn't the right tool atm. TW plugins can't be used without TW core. ... so they are pretty much useless

That is not a problem at all. There are many projects that use npm to distribute their plugins and there is nothing wrong about it. Have you ever used a gulp plugin without gulp ? Have you ever installed a yeoman generator without having yeoman installed ? Typescript type definitions ? PouchDB plugins ? Babel plugins ? Angular extensions ? Just to name a few, and I don't see anyone going crazy about that .

Maybe npm is not the perfect tool for the task, but it is the best we have , with a widespread community and totally free.

sukima commented 7 years ago

I personally would support an effort to add npm compatibility to TiddlyWiki. One of the driving force for many open source projects is plugins. For example Ember.JS would not be where it is today if it were not for its thriving addon community.

I realize the complications involved. Especially the need for a consistent (and enforceable) set of plugin standards. This feature would be a huge lift. I still feel it is worth discussing.

Can we start collecting requirements and technical gotchas. Maybe addressing the smaller details individually will shed more light on the topic.

danielo515 commented 7 years ago

Implementing #2835 will help with this. For example using a tiddlywiki command that installs the plugin from npm and then adds the plugin dist folder to the plugins path array.

sukima commented 7 years ago

In the Ember world an addon is an NPM module which includes the following in their package.json:

"keywords": [
  "ember-addon"
],

When the build starts it will search the node_modules for any packages with that in their package.json and know those are addons specific for Ember and not normal npm modules.

Much like TW plugins are different you can use npm to host and distribute but they are inert unless run with the tiddlywiki command which is smart enough to hunt out those modules with the tiddlywiki-plugin keyword.

danielo515 commented 7 years ago

@sukima what do you think about my proposal at #2835? Do you see it as a viable approach for this ? What I like about your proposal is that it scans the available plugins at each start

andypalmer commented 6 years ago

I've been experimenting with TiddlyWiki and had the same desire to add plugins and themes to the package.json file. I'm also installing and running TiddlyWiki from the package.json file rather than installing globally (i.e. npm run tiddlywiki -- . --server). Being able to install the wiki-core and plugins via package.json means that we're describing the infrastructure and the content separately, which I like in a separation of concerns way.

While based in Ruby, the https://github.com/twp/little-plugger mechanism is a nice one for plugin fetching and discovery by the core. There may be an equivalent in node that could help add this functionality to TW

linonetwo commented 3 years ago

Github as plugin storage and discovery

I take part in developing https://github.com/tinytacoteam/zazu before, it uses GitHub as plugin storage, if you put a GitHub repo name into the config file's plugin field, it will use downloader ( https://github.com/bayleeadamoss/zazu/blob/master/app/lib/download.js ) to clone the GitHub repo to Zazu's config dir.

And the Zazu will use npm to programmatically install the repo's dependencies, and load that plugin.

I think this GitHub-downloader mechanism can be a good way to install plugins for the NodeJs wiki. And we can use https://stackoverflow.com/questions/60005174/how-to-search-by-tag-with-the-github-search-api to search for a repo that contains a TiddlyWiki plugin. Just add "tiddlywiki" and "tiddlywiki-plugin" as repo topics, we can get a list of plugin repo. And we can also support GitLab API to search for more plugins that published on the GitLab.

But each repo may have different structure, so I think we need a package.json file, with a field, I suggest using "tiddlywiki-plugin", as a pointer field, just like how "main" and "typings" works:

  "main": "index.js",
  "typings": "threads-plugin.d.ts",
  "tiddlywiki-plugin": "src/backlink-plus"

How to implement

As a prototype, I will first implement a temporary mechanism in TiddlyGit's environment, TiddlyGit exposes git APIs to the TiddlyWiki. So with git API, TW can search GitHub to list plugins, and can click a button to clone the git repo into a temp folder, and copy&paste the downloaded plugin folder into your wiki's "plugin" folder.

Later, there can be a nodejs wiki plugin, check for local git existence, if git is installed, it will try to do what TW can do in TiddlyGit environment.

About npm

Similarly, we can use npm as a plugin storage and discovery mechanism, just publish them and add npm tags. But I think everyone who publishes an npm package will have a git repo, and not every git repo owner will have the patient to publish the npm package, so I think we are better to just use Github as plugin storage and discovery.

About plugin repo

To make the mechanism simple, I think we can keep each repo only have one plugin, and have a package.json in the root level. And user can just use the repo name to ask the plugin manager to download a plugin.

Interoperation with the current plugin mechanism

We can keep the current plugin manager, which only manages the local plugins. We can add a page that lists all the online plugins, and manage the download&copy of the online plugin. And after "online plugin manager" write plugin info to the tiddlywiki.info file, it will just hand the plugin to the "local plugin manager".

This is similar to VSCode's extension manager, in that it can list all online plugins in its central plugin library, but also can just list all local installed plugins for you to manage.

simonbaird commented 3 years ago

I don't know anything about zazu, but +1 for the "github repo as plugin distribution mechanism" idea. It works okay for vim. Vim plugin managers let you install vim plugins just by naming their location in github. Maybe something like that could work in TiddlyWiki.

linonetwo commented 3 years ago

This https://www.npmjs.com/package/live-plugin-manager looks promising, we can install plugins on runtime if they are not found in the plugins/ folder, but presented in tiddlywiki.info

joshuafontany commented 2 years ago

I have started experimenting with using NPM to manage plugin packages that are "namespaced", i,e, @tw5/yjs.

This allows me to include the local ./node_modules directory in the TIDDLYWIKI_PLUGIN_PATH env variable. I use npm to install my dev branch of the TW5 repository, and then npm to install the namespaced plugins. The namespace allows this folder structure to match the tw core's requirement of plugin names having 3 parts starting with $:/plugins/.

linonetwo commented 2 years ago

Tidgi is now using https://github.com/tiddly-gittly/TiddlyWiki-CPL to enable "click button" plugin installing, because I think letting new-bee tiddlywiki user to know about npm or tiddlywiki.info file is hard.

But surely this can enable us to dynamically install a large plugin that may depend on NLP data package or native modules. But we may need a new UI to trigger the install.

TiddlyWiki / TiddlyWiki5