enabling npm scopes for plugins

dobbs commented 6 years ago

symptom

While working on the calendar plugin, I've tried to deploy a wiki from npm scopes. I've published my forks as:

@dobbse/wiki@v0.12.1-e
@dobbse/wiki-plugin-calendar@v0.2.3-a

My custom wiki modifies package.json to point to my custom wiki-plugin-calendar. All the interesting stuff is in calendar.

Wiki does not find my plugin when installed this way:

$ npm install @dobbse/wiki@v0.12.1-e

I see that npm does install the scoped plugin:

$ find node_modules -type d -name wiki-plugin-calendar
node_modules/@dobbse/wiki/node_modules/@dobbse/wiki-plugin-calendar

wiki --version finds these plugins that start with c, notably excluding calendar:

$ wiki --version | grep plugin-c
wiki-plugin-calculator: 0.2.2
wiki-plugin-changes: 0.2.5
wiki-plugin-chart: 0.3.2
wiki-plugin-code: 0.2.2

cause

npm install puts my scoped package into a scoped directory tree which fails to match the filesystem glob that searches for plugins on disk.

In cli.coffee

# starting at line 110:
config = cc(argv,
...
  cc.env('wiki_'),
...
    packageDir: path.resolve(path.join(__dirname, 'node_modules'))

# a little later on line 142:
  glob 'wiki-plugin-*', {cwd: config.packageDir}, (e, plugins) ->

Here we see that --version searches config.packageDir for folders starting with wiki-plugin-. My scoped package landed in @dobbse/wiki-plugin-calendar.

The @dobbse scope hides my plugin from --version, and presumably from other places too.

potential remedy

I propose to inspect package.json instead of file globbing.

Something like this would work for --version, though similar changes are needed elsewhere:

require('./package').dependencies.keys.reduce(
  ((acc, name) -> acc[name] = require(name).version),
  {}
)

other things I notice

This area of code has been through some other issues:
Plugin management really wants to live in plugmatic.
- Maintaining a Custom Wiki mentions: "Warning: This workflow has proven to be unreliable and will be replaced by plugin support within wiki. It is left here for historical reference. See About Plugmatic Plugin"

There's a good chance I haven't discovered all the things tugging on this code, and warrants futher discussion.

WardCunningham commented 6 years ago

wiki-server and others scan wiki/node_modules looking for wiki-plugin-* and uses what it finds in several ways. I don't recall all the ways but these are surely included:

serving client js upon request
serving factory.json
serving page/*.json
launching server/server.js
reporting in plugmatic

I think all of these could be made to recognize namespaced plugins. Conflicts could be resolved by favoring larger version numbers with perhaps file date breaking ties. We would want to change all the places at once so good reconnaissance will be in order.

We should consider version number suffixes at the same time but not necessarily applying the same methods. This is more a plugmatic issue than wiki-server I would think.

WardCunningham commented 6 years ago

A work-around might be to choose a slightly different name for your plugin while still obeying current conventions. There is certainly something unattractive about this when npm is trying so much harder.

dobbs commented 6 years ago

For my immediate calendar plugin development, I have found some docker-based work-arounds that loosely model the npm link approach you showed me last week.

In this issue I'm thinking about the future wiki pages stewing in the back of my mind to recommend development of plugins and custom wikis. I really like the structure of npm packages and the many options the npm CLI offers for publishing and downloading packages: https://docs.npmjs.com/how-npm-works/packages

My earliest experiment with calendar plugin used github urls in package.json. That worked quite well. I've considered extending my Caddyfile configs to host a small site to serve up tarballs of packages, or even to include those under the new assets folder. I haven't tried tarball options yet, so not positive those packages land on disk where wiki expects to find them.

I found this issue while exploring the npm scoped publishing options.

It may be easier to recommend github or tarball publishing instead of scoped npm publishing for plugin developers who want to share work in progress or build custom solutions.

dobbs commented 6 years ago

I have to add, I'm pretty fond of the idea of tarballing the plugins and serving them out of assets, if it could be proved to work. Makes a nice way for one wiki to host its installed plugins from which another wiki might install them via plugmatic.

WardCunningham commented 6 years ago

Wow. I love the idea of managing plugins as another class of assets. We have to think through the sharing logic to be sure we are not distributing viruses to unsuspecting site operators.

One thing to remember, npm_modules is bound to the server while assets is bound to the site. As such, plugmatic requires admin privilege while site tooling, like flagmatic, requires only site ownership.

I've tried to explore these issues in http://own.fed.wiki but may not have my metaphors right.

paul90 commented 6 years ago

There is some old discussion of this problem in fedwiki/wiki-server#107, and fedwiki/wiki#59

I am not sure if the issue with scoped packages in npm, that I mention back in 2015, was ever solved.

Not sure that I like the idea of using assets to distribute plugins - at least not without sufficient checks that they have not been tampered with.

WardCunningham commented 6 years ago

I heard mention at work yesterday of a "mini npm" which can proxy to the normal site and serve local content in front of it. I'll ask for more details because I couldn't find this on my own.

WardCunningham commented 6 years ago

Perhaps the simplest thing would be to improve plugmatic to safely and conveniently install from git repos directly. The markup would have to recognize this and somehow relate it to our more traditional sources, perhaps with a caution since plugmatic pages do get passed around. The win would be the ability to have colleagues experiment with new versions without publishing them in the standard repo.

dobbs commented 6 years ago

npm supports git urls in package.json. I'm experimenting with that again to see if they land in the right place on disk for wiki to find them. I'll keep you posted.

dobbs commented 6 years ago

@paul90, thanks for pointing to the earlier PR and discussion. I've finally had a chance to review those more carefully. Not surprised this territory was covered rather throughly before. Even as I opened this issue I wondered if parsing package.json would be introducing too much coupling to npm.

I did also get time to experiment with my local, docker-based wiki and found that github names work fine with NPM. Here's my fork of fedwiki/wiki: https://github.com/dobbs/wiki/blob/79fbbae861f7396b72eca9ce81c0faa1c73e30a5/package.json#L40

That points to a branch in my fork of fedwiki/wiki-plugin-calendar where I have also committed the compiled javascript. diff is here (using SHAs instead of branch names for future proofing): https://github.com/fedwiki/wiki-plugin-calendar/compare/0a25fe5...dobbs:34eb620

Works okay, but needs docs to explain how and why to compile the javascript and put it in a special branch.

dobbs commented 6 years ago

The code as-is supports development of plugins to proceed independently of the official published plugins, thanks to npm's support for pulling packages from github.
The question about distributing plugins from the assets folder does raise difficult questions about integrity and security of the code. Seems worth deferring that work for a while. There are quite a few different projects trying to build integrity and security into peer-to-peer systems using public keys, block chains, or other cryptographic gears. Maybe let all of that mature a bit and see how it can be applied to federated plugin distribution.

So, do you think we should adjust the code to find plugins within scoped packages?

It would open another path for independent plugin development. Maybe not a big gain.

It is not a small amount of work.

WardCunningham commented 6 years ago

I'll see if I can add git repos to plugmatic this weekend. It will be easier to figure out all of the consequences if we just start using it. Some things to think about:

[ ] must checkin built js files
[ ] don't be github specific
[ ] traffic light color? blue?
[ ] merge with npm versions
[ ] warning at install button

WardCunningham commented 6 years ago

Couldn't get started on this. Wrote some more thoughts instead. http://ward.asia.wiki.org/experimental-plugin-deployment.html

paul90 commented 6 years ago

A possible alternative might be to use a private npm repository like sinopia - there is a section on override public packages which might be what you are looking for.

I have not found anything more recent, though Ward had a recently conversation that mentioned something similar - but we couldn't find it when we looked yesterday.

dobbs commented 6 years ago

Sinopia work seems to have moved to https://github.com/verdaccio/verdaccio That does look promising.

Following that thread taught me a fair bit about npm registry and configuration that I thought I'd move to wiki: https://wiki.dbbs.co/private-npm-registry.html

WardCunningham commented 6 years ago

I found the micro-npm server that is properly spelt unpm.

https://www.npmjs.com/package/unpm

The author, @hayes, has worked for us at New Relic which may be why we've employed this code.

fedwiki / wiki