npm-on-ipfs: the desktop app! A long lived client to encourage co-hosting

olizilla commented 5 years ago

Having teams co-host the modules they use would be rad! But that needs a long running process on the developers machine. We could have the npm-on-ipfs client spawn one, but it's not very visible to the user, and the current expectation of the npm command is that it does its work and then exits, so there would be a UX challenge to alter that.

What if we had a menubar app like ipfs desktop that ran an ipfs daemon process to re-host modules you fetch? It could show you stats about what modules you use a lot, and which ones other folks have fetched from you.

Would this be valuable? Could it help us usher in the network effect of having many smaller, partial caches of popular npm modules available on the network?

olizilla commented 5 years ago

This came up again in the GUI team OKR discussions, but I figured it'd make more sense to discuss it here

daviddias commented 5 years ago

I like it! It would be a great way to have people to feel engaged with the process of replicating the registry.

Also, it would be great if this Desktop app would know how to connect to multiple nodes (e.g. IPFS Desktop, some other local or remote daemon)

achingbrain commented 5 years ago

I think this would be really useful & increase visibility

lidel commented 5 years ago

I am bit worried about adding another desktop app responsible for daemon orchestration and updates. We should avoid creating new apps, and provide more and more incentives for people to use existing one :)

Could this idea could be somehow folded into existing IPFS Desktop? What if..

For gathering local stats: IPFS Desktop exposes a separate port for use in npm-on-ipfs that proxies requests to real Gateway, but in the process of proxying gathers stats on npm downloads
For progressive UX: If this special Gateway registers a hit, it means it is used by npm-on-ipfs, then we could either display NPM stats in Web UI or add a menu item to open a dedicated screen with them.

olizilla commented 5 years ago

it'd be rad to be able to get "bandwidth per CID" stats from IPFS, so we can say "you have shared the async module 6 times". Right now we can only say which peer IDs you've shared with which is less interesting than what you've got that is popular.

https://github.com/ipfs/go-ipfs/issues/4740

olizilla commented 5 years ago

@lidel this could go either way on "npm-on-ipfs.app vs ipfs-desktop + npm skills". It'd be rad to figure out a plugins mechanism for ipfs-desktop, but I think I'd be more likely to install a focused app like "npm-on-ipfs.app" than i would "install ipfs-desktop and enable npm proxing"... but as you mention, it'd be nice not to have to maintain more apps...

I was imagining npm-on-ipfs.app having very little surface area, just a menubar icon that shows you some stats, and an excuse to keep a long running ipfs daemon going... with the npm-on-ipfs cli doing the work.

lidel commented 5 years ago

My main fear is that adding another menubar app will just feel noisy and wasteful to users who also choose to install IPFS Desktop at some point.

Separate app adds unnecessary user confusion and maintenance burden of duplicated features such as auto-update, daemon orchestration: If I have both, do I have two daemons, or one? Which app is the owner of the daemon? Do we promote both apps at https://ipfs.io/#implementations ?

I believe we could design UX so that npm-on-ipfs detects IPFS Desktop and enables "NPM support" in it automatically. It should incentivize people to run a single IPFS node, not multiple ones.

olizilla commented 5 years ago

My assumption is that we would be focusing on "exising npm users", rather than "existing ipfs users". If we can make a really simple and slick UX around ipfs-on-npm then we could get a huge number people co-hosting packages over ipfs.

That said, I do see there is an overhead, both for us to have multiple apps, and potential ipfs + npm-on-ipfs users, and there is a potential win of getting more folks using ipfs desktop. I think either way, we'd build it so that we use an existing daemon where available, so I think the "one daemon vs many" is less of a consideration.

It be worth converging on what our top goal / desired outcome for an npm-on-ipfs app would be. My assumption is it's "large increase in end users co-hosting pacakges via ipfs rather than http". I think the best way to do that is a focused app. If the goal is "get more devs using IPFS in general" then I think we'd probably go for an extention to ipfs desktop.

Or we could do both? A targeted app for the npm user that is decentralisation curious, and an extention to desktop for the IPFS user that wants to share their modules.

hacdias commented 5 years ago

Just a fast feedback: I think we could somehow work a way to add this to IPFS Desktop 😄 instead of creating a new app. Some kind of extension or some kind of option on IPFS Desktop that would download the required additional files for the functionality.

daviddias commented 5 years ago

Some kind of extension or some kind of option on IPFS Desktop that would download the required additional files for the functionality.

AFAIU, If an MFS folder gets exposed on the IPFS Gateway (not the case today), then effectively what an IPFS node needs to do is to keep replicating a CID (through IPNS or registry.js.ipfs.io) and any npm client could just point the install at the local IPFS Gateway.

That said, there are still benefits to have a tiny npm on your laptop (could be an IPFS Desktop theme), it will get people psyched to visually see that they are contributing to distributing the registry

achingbrain commented 5 years ago

I think the user experience could be better & more focussed if it's a new app rather than adding features to the existing app, plus not everyone running IPFS Desktop will be an npm user so they are likely to see the addition of npm-stuff as bloat.

As long as the maintenance burden doesn't get too much I'm vaguely pro new-app.

cwaring commented 5 years ago

Personally I also feel the best path to deliver this functionality now is via the current desktop app, even if there could be a case for separation in the future, starting as a new app will only add confusion and development/maintenance overhead at this stage.

Thinking longer-term; I agree with @lidel on the view of incentivising people to use IPFS Desktop and exposing these experiments in order to attract a new audience. I see npm-on-ipfs much as an extension in vscode so I would push to align our design efforts around creating that experience first, which should in turn open a channel for us to test new x-on-ipfs ideas faster.

agentofuser commented 5 years ago

Sorry to parachute into the conversation, but in my opinion it's worth it considering integrating IPFS as a storage backend for Verdaccio, an open source npm registry/cache/proxy which has been gaining a lot of popularity.

It's super easy to get started with, has a beautiful React-based UI on the way, and integrates seamlessly into regular npm/yarn/private-corporate-registry workflows.

The IPFS plugin could fork from the local-storage plugin, and I think it would be more familiar and low-energy to adopt than a dedicated solution.

If this isn't the right place to suggest it I can open a separate issue and delete this one, np. (I have no affiliation to the project btw.)

olizilla commented 5 years ago

Thanks @agentofuser, I'd not heard of verdaccio but that looks like a great project to integrate with!

jessicaschilling commented 5 years ago

Hopefully not being too simplistic or optimistic here, but wanted to pull a few thoughts out of this thread ...

As @olizilla said:

I was imagining npm-on-ipfs.app having very little surface area, just a menubar icon that shows you some stats, and an excuse to keep a long running ipfs daemon going

As @lidel said:

I believe we could design UX so that npm-on-ipfs detects IPFS Desktop and enables "NPM support" in it automatically

I'm not sure if these would need to be mutually exclusive if designed with a progressive user experience in mind, e.g.:

Some users just want something that's not much different than the ordinary experience of using npm. They benefit from @olizilla's menubar icon model, where they get a little bit in the way of stats and general assurance that the thing is working.
Some users might already be running IPFS Desktop, and they'd benefit the most from having npm-on-ipfs features in IPFS desktop exposed to them automagically once they start using npm-on-ipfs (@lidel model; new features in existing tool rather than new tool)
Some users might be using IPFS in the browser, and could potentially be incented to move to the desktop app through the promise of better integration with npm-on-ipfs in desktop rather than having to go between menubar features + browser features?

Strictly speaking, this would be duplicating feature creation effort for us, but for the user, it's gradually exposing functionality by presenting a lite -> fully-featured progression and a means of wading into the IPFS waters bit by bit.

andrew commented 5 years ago

For npm-on-ipfs to be successful at distributing the load of serving npm traffic it needs lots of long running daemons, to achieve that currently you need to:

install npm-on-ipfs in the command line
install and run a long running ipfs daemon that connects to the registry.js.ipfs.io swarm
remember to use $ ipfs-npm whenever you would use $ npm or $ yarn in the command line plus pass the --ipfs-node option to your long running ipfs daemon
remember to start their ipfs daemon again after restarting their computer

Very few people are going to do all of those things manually every time they go to install something via npm, so having a set-and-forget installer would have a big impact.

Ideally installing the “app” (whether it is ipfs desktop or a separate npm-on-ipfs app) should:

Start a long running ipfs daemon (that connects to the js.registry.ipfs.io swarm)
Prompt to install npm-on-ipfs command line tools, aliasing it as $ npm and $ yarn
Run the npm-on-ipfs express server
Launch on startup

Extra features like stats on sharing are nice but likely beyond the scope of Q2 given IPFS camp and other GUI work.

The other problem that we’ll quickly run into is that for first few users, the initial performance of npm-on-ipfs is going to be significantly worse than using regular npm over http.

The quicker we can get more people sharing packages over IPFS the faster it gets for everyone, reducing the poor first user experience which may discourage those users from telling their friends to install it too.

Adding the option to install+enable npm-on-ipfs to the existing desktop app would give a significant bootstrap effect for adding more users who are already bought into IPFS.

I’m also a big fan of @jessicaschilling’s suggestion, having a progressive pathway into IPFS desktop by having the option to essentially install an npm-specific fork of ipfs desktop that can give them the option to learn more about IPFS when they are ready.

There's also probably some trademark issues around "npm" but IANAL 🙈

TL;DR

First add npm-on-ipfs opt-in support to IPFS Desktop that installs and aliases and configures ipfs-npm to get an initial wave of already primed people sharing packages and testing performance
re-release updated IPFS desktop as a slightly altered, npm branded version for people who aren't already in the IPFS world but think distributed npm is cool

andrew commented 5 years ago

On a related note, I opened up ipfs desktop this morning and within a few minutes the coffee shop wifi had completely fallen over, could be a coincidence but that coincidence seems to happen a lot!

ipfs-shipyard / npm-on-ipfs

npm-on-ipfs: the desktop app! A long lived client to encourage co-hosting #94