IPFS as a modular plug and play service system

[mar1n3r0]

Create open, modular and extendable boilerplate kit from go-ipfs monolith

Description

At present go-ipfs is a single monolithic binary and any attempts to make it more modular end up as new APIs rather than a full revision of the architecture. This prevents the exploration of new use cases such as using IPFS as a decentralized backend with plug and play support for custom components that serves self-contained WASM client apps in the browser.

Example: a wasm app can not access the host OS resources. Because of this we need to call the go-ipfs-http-client to gain access to the local IPFS node but if we want to use orbit-db or any other custom p2p component we don't have IPFS integration for it and we can't use it directly from the client app.

With so many great projects being developed in the IPFS ecosystem what we are missing is interoperability and deeper integration between them.

State of the Art

This survey on the State of the Art does not have to be exhaustive but it should provide a good entry point to existing work on the topic. Later contributors can augment the survey via PR.

Current approaches within the PROJECT Ecosystem

Existing attempts and strategies to solve the problem within the context of the relevant PL project

A good example of existing services that are exposed: https://github.com/ipfs/go-ipfs-api https://github.com/ipfs/go-ipfs-http-client While both APIs do well at exposing existing functionality they are not aiming at opening up the stack for custom component integrations. The single binary architecture prevents that in the first place.

Current approaches within the broader research ecosystem

How do people try to solve this problem more generally?

There are various companies across the Protocol Labs Network that are working on creating application platforms(PaaS) on top of IPFS - such as Fission, Fleek and Pinata to name a few.

Known shortcomings of existing solutions

What are the limitations of current solutions within the project ecosystem and elsewhere?

The existing solutions are creating their own specs and build their own stacks on top of IPFS rather than being interoperable integrated parts of the ecosystem.

Solving this Open Problem

Estimated impact

How would we and/or the world benefit from solving this problem?

We would be able to cover new use cases such as - IPFS as a plug and play service system, 3rd party integrations, IPFS as a backend, IPFS as a platform.

Proposed solution definition

What defines a complete solution? What hard constraints should it obey? Are there additional "soft" constraints that a solution would ideally obey?

A completely decoupled system either as native multi-binaries or microservices on top of a local cluster based on CQRS pattern with event sourcing which will enable features such as - global app state, events replay, real-time auditing and history logs, 3rd party integrations, adding custom functionality as microservices on top of the core IPFS functionality. This will lead to many custom versions of the boilerplate kit which expose a different set of services covering various use cases. Ideally all existing domains should be extracted to separate binaries/microservices and implemented on top of an existing open source boilerplate kit such as: https://github.com/vardius/go-api-boilerplate

Supplementary Material

Existing conversations/threads

Link to prior discussions relevant to the problem

https://discuss.ipfs.io/t/proof-of-concept-interplanetary-service-system/10245 https://discuss.ipfs.io/t/a-preliminary-discussion-of-the-workswap-protocol/12320

Extra notes

anything else deemed relevant to the problem

Areas which are not covered in depth yet are: combining of open source licenses and the network layer abstraction in case the kit is based on a local cluster.

The idea is not related to the existing ipfs cluster because it's about turning IPFS itself in a cluster of its own microservices rather than a cluster of peers.

[ebryson2]

Hi Marin,

Thanks for this interesting proposal. It's actually outside the scope of what we would fund as a research grant, but it might fit really well for consideration through the IPFS devgrant program. I encourage you to follow the process here

Thanks, Liz Research Admin

[mar1n3r0]

Hi Liz,

Thanks for the advice. I have looked into the IPFS devgrant program but unfortunately the RFPs are on hold there. Indeed such a proposal is at very large scale and can only happen as part of Protocol Labs strategic decision making. If it gets attention and reaches the stage of internal planning I would be more than happy to get involved.

[codi0]

@mar1n3r0 @ebryson2

I wonder if there is an interim solution that could be implemented into the core libs with relatively low effort, to see if / how people would begin to extend the core?

I think that it's an excellent, and ultimately necessary thing, to offer a pluggable architecture (to really fulfil the potential IFPS has). The first thing that went through my head was whether key events could be identified and something akin to the WordPress plugin/hooks system implemented in the js and go libs. I imagine that might be a relatively easy way to extend core functionality to prove whether a larger investment in architectural changes might be worth it further down the line.

[mar1n3r0]

Hey @codi-si I am currently extending go-ipfs on my own. In my case what I need is interoperability rather than extending the core itself.

For example I need to be able to initialize and work with orbit-db from within the ipfs daemon which is based on ipfs but a separate project. There are several APIs where integrations can be plugged in. In my case https://github.com/ipfs/go-ipfs-api was the most convenient API to extend.

Will post a link to the fork in the thread when ready as I am still testing it locally.

[codi0]

Hey @codi-si I am currently extending go-ipfs on my own. In my case what I need is interoperability rather than extending the core itself.

For example I need to be able to initialize and work with orbit-db from within the ipfs daemon which is based on ipfs but a separate project. There are several APIs where integrations can be plugged in. In my case https://github.com/ipfs/go-ipfs-api was the most convenient API to extend.

Will post a link to the fork in the thread when ready as I am still testing it locally.

Ah, sorry if I mis-understood the intent. From browsing various discussions on ipfs, one thing I noticed that keeps cropping up are ideological debates around what the scope of ipfs should or shouldn't be. All valid debates, but it struck me that there doesn't currently appear to be a good way to let people prove their hypothesis by building "a plugin" that people could install on a default node implementation to extend its functionality in some way (hence why I used WordPress as the example, since they nailed it on that front).

You're left with having to fork the core libs, which is fine for personal projects, but less so where the intent is to launch something on top of a default node that requires the involvement of many peers. I think ipfs is the base for the future of the decentralised web, but I worry innovation may be stifled if the only options are to convince the core team to change an API or create a fork.

My final observation is there's a whole ecosystem of ipfs-based stuff (whether it be encryption layers, storage options, computing options, developer tools etc) that would really benefit from a node plugin architecture. No need to debate whether these things should go into the core protocols (I think keeping them as lean as possible is absolutely right), because it would be easy to provide them on top of the core.

[mar1n3r0]

I felt the same way a couple of months ago since IPFS is a great option to replace cloud providers for the development and hosting of p2p dapps. Hence why I created this thread. A couple of observations along the way:

• go-ipfs, the original project was started about 10 years ago with the intent to be a native CLI app. Thus architecture is pretty much monolithic and even core team efforts to decouple it are quite cumbersome
• My idea was to refactor the whole app from a CLI app to a local cluster so you can split it into plug and play micro services and also have a finer control of resources
• Even if it happens in the future it will be a separate project because it requires a very significant amount of time and manpower to port 10 years of development
• It can't happen without the willingness and full cooperation of Protocol Labs
• So far I feel that our use case is not in line with the direction where IPFS is heading itself if you look at the other projects under the umbrella
• Forking is still the best way forward so that we can have autonomy and variety rather than top down central decision making, after all this is at the forefront of the open source philosophy

You're left with having to fork the core libs, which is fine for personal projects, but less so where the intent is to launch something on top of a default node that requires the involvement of many peers.

Yep, that's a valid point but it's not much different to ask the user to install your forked version of the ipfs daemon rather than a plugin.

[codi0]

All reasonable points. I certainly don't want to diminish the role that forking plays, since as you say it is a fantastic tool for autonomy and variety. Central decision making always slows things down.

The trouble with relying purely on forking is you can't easily compose forks. If you convince me to install your fork, I'm potentially tied to that implementation and if another fork comes along, I either have to merge the forks or just choose one. I think there's a critical-adoption point where pluggable becomes necessary.

Having said that, maybe a fork is needed to prototype a pluggable (node) architecture, given the points you've raised above. As yet I can't see a killer point for why it wouldn't be a boon for the ecosystem in the long run. Does seem like the perfect thing to give a grant for.

[mar1n3r0]

If you convince me to install your fork, I'm potentially tied to that implementation and if another fork comes along, I either have to merge the forks or just choose one. I think there's a critical-adoption point where pluggable becomes necessary.

Unfortunately yeah, forks are a workaround not a solution. They can't solve the monolithic problem.

As yet I can't see a killer point for why it wouldn't be a boon for the ecosystem in the long run. Does seem like the perfect thing to give a grant for.

Totally, I still feel the same way and would be happy to be part of such an effort. Over time I realized though that PL has put years of work without monetizing it and now the whole project becomes profitable through Filecoin and dependent on the crypto market. Thus the whole focus moves to what's keeping the project sustainable. Otherwise the use case we are talking about is as big if not bigger than that since it can eventually disrupt the 300B+ cloud market. Maybe they don't want to enter this competition and prefer to focus on file storage as a lower competition market.

In any case you are right that interoperability of the whole ecosystem is key to success in the long-term.

[codi0]

This whole conversation prompted to research out to PL by email just to see whether they'd consider a grant for something like this, and if so what RFP it might best sit in.

I think this is the future, if we're talking about changing the internet for the better, so I'm very interested in helping out on exploring this.

[mar1n3r0]

https://github.com/ipfs/devgrants/pull/139

Hope they reconsider in the future. It is definitely not a one man show job and would require a quite a lot of dedicated core dev engineers to work with external developers. Bear in mind also that what IPFS does is basically post-capitalism since it democratizes 300B+ cloud services converting them to free p2p hosting. As long as we have the concept of money cutting edge innovation like this will always be hindered by the need for profitability and of course the opposition of the established monopoly.

[codi0]

I wouldn't take what they said there as a no, more just "we need more info in order to consider it". The thing we might need their help with is reaching enough of the dev community to ask the right questions to, as evidence it's worth doing.

There's a precursor of defining exactly what the scope of "pluggable" would mean, but I would like to understand from potential developers in the ipfs community whether they would (a) be interested in building plugins on top of the default/core implementation and (b) what kind of functionalities they might want to add.

Honestly, these are fundamental questions that I think the core team should have an interest in gauging community sentiment on.

[mar1n3r0]

Even being able to use what's already available in the ipfs ecosystem together would be great. When it comes to the js-ipfs implementation they have this modular design already. Consider the following example:

https://github.com/orbitdb/orbit-db#module-with-ipfs-instance

If you are using a js frontend that's already a plug and play p2p db integration. So JS devs already have this to a great degree.

My use case was based on using a WebAssembly frontend and wasm doesn't have access to host resources. Thus I need to go through the ipfs shell to create an orbit db instance. Since we are very few developing in WebAssembly and 99.9% are developing on JS they have no interest in such a use case.

That leads to a bigger problem. Web frontend is completely monopolized by a single language. Now that browsers also support wasm natively(most languages can compile to it) the frontend becomes democratized but it will take a lot of time if it ever happens to see frontend being developed in anything else than JS. Companies and devs love the monopoly - it saves time, money and creates an abundance of developers.

[codi0]

I see what you mean. Different libs have different levels of modularity right now. I still think the js implementation lacks "pluggability" from the perspective of being able to add new elements of functionality without the core having to be changed (which was where my thinking began from). But yes, the go implementation seems less modular.

I guess my thinking was that if a set of common pluggable events (for want of a better term) were identified that would benefit the majority of dev use cases, then they could be implemented in both js and go implementations. Effectively standardising points of integration/extensibility.

As an aside, whilst I haven't yet played with WASM too much myself, I do believe that ultimately "universal compilation to wasm" is likely to be a core tenant of apps on the decentralised web. Aim one day is to create a web3 dev platform, that utilises those concepts (codi.io/web3).

[mar1n3r0]

Yeah, I got your point when you compared it with Wordpress. A hooks system based on events would be nice but doesn't decentralize the app. It's still a monolith run from a single binary rather than a composition of microservices orchestrated in a cluster.

As an aside, whilst I haven't yet played with WASM too much myself, I do believe that ultimately "universal compilation to wasm" is likely to be a core tenant of apps on the decentralised web. Aim one day is to create a web3 dev platform, that utilises those concepts (codi.io/web3).

Love the summary, very well put. I am currently developing an open-source post-capitalism economy simulator/game based on ipfs and WASM. It's like a MMORPG. Should be ready by the end of the month and I will release the forks and the game publicly.

In my opinion there are good things about web3 and there are bad things. Bad things are heavily funded and will eventually dominate and define the term - read total mass control, censorship and the concept of privacy gone.

It will be the same pyramid of hierarchy as it is today just tagged "decentralized" as we can see with crypto.

[codi0]

Yeah, I got your point when you compared it with Wordpress. A hooks system based on events would be nice but doesn't decentralize the app. It's still a monolith run from a single binary rather than a composition of microservices orchestrated in a cluster.

You're right. My suggestion is probably complimentary to what you laid out, but doesn't massively help your use case. In an ideal world, fully modular libs for both go and js + hooks for extending it would ensure ipfs innovation takes off. The only thing I can think is that hooks might be a much easier first step, and allow a path towards refactoring the go lib more fully (since you wouldn't get full interoperability and extensibility without doing so).

In my opinion there are good things about web3 and there are bad things.

I'll simply say my view is that web3 will grow to be the over-arching label for "the decentralised web" and that crypto will be a much smaller part of it (if that doesn't play out, I'll go with another term like "dWeb").

[mar1n3r0]

My suggestion is probably complimentary to what you laid out, but doesn't massively help your use case. In an ideal world, fully modular libs for both go and js + hooks for extending it would ensure ipfs innovation takes off.

As said previously 99.9% of developers don't use go-ipfs+wasm on the frontend so they don't have the same requirements as the users of js-ipfs. Thus go-ipfs currently suits the needs of those who use it. For the foreseeable future we are a niche minority and that's why forks will be the only way in that direction.