Sanity check: go-app as a generic decentralized app framework

[stevegt]

Hi All and Maxence,

I'm thinking of using go-app as a layer in a decentralized app. By "decentralized app" I mean browser-side persistence, lateral communications between browsers, and so on -- more details below. I see that @oderwat and @mlctrez are apparently already playing with similar ideas: #789.

Is this a reasonable use of go-app, given current capabilities and roadmap? As far as I can tell so far, the answer to this is "yes". But I have an entire group of people who will be following along and working with me on this effort, and so am checking in here first before we commit several weeks to a POC.

Background

I host a working group developing a collaboration platform for other groups and teams. This effort is a spinout from the Nation of Makers community, though not exclusive to nor an official effort of that organization; standard disclaimers apply. So far it's just me and a handful of other folks having weekly video calls to think this through.

The platform is to be open-source, written in Go, though as I mention in a reddit post, I'm not finding many existing examples of Web3-style apps in Go -- most apps in this space appear to still be javascript, with a little Rust. My guess is that Go/WASM is still too new; the payload size may be a contributing factor.

I believe go-app might be a good fit for filling this gap.

Proposal

If we were to use go-app, we would work with @maxence-charriere and everyone else here to discover and work out any issues that might be lurking with using go-app as a generic-enough framework for decentralized apps, including support of the wishlist items below. Go-app already supports most of this, or at least doesn't get in the way.

Wishlist

By "Web3-style" I don't mean necessarily defi-related. I do mean the following items, adapted from another reddit post:

• decentralized to the extent that the app can be served from e.g. github pages and run client-side in WASM, likely using IndexedDB for persistence.
• able to communicate with other nodes through multiple protocols e.g. websocket, webrtc, nats, etc.
• able to run the same code "headless" in server-side nodes to provide additional persistence and relays between firewalled clients
• SEO-friendly enough that googlers can find content rendered on server nodes
• interoperable with a few javascript libraries (e.g. the tiptap editor)
  • likely a micro-frontend architecture style, with comms via the DOM
• be future-proofed enough that we don't need to deal with breaking changes in the framework
  • go-app does appear to be using semver -- do I have that right? Have there been any hiccups with breaking changes at non-major release points?

Thanks All,

Steve

[oderwat]

Hi,

Just some quick thoughts of mine:

1. I think it is perfectly fine to run go-app from static pages and use indexedDB (I do it myself).
2. Hi, I doubt that you can run a PWA that connects to other PWAs without a central hub of some sort. You will need a server. We started to use NATS now. So technically, you can always only communicate with the server and that can proxy this to other nodes. Like every other web application/site. But I guess that you know this. Still, you write about "decentralized apps". I fear that is not to be found here. But maybe you meant running multiple nodes with a Go-App based frontend?
3. I do run some frontend code "headless" but then it is just a part of another Go compiled executable. Go-App is not suited (or the right tool) to write WASM code outside what gets used in the browser. However, I do not understand your intent. What has anything of that to do with firewalls? You can't run a server / any kind of TCP even only outgoing in a browser and also not really (easily) in WASM. It is a sandbox. You can have outgoing web sockets (in a browser). We get more with WASI in other environments, but that is in development and still needs the environment to make the connections. But I may have not understood what you mean :)
4. Go-App lets you render whatever you want for any visited URL at the server. It is not really bound to the front-end code and paths. By default, it renders data for the defined routes in the frontend. We usually do not use that at all because we create Apps and no websites.
5. You can just interface to any JavaScript library. There remains the question of what the PWA is for in that case?
6. I can only speak for me and the last years: I am always ready to fork and fix whatever needs to be done when using go-app. I think it changed quite a lot between version. It is at version 9, I think that speaks for itself.

I may have understood your endeavor and wishlist wrong, though. So take it with a grain of salt. :)

[prologic]

I just want to say a couple of things (not sure what your "app" is intending to solve here...)

• Web3, Web4, Web5 are all total non-sense 😅
• Let's be really clear on the terms "decentralised", "distributed" and so on. What you are describing is mostly a client that may connect to a "distributed network" of nodes.

Good luck in your project! 🤗

[maxence-charriere]

Hello @stevegt. Glad you are considering go-app for building things. Gonna try to answer the best I can to those points:

decentralized to the extent that the app can be served from e.g. github pages and run client-side in WASM, likely using IndexedDB for persistence.

go-app based apps can be served on GitHub pages. https://go-app.dev and https://lofimusic.app are both running on GitHub pages. => https://go-app.dev/github-deploy

Concerning IndexedDB, there are currently no plans to provide an API within go-app. You will have to implement a bridge from the go-app Javascript Interop API that follows the standard syscall/js package.

able to communicate with other nodes through multiple protocols e.g. websocket, webrtc, nats, etc.

This should be possible, never tested it personally.

able to run the same code "headless" in server-side nodes to provide additional persistence and relays between firewalled clients

go-app based apps are WebAssembly binary that run on the browser. The server part only provides the resources for a browser to make it work. It also generates static pre-rendered HTML files for SEO purposes. See it like an app installed on a computer or a phone. You are free to build whatever API you need to be consumed by a go-app based app.

SEO-friendly enough that googlers can find content rendered on server nodes

As mentioned before, you can generate pre-rendered HTML files for SEO purpose. => https://go-app.dev/seo https://murlok.io is using that mechanism.

interoperable with a few javascript libraries (e.g. the tiptap editor) likely a micro-frontend architecture style, with comms via the DOM

go-app is interoperable with javascript. It does not play well with javascript libraries that modify the DOM though.

be future-proofed enough that we don't need to deal with breaking changes in the framework go-app does appear to be using semver -- do I have that right? Have there been any hiccups with breaking changes at non-major release points?

go-app is in version 9. I have a lot of things that run on this version. I am happy with the API and I don't see anything that would make me want to bump to v10 for now. There are some things I'd like to change but it will not require a bump in version.

Hope this helps.

[stevegt]

@oderwat @prologic @maxence-charriere Thank you. I can see that there are a few things I didn't describe accurately, and a few others I likely have some misconceptions about. Either way, you've given me some things to think about and experiment with, so I'm glad I asked.

This may be the most crucial piece:

go-app is interoperable with javascript. It does not play well with javascript libraries that modify the DOM though.

That's important to know, and I'm going to have to do some experimenting. Other than the tiptap editor, I'm pretty sure I'll need some other pieces of functionality that only exist in javascript or Rust/WASM and that may modify parts of the DOM.

go-app based apps can be served on GitHub pages. https://go-app.dev and https://lofimusic.app are both running on GitHub pages. => https://go-app.dev/github-deploy

Hi Maxence! Those examples are a large part of why I'm looking at go-app in the first place. (And I have lofimusic.app running in the background right now.) :musical_note: Even if the DOM modification issue turns out to be a problem in my case, I've spent enough time digging around on github looking for prior art in Go that I can say with some certainty that you've done some pretty unique and amazing things here -- I haven't been able to find anything else that comes close to what I'm looking for.

---

Below are followup answers to the other questions you each raised. I'll close this issue if there are no additional comments in the next several days. Meanwhile I've started a tentative repo for the project and will integrate all of this info over there: https://github.com/stevegt/incode

I doubt that you can run a PWA that connects to other PWAs without a central hub of some sort. You will need a server. We started to use NATS now.

NATS is good, but another of my groups ran into some architectural issues on a related project a year or so ago -- might be worth me taking another look. More about this below.

Another possibility that's been raised is webrtc with TURN servers for relaying.

There's a more generic decentralized computation protocol I've been working on as well, though I don't feel competent to describe it in English yet.

I do run some frontend code "headless" but then it is just a part of another Go compiled executable. Go-App is not suited (or the right tool) to write WASM code outside what gets used in the browser. However, I do not understand your intent. What has anything of that to do with firewalls? You can't run a server / any kind of TCP even only outgoing in a browser and also not really (easily) in WASM. It is a sandbox. You can have outgoing web sockets (in a browser). We get more with WASI in other environments, but that is in development and still needs the environment to make the connections. But I may have not understood what you mean :)

WASI is one of the ideas we've talked about for server-side. Likely simpler to just keep core code in a library that can be compiled into a WASM module for client side or native binary for server. The main loop would be different on the server, and of course the server would have to be listening for e.g. websocket connections.

Instead of "firewalls" I should have said "NAT".

I think a source of confusion is whether server nodes nodes are serving websocket, HTTP assets, or both, and whether a server node is stateless, or if it acts as a persistence cache for the data it's relaying between browsers. For deployment simplicity, I'm thinking of a single server binary that is able to perform any or all of these functions depending on flags.

My recollection is that the NATS architectural issue we ran into a year or so ago was that there was not a way to import the NATS server code as a library such that a NATS server daemon could also also act as an agent in the conversations it was relaying. As I recall, it was missing a callback hook that could intercept messages and pass them to a local goroutine. I could be misremembering, and the code may have changed since then. I'm also not recalling why we didn't fork it and send them a PR to add the feature.

Web3, Web4, Web5 are all total non-sense

I'm bothered as well by the association between Web3 and the hype around the community that spawned it, but I don't know any other shorthand for "a decentralized app that runs in browsers and needs no servers other than for relaying data between NAT'd clients and possibly for redundant persistence in case all of the browsers close and of course you need static servers for downloading the code in the first place". :facepalm:

Maybe "p2p PWA" might work.

• Let's be really clear on the terms "decentralised", "distributed" and so on. What you are describing is mostly a client that may connect to a "distributed network" of nodes.

In the case of a multiuser collaboration app, the internode connections aren't optional because they are the entire point of the app. I'm getting the sense that this might affect some of the architectural decisions in any underlying framework.

Thanks again everyone,

Steve

[gedw99]

Regarding IndexedDB, https://github.com/hack-pad/hackpadfs is a possibility.

Its all golang and compiles to wasm.

Your go-app running as a Service worker, can access the data in the indexedDB.

[oderwat]

Regarding IndexedDB, https://github.com/hack-pad/hackpadfs is a possibility.

I use IndexedDB quite a lot from inside go-app, but I use a nice JS-based KV abstraction to make it less cumbersome. That file system looks like some interesting option. I just wonder for what exactly :)

Your go-app running as a Service worker, can access the data in the indexedDB.

That is not 100% correct. Your go-app code is not running as service worker. It just has a service worker that proxies fetch for offline mode, checks for updates and can handle push notifications. Your code will also run "for every tab" but the service worker just runs once per "subdomain". Nevertheless, the service worker can also use IndexedDB, that is true.

[gedw99]

Thanks for the clarification .

Here is what I am trying to use it for.

HackFSis a KV ( backed by indexeddb ) that can represent a file system using a KV structure. That’s what it seems to do if when I look at the raw indexdb in chrome debugger.

The “ per tab “ aspect is interesting .

When NATS ( or Web Sockets ) send a message to the go-app, I want it to update some data managed in the IndexDB, and update the GUI.

A simple example is a file system that is represented in HackFS. You want all tabs that have a view that represents the file system to update . Sort of like Google Drive .

I also only want the tabs with a view of the file system that was changed by the original NATS message to update.

I don’t know yet is if all the users tabs ( for the respective app ) will update in the way described above.

[oderwat]

Per tab means you are running the app per tab, the IndexedDB will be the same for all tabs (on the same subdomain), which may cause problems by itself. In addition, there are multiple ways how IndexedDB can go wrong. I use it like this: "It may be there and all is good, but it may vanish, and then I need good defaults".

Larger PWAs we developed in the past store all data as more or less a single value at a point. Changed data will usually send to the server asap. We only snapshot the data into the IndexedDB and never do lookups or something like that.

You cannot rely on "having data in the IndexedDB" beside the fact that the user can (and probably will) clear it at anytime, there are also numerous of bugs or inconsistencies I read about. We faced that data which was there suddenly vanished some minutes later without anybody touching the device.

So, I would be very cautious to use IndexedDB. I use it only for storing preferences I can't put in a cookie. That is state 2021 and having Safari IOS as one of the main systems. Stuff worked better in Chrome Android at that time. You will find easily some horror stories about Safari and IndexedDB.

We sync the data to a server and avoid having "much" data in a PWA or SPA context. This is not always possible, I have a PWA that has some megabytes of data eventually. We try to make this save by communicating updates to the data like a "diff", so we do not snapshot all the data, all the time.

It is also a bit problematic to store large amounts of data inside the browser of the user. He may open up another browser on the same system and expect to find the data too :)

One reason why we want to use NATS much more is, that we also use it's Jetstream, KV, Storage systems to store and replicate data. I, personally, would not want to be responsible for any application that has the data in IndexedDB one click and everything is gone. But it is true, that it is hard to tell somebody how to do these clicking when you need them to clean the data for debugging. In that case, you will never get rid of that, of course.

I think one alternative to store data maybe using the fetch interface and the browser cache. We do that for some data so that it is available and does not get reloaded until it gets changed. It will also vanish with the browser clearing site data, but the difference for me is, that I "think of that as normal". A cache is a cache.

Maybe all of this is "the past" and IndexedDB is more stable "through all implementations" or people use less esoteric stuff. The problem is: Usually you can't tell the client that their users need to use this or that browser or system. Especially with Mobile Apps.

P.S.: If you allow multiple tabs and have just one source of truth, you do need to update them all at once, in my opinion. But there are many SPAs, and probably nearly all PWAs out there, that just don't allow multiple tabs (or windows). With the PWAs that is a given because you can only install one at the same time (normally).

[gedw99]

i forgot how indexed is totally a thing that the OS browser will vacuum away as it pleases.

its best to use both IndexedDB and cache as a "cache" only. https://web.dev/storage-for-the-web/

[prologic]

Btw... This is really triggering me and bother me a lot... Can we change the title of this issue? The whole "Web" and "decentralised" parts are just misleading as hell 😢 Sorry 🙏

[oderwat]

@prologic Or we add: Blockchain, NFT and cryptocurrency to round it up? I guess, we also could add Metaverse and NATS, oh wait, NATS is real 😜

[stevegt]

Btw... This is really triggering me and bother me a lot... Can we change the title of this issue? The whole "Web" and "decentralised" parts are just misleading as hell cry Sorry pray

Done! Bothering me too, wish there was a better term.

[prologic]

@stevegt Thank you 🙏

[prologic]

And btw, I'm very keen to see what you end up creating, I love what we can do now with Web Assembly, and go-app is amazing 🥳