WIP: Maki Stack

[martindale]

This imports all the work I've added on top of the converse base. There's another WIP proposal (martindale/converse#24) that pulls the changes into converse itself, but that's largely for archival purposes. This pull request implements the API and messaging infrastructure for a single local node, to later be joined to the p2p network as described by @ryanxcharles.

How this works, generally: Models that are specific to the DATT network are defined as Maki "Resources", which have a static set of methods (query, update, destroy, etc.), hooks (pre:create, etc.), and events (vote and gilding, for example). Here, we define the loose behaviors that are necessary for a single node to validate individual operations (or "transactions") that mutate the state of the DATT database for a single node.

Most of the core logic is in index.js. While monolithic and somewhat daunting at ~400 lines, this can easily be broken out and modularized – I propose a simple folder with one-resource-per-file, perhaps in a location like ./resources.

API documentation is also automatically exposed via /api. For an example, take a look at the currently deployed converse API.

[ryanxcharles]

Awesome work. Reviewing. First thought is that there is a super huge amount of commits here ... that may be fine for this PR so we can keep the commit history, but normally that wouldn't be the case.

[martindale]

Yes, quite a few commits – I started the original project over a year ago, and this simply extends that work. I'm happy to squash everything into several logical commits if that makes more sense from a historical perspective.

[ryanxcharles]

First of all, I think we should consider pulling this in since it would be a way to demo what we want the thing to be. It would be nice to have an app that we can all work on. And getting to a demo quickly will help validate the idea. However I have criticisms:

• Although there is some basic functionality here, it isn't actually decentralized. It's a normal web app with a server and a client (no?). That's good for a demo so we can show people what the thing is supposed to look like, but it's hard to see how we can take this and "make it decentralized". I'm worried a bunch of the functionality here would actually hinder the main goal, because it would be really hard to translate to a decentralized platform.
• You overwrote some of the work I did, which although it was very minimalist and incomplete, had one very important property: the same app ran node and in a browser. Since the bulk of our code needs to run in both places, I think we need to be careful to put all the logic of the app in a single app (presumably a bunch of files for node that compile to a single bundle for the browser). It doesn't seem that's how this app works.
• My work also had full test coverage and this has none. The tests also ran both in node and a browser. Even if we don't necessarily desire 100% test coverage for a prototype, it's very useful to at least be able to write tests that run both in node and a browser. IMO, writing tests and code at the same time is not a burden at all, and is most helpful when refactoring things so that you know what you're breaking.
• Looks like it uses mongo, which is not the datastore I would use, because we don't need 99% of the features of mongo, and if anybody uses them, it will just break the p2p protocol. There are only a few types of things that need to be stored, particularly comments and actions, both of which can be represented in a key/value store without too much effort. Note also that reddit in particular migrated from having a complex database to having something resembling a key/value store in order to scale.
• Still not convinced about Maki. As usual, not saying there's anything wrong with it, just saying I don't really understand it. Maki sounds like it would work well with an app that has a huge and growing REST API, like most web services. I don't think our app is going to be anything like that. Once the protocol is set, it won't change much. Also, I'm concerned about using it simply because I don't understand it - it's really hard to judge its appropriateness without grokking the principles.
• Your dependency on maki, which links to your github, should reference a specific commit, so that if maki changes, this app doesn't change unless we explicitly change it. This is both for security, in case the github maki repo is compromised (unlikely), and for ensuring that changes to maki don't break the app.
• I think most of the commits should be squashed. If the commits gave detailed explanations of the code then they would be useful, but they are mostly one-liners. I would try to find a few important "break points" and squish the commits down to a handful. (I believe "git rebase -i" is your friend here.)
• I would never use jQuery :) We should use the ember platform that @go1dfish created. That may not be necessary for this PR though as we can add it later.
• My work was done in ES 2015 and this is all clearly ES 5. That's not necessarily an issue for a prototype. However there are some callbacks in here, which I'm not a fan of. Promises are 10x better, and promises + generators are 100x better.
• I'm unsure how difficult it's going to be to integrate some of the unusual features we need to build that are custom for this app. I know I already asked, but ... how does this mesh with a p2p protocol? How does this mesh with using IndexedDB for storage in a browser? How does it mesh with web RTC?

So my basic suggestions are:

• Somehow salvage the work I did. It shouldn't be deleted. Though I'm not sure how to fit it into this. Again, my basic idea was to have a single app that ran in both node and a browser. That will save us headaches. Not sure why we would want to do it a different way.
• Add a test framework (mocha) that works both in node and the browser. My code already did this so that shouldn't be hard to keep. However, it may not be trivial to integrate into your stuff. Add at least some tests - not having any test suite is the easiest way to not write any tests at all, which is going to become unmaintainable in a hurry with different developers working on it.
• Squash the commits down to a few that matter. If you want to keep iterating, you can save the squashing until things are done.
• If it's not too fundamental, I would get rid of mongo and use leveldb. IMO using mongo at all is a recipe to create something that doesn't fit our product description. We can't store arbitrary schemaless JSON data, because that will look nothing like our p2p protocol. We need to store very specific items (comments and actions) which are easily represented as key/values, and nothing else.

I'm worried this PR may go too far in the direction of "make something that looks like it works so we can show people", but that doesn't actually solve the problem we're trying to solve, and will make it harder to solve the problem, because it introduces a bunch of patterns that may be inappropriate once we actually solve them.

[martindale]

Good feedback. As discussed on Slack, I do already plan on addressing several of these items – not the least of which is isomorphism, which also depends on writing a new datastore for Maki that replaces the MongoDB dependency. This is already planned (see martindale/maki#58), just will be accelerated for inclusion here.

Other relevant issues that address your points include martindale/maki#21 (P2P requirement), martindale/maki#77 (View Compositing), and martindale/maki#80 (Offline Sync, or network independence).

I'll have more updates come daybreak. Thanks for the extensive review!

[ryanxcharles]

Eric, I don't believe Maki or Converse are appropriate technologies for datt. It doesn't seem to me that Maki solves any of our problems. Converse looks like a centralized bulletin board. I don't believe we are better off trying to leverage this technology. I think it will make things harder.

When I worked at reddit, I collaborated with Chain to add a bitcoin tipping feature into reddit. Since reddit.com is open-source, this was easily done. It took one day. If we wanted to start with an existing reddit-like repository, reddit itself is an option. However, we did that only because I worked at reddit. Trying to take that project and make it decentralized means we start with a huge amount of technical debt. None of it was designed with decentralization in mind. It is harder to convert a centralized application and make it decentralized because everything has to be rethought.

I believe the way forward is to build the p2p protocol I outlined along with a database into an app that works both in node and a browser. Browser-side, we hook this app up to the datt ember UI along with the datt design work. I don't see any way Maki or Converse helps with that.

[martindale]

On the contrary, Maki's been built from the ground up with decentralization in mind – take a look at the fabric branch to see how peer-to-peer messaging works between nodes (tracked since last October in martindale/maki#21). It should to be much easier to leverage the infrastructure I've established for all the other Maki-powered apps, and I think we'll get it done faster.

I mentioned yesterday that I now have the peer messaging working in the browser; I'll be pushing an update to this pull request once I've consumed your proposal for the protocol, and can include a working demo of multiple nodes communicating. I've also begun migrating the datastore class towards an isomorphic store, but we shouldn't wait on that change – in the "make it work, make it right, make it fast" mantra, we're still in the "make it work" part.

I don't have any particular affinity to Maki over building something from scratch here, but I am fairly convinced that we're incredibly close to a working prototype already and think we should move forward with what we have.

[ryanxcharles]

We've settled on an architecture based on what was datt-node and dat-react.