Application state and architecture: are we alone?

[xbill82]

Hi Dan, the front-end web development community has been carrying on a deep reflection concerning the architecture of applications. Facebook had a great impact on that by introducing React and Flux. Then Relay and Redux came and then Cycle.js. These are only a few examples of projects pushing things a step further, like every six months, which gives a sense of recurrent hype-race (which, imho, it is not).

So, are we (the JS-related community) the only ones asking those questions? Do the iOS / visual studio / Java spring guys ask those questions as well? If yes, are we communicating with them? If not, why? Do they have answers? Or maybe they don't have the same problems we have? Or they just gave up maintaining an application state consistently in a scalable way? Did they give up responding to user interactions without producing spaghetti code? I'd hardly believe so, but I really didn't find relevant content on the internet.

It resumes to, are we alone? If not, are we joining forces?

Thanks in advance.

[gaearon]

Great question, and I don't know enough to answer it. (I hope somebody will help.)

There's definitely innovation coming from native platforms. Come to think of it, Rx (that Angular 2 and Cycle use heavily) was part of .NET originally, and gained traction in the native community first (RxJava, Rx .NET). There also things like ReactiveCocoa (iOS).

I haven't seen a lot of native frameworks for state management, but my guess is that it's because mobile platforms usually show a single screen of UI which is a really nice limitation because you don't have to keep 10 related parts in sync. So MVC does its job well there.

When UI gets complicated, of course it becomes a mess. But at least it's mobile screen-sized mess, and it is often created with stock controls that usually have rich APIs so you can usually delegate a lot of UI handling to them. Not true for DOM (where's my popover?)

A big impediment to powerful state management is a lack of powerful UI engine that lets you create declarative views. You can't use something like Flux, Redux, or Cycle, if you render the views imperatively. The imperative OOP UI is something Apple and Google have been working on for tens of years, so naturally their frameworks aren't the best fit for declarative paradigm. That's where React Native shines. As native frameworks begin to adopt its approach, I'm sure that we'll see even more innovation in native UI architectural space.

[koresar]

Native platforms have never moved further than MV* patterns in ~40 years. It's all about language limitations and old habits aka "best" practices. It was ultimately hard to experiment with UI in C++ for instance. ES5 gave UI a quick boost because freedom, and because absence of type checks and full control of the screen real estate. Freedom is the corner stone of innovation.

[ghost]

Hi. I'm a big fan of your work on Redux and Hot-Reload, they have inspired me to think much wider in terms of architecture and programming. Sorry if i'm nudging, just wanted to leave my 2cents about this...

There actually has not been that much innovation (by what i have researched), only a lot of "reinvention" throughout the last decade. Taking into account the history of computer technology the web is a huge embarrassment to the field, having major problems fixed only in the last few years (security, performance, programming capabilities). The frameworks/libraries being released lately are only the application of concepts and ideas that were developed in the last century and ignored by ignorance from the majority since the 80's.

The functional ideas of simplicity, immutability, single way data flow et al. have been in development for more than a century together with the other models of computation, it's a great time as we are starting to bring these ideas and concepts to mainstream web/mobile development. (Thanks Dan again for great work on Redux)

An example of how the web is a delay in app development innovation can be seen comparing the state of development today with the HyperCard technology developed at Apple in the 80's. This video (Hypercard (1987)) shows the usage of HyperCard. If you research on how HyperCard works you'll find that it contains similar elements to the document model of the web. It has a notion of events and event handlers that initiate a process, related data and an attempt to incorporate natural language through a script :eyes:

IMO the enterprise and foolish competitiveness by companies in the web space during the initial years of the web created a huge ""cancer"" that is only being relieved now. Imagine if Netscape, Microsoft and others worked together in the 90's like they do now, imagine where we would be today in web development.... It's not that there is a tremendous innovation happening, it's that we are finally fixing the problems created by selfish competitiveness from companies in the early 2000's

"Why don't Java, VS, e.t.c people don't do this?", well, why are there still people doing OO ? I learnt programming by OO, i started in C++, then C#.... today anywhere i see private something i cringe, :smile: I find OO an enormous waste of time and have embraced functional concepts (specially lambda calculus concepts) gratefully. But go tell a hardcore OO person that their paradigm is a waste of time, go tell a religious person are prayers is a waste of time, they might get a bit... pissed off...

To answer your question @xbill82 -> Douglas Crockford The Better Parts:

And the reason these things take a generation is because ultimately we do not change people's minds. We have to wait for the previous generation to retire or die before we can get critical mass on the next idea. So it's like we look around: "Are they gone?

Uncle Bob also describes this behavior through his interviews and talks.

[slorber]

I'm not sure if you want to discuss about UI or backend technologies. The term "native" seems obscure to me: is a Java REST API the same as a Java transient Swing application?

About UI technologies, React has shawn that its concepts can be ported to any UI. So the Java guys that are still stuck with Swing could build something like React-swing, and use Flux to manage that UI. Same for iOS (but there's ComponentKit).

About backend UI technologies, the initial problem to solve is different: you want to serve HTML based on a snapshot of the DB state at the request time. The state does not change over time: once transmitted over the wire you can not update it anymore. We have declarative UIs in the backend, it is all these templating frameworks, and some of them make it clear that the template is a function taking state and returning HTML. The main advantage of React over these templates is that React is designed to solve the problem of state that changes over time. You can notice that in React server-side renderings, you render once and serve the response to the client: it's always a one shot.

About non-UI backend technologies, there's a lot to learn from. The concepts behind Flux are not new at all and you can find these concepts at work in distributed systems and how database synchronize themselves in clusters. See these links from the authors of Kafka: http://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying So the concept of firing actions, so-called "events", and recording them for undo/redo/replay/time-travel is nothing new and is the basic principle behind database replication, database point-in-time backups distributed systems synchronisation... If you use MongoDB, think oplog. All modern DB's have something like that.

For simplicity, I'll call the ordered list of events the "event-log", and the technique to record them "event-sourcing" (that term is not used in all communities, like the guys from Kafka that use "stream processing")

So as you have seen, the databases themselves use event-sourcing. But some people also choose to say: "hey, let's consider that instead of letting the DB do the event-sourcing, we will do it ourselves one layer above: at the backend app layer". The cool thing about that, is that now your database is no more MySQL or MongoDB, it is your event log: your unique source of truth. You will only use your traditionnal DB like a query engine, like an index that solves a particular problem. If you want, you can trash your DB/index, and then take all your events, and project them again in your index: you will get the exact same DB as you had before trashing. It is exactly like replaying your redux action log on a reducer: the result is always the same.

So you can build any DB from your event-log. This is nice, but it also lets you easily change the projection logic, so that you can easily migrate the shape of your query indexes without having to deal with very complex DB migration: just change the logic (aka reducer) and replay the events: this is exactly like hot module replacement for Redux stores.

You can also decide that instead of projecting your event-log into MongoDB only, you will also project it into another index like ElasticSearch, Cassandra, Redis, Memcache or Hadoop: think of these technologies as reduces: you can spawn as much as you want, and it is not forbidden they listen to the same events. This permits to scale easily an application and to design indexes that match close to the views or jsons we want to serve or (like you would design flux stores that match the UI too)

Now there's something more we do with the backend: we use "commands". Commands represent the intent of the user, events represent what just has happened (always in the past!). A command can be rejected, or accepted and generate events (and only then the DB/indexes will update: never before). In this case it's different from the UI because in an UI, somehow there's only events: you don't show buttons that do nothing at all, at least it should display some error message, which is still a UI state change... An API on the other end, is accessed asynchronously and thus the client may attempt to do an action that was available, but is not anymore (because the client had stale data), probably due to a concurrent modification of an item. So commands can be rejected.

So all this Command + Events (source of truth) + projections in DB (indexes) is often called "event-sourcing" + CQRS

The next part is Domain-Driven-Design. It splits the business logic into "bounded contexts", simplifies it and make it more explicit (I mean it's like at scool: you really do car.start() ) and provide a way to ensure that our data keeps being consistent over time. I won't explain more as I'm not an expert yet and it's not easy to grasp in the beginning :)

---

So the backend has already done a lot about state management, but it does not have the exact same problem than an UI:

• The UI is single threaded so you generally don't have that much concurrency problems. Now imagine what would happen if you delayed all the dom events you receive by a random amount of time: this is exactly what the backend has to face, and very often it is programmed very optimistically, allowing inconsistencies to happen (like "ooh now, this user edited the same doc and has overriden my edit!") (Domain-Driven-Design aims to solve this). You can notice how more difficult it is in JS to be predictable when you introduce ajax requests, timers and intervals.
• The backend run in clusters, and you can consume the same event-log from multiple devices, while in your UI all the load is for your single browser. I've tried to project the flux action log to many stores (than could end up hosting the same data) but it does not scale very well, because in this case you can't make the immutable state mutations on multiple computers.

Another thing to think about is how to integrate backend event-sourcing with frontend Flux (or something we could call "full-stack Flux")

I think things like Relay, Falcor are not so bad but do we really need Ajax requests anymore? Why can't we just stream events from and to the frontend? See some of my thoughts here: http://stackoverflow.com/a/26633455/82609

[slorber]

@thelambdaparty about OOP, the thing is that today many people use OOP in a way it was not designed to be used in the first time: we create a DTO objects and add some methods to mutate it, extract data from it... we add serialization annotations, ORM specific stuff to it... Finally is it still OOP?

I also love functional programming, and program in Scala all days. But I still think OOP can be a nice paradigm in very specific cases, when doing it the right way.

This is somehow what the Domain-Driven-Design teach you: you first design your domain in term of interactions. You will design like at school the main objects that can interact with each others, with all the complex business rules, and without any UI except the logs. This design should absolutly not depend on ANYTHING: no ORM, no technical framework, no persistence layer, no API stuff... Just the OOP language itself. Once you have your core domain model, then you use it as a library and integrates it with technical stuff. The idea is that your core domain should remain pure. If you decide to change your DB, ORM, web server or anything, you should never have to change it, unless your business has changed: this is not the case in all the backend softwares i've worked on.

I mean your core domain (ie AggregateRoot) should somehow look like that:

case class Car(carId: String, speed: Int) {

  def  accelerate: Event =  {
    require(speed < 6,"Can't accelerate more")
    return CarAccelerated(carId))
  }

  def  stop: Event =  {
    require(speed != 0,"Can't stop a stopped car")
    return CarStopped(carId))
  }

  def applyEvent(event: Event): Car = event match {
    case CarAccelerated(_) => copy(speed = speed+1)
    case CarStopped(_) => copy(speed = 0)
  }
}

In this example there's no depency at all on anything. And the technical code is plugged from the outside:

object CarCommandHandler {
  def handleCommand(cmd: AccelerateCar) = {
     val car = fetchCar(cmd.carId)
     val event = car.accelerate
     var carUpdated = car.applyEvent(applyEvent)
     persistCar(carUpdated)
     publishToKafka(event)
  }
}

[ghost]

@slorber Yes, very true, OOP is not used like it should, i did it wrong a lot before learning what it was truly about. Alan Kay rants on that too, lol. The code you sent reminded of "Good OO programs are good functional programs" (heard uncle Bob say that)

[xbill82]

Hey guys, thanks a lot for your awesome contributions to this topic. I'm really impressed by the height of the stakes here. Focusing on one aspect of my question, i.e. "Are we alone?", I try to draw a conclusion.

It (oddly) seems so, at least in many cases. And it seems this is mainly due to cultural inertia, which is very very deceiving. @gaearon I don't agree with the point that "mobile screen keep things little and easy": look at mobile applications such as Facebook, Evernote, MailChimp or AirBnb. They're huge, data-intensive apps with a lot of dependencies among screens. It's quite sure that problems such as asynchronous state transitions and UI-state bindings, are not strictly inherent to the JavaScript community, but basically it seems this is the only community trying to push things forward (and yet, as @thelambdaparty says, we've been swimming in s**t due to useless competition among the big companies ruling the web in the early years). So, what's up? Why is that?

As far as I get it, this movement is being pulled forward by entities that have particular constraints:

• their products are bound to the web;
• the web is inherently open source (am I right?);
• open-sourcing solutions and creating hype around them attracts talented developers, which allow these companies to stay lean and go fast;
• the web is a potentially conquerable no-mans-land;
• their business demands fast development pace, I mean, when you're a web-based startup, going a little faster, actually makes a lot of difference (and, these days, even Facebook acts like they're a startup);
• their products are often cross-platform, so uniforming languages is a step forward (which motivates huge, collective efforts like Node.js and React Native).

On the other side:

• tools such as C++ and Java are mostly bound to corporate environments, more closed-source, thus less prone to our "spread the love" thing;
• universes like Java EE or Xcode are more "ruled", since they depend on the decisions of a small amount of people/companies releasing the SDKs (so it's kind of "take it or leave it");
• corporate environments are full of money and old people; not the best place for shared innovation.

To sum up, the shape of web-based businesses seems to motivate this collective effort towards innovation.

Beyond these, there are some communities that innovate on the back-end, like the awesome Scala language, and even languages that implemented "revolutionary" patterns like decades ago (people saying Perl implemented most of the ES6 shiny features since the early '90s, .NET introducing Rx programming, SmallTalk being praised by Crockford, etc...) So they're actually the ones that the JavaScript community joins forces with.

This thread has definitely enriched me a lot so far. Please, feel free to notify any errors, I'm sure my answer is not bulls**t-prone. I'd be glad to read your thoughts about it.

PS: @gaearon are you ok if I tweet this thread?

[ghost]

@xbill82 i find your list to sum it up perfectly :+1: , specially first item, having the web and JavaScript as a requirement for reaching a wide audience.