New Backend Architecture

[sunjay]

This is a new architecture for the backend of Rhino. It does not need to be implemented before 1.0.0 because it will not change existing functionality at all. It will however provide the following benefits:

• low-level, really fast API
• plugins
• extremely non-blocking everything

This is heavily inspired by this awesome talk about the xi editor.

Resources:

• https://medium.com/@raphlinus/towards-a-unified-theory-of-operational-transformation-and-crdt-70485876f72f#.dxxheuyxl
• https://medium.com/@raphlinus/working-code-for-operational-transformation-crdt-hybrid-9d04a57309da
• http://www.codecommit.com/blog/java/understanding-and-applying-operational-transformation
• https://www.ibm.com/developerworks/library/j-ropes/

Proposed Actions:

• [ ] Change the main process (the "core") into a pubsub server (like xi) which is capable of spawning other processes for things like the UI, plugins, etc.
  • We make the main process spawn everything so we can't have untrusted sources interfering with the application
  • If the main process is doing process management, it can re-spawn any killed processes too
  • The main process can spawn as many workers as necessary to do its work
  • Killing the editor is just a matter of killing the main process tree instead of hunting down however many processes might have been spawned
  • This architecture is simpler to manage and maintain because all the control is in a single source
  • Processes will communicate over stdin/stdout because the main process is spawning them
    • Other options: ipc-channel, man 7 unix, environment variables (for sharing host/port info), etc.
  • Should be ways to send messages to other processes if necessary (similar to android intents?)
• [ ] Implement Operational Transform
  • Single copy of the source in the main process is the source of truth
  • The main process reconciles all change requests and commands issued to it
  • https://github.com/google/xi-editor#design-decisions
• [ ] Read about the rope data structure and consider if there is a way to adapt that or something similar to work with images (of any size)
  • There is a problem currently with sending large image files
• [ ] Implement a versioned API that the backend implements and clients/plugins can use to query or mutate the editor state
  • Each API method is versioned with a number (i.e. added/updated/changed in version x.y.z) so you must declare in your plugin that you support up to a certain version of the API. All the methods in that version and below will be available to your plugin and the main process can determine whether your plugin can be functional with the current editor version being run.

Use GraphQL

[sunjay]

Some very rough and early notes about plugins, async and the overall multiprocessed architecture.

Image worker's new design should support several things in addition to what has already been stated:

• multithreading support so each action is taken in a different thread
  • future-based API
  • action cancellation (kill the thread or cancel the future)
  • progress (by adding a bunch of code to the image crate behind a feature gate)
• multiple "contexts" that represent the projects being worked on

All the async stuff should be decoupled from everything else so this is testable.

For implementing progress, we want to add it to the image crate without impacting other people's performance.

Submit an RFC issue to the image crate and describe the approach.

Could be worthwhile to look at using a futures::stream. Instead of having the image processing functions return a synchronous result.

There could be an async feature that adds _async versions of all image operations. These would return futures or streams. In async mode, image operations would run in their own thread. Cancellation should be possible.

When a progress feature is enabled, async functions would return streams that yield an enum representing a change in progress or the final result. Otherwise just a future is necessary. It's possible that people may always want progress in which case we don't need to implement the two feature gates like that. It may be useful to implement the progress struct behind the gate anyway just in case.

The progress should be modelled using a struct that requires a "total amount of work" (usize) when being constructed. Instances of that struct provide a method to progress which should allow the progress to go up or down but never over 100% or under 0%. If those invariants are not met in development mode, panic. If those invariants are not met in release mode just round to the nearest value.

If the progress feature is not available, the progress struct should be empty and its methods should all be implemented as empty so that it does not impact anyone's performance.

The progress feature depends on the async feature.

This stuff is based on a tweet about plugins as components. I haven't figured it out fully yet but I want to keep considering that instead of just going with plain plugins.

[sunjay]

It would be really cool if the UI plugin architecture could be completely agnostic of the technologies and frameworks being used in the Rhino UI. These technologies (currently react/redux) will likely change or grow or break in the future. It would be cool if the UI plugin framework was so flexible that you could run the plugins in a webworker and simply pass data around.