Project plan

[gabrielecirulli]

For lack of a better repository, this issue will be hosted and discussed here for now. I will migrate it to the server-side Koral repo once created.

This is a living document, currently in draft state. It will outline the plan for the project and host related discussion, but won't go into much detail to avoid paralysis by analysis.

Project overview

Koral is built in two parts:

• The server-side application will handle file storage and versioning, and serve a client-side web app that renders the project browsing interface.
• The desktop application will track changes to files within a specific directory and sync them to the server-side application.

Koral is meant to be deployed manually by each team that wants to use it. The team is responsible for setting up the resources that Koral needs in order to run (servers, storage). The deployed Koral server and all of the clients installed and connected to it are referred to as a group.

High-level features:

• Support for PSD and Sketch files;
• Directory structures within projects, used to keep assets in order.
• Ability to browse previews and history of each file and, if necessary, revert it to a specific point in history;
• Support for threaded comments on each file (similar to how LayerVault implemented it)
• Desktop syncing: by installing a client you can work directly on your mockups and have them uploaded in the background, similarly to how DropBox works.
• User accounts: every user has an account, and you can easily invite new users to your group.

  Technical overview

The chosen language for this project is JavaScript, and Node.js will be used. The desktop application will make use of atom/electron to provide the necessary interface and affordances.

Architectural details

Stores

Due to the roll-your-own nature of Koral, we need to support a multitude of storage methods when it comes to the raw design files. They include:

• GitHub LFS
• Amazon S3
• Mounted local filesystem

Due to the differences between these services, we need to develop the server-side application in a way that is agnostic of the underlying store, by making use of adapters (specific to each store) which abstract all the necessary functionality, namely:

• A way to access and transfer any file's contents by providing its unique ID.
• Full revision history for each file (any revision should be downloadable, however fault-tolerance should be present to handle data loss);
• Low-priority hosting of snapshots (rasterized PNGs) for all revisions of all files;
• Garbage collection to avoid piling up of unneeded data (deleted files and untracked revisions).

Adapters will all have to expose the same API. This makes them fully pluggable, and new adapters may be implemented with ease.

A basic file-system will be implemented in the actual server-side application to allow creating directories and sub-directories. In this system, files will be referenced using unique IDs. Adapters will not need to be aware of the directory structure. They must however be able to provide a file's contents at any specific revision when given the file's ID.

The code necessary to generate a snapshot of a file will also be part of the server-side application, and it will work by fetching the file contents with the adapter, generating a snapshot and feeding it back to the adapter. Adapters will handle the storage of the rendered snapshots, and provide ways receive a file's snapshot when requesting a specific point of its history.

The details of the abstraction are explicitly left to the store adapter, so that they may simply expose a common API that can be plugged into by the server directly.

Web client

The web client should offer the following:

• An overview of the latest changes made in a project (a timeline) which makes it possible to tell what files have been updated and how many revisions have been made for each, at a glance;
• Ability to create new projects;
• Ability to enter a project;
• A fully browsable directory tree for the selected project, displaying all files and folders;
• Previews and history for each file;
• Ability to upload files and revisions to a specific place. This is meant to act as an alternative to client-side syncing.

  Desktop syncing protocol

The server and desktop applications will sync changes to each other in order to keep file versions updated. A pre-existing sync protocol should be used if possible, in order not to spend engineering effort on something that has already been invented.

The main requirements are:

• Ability for the server to broadcast new revisions to all clients (within the group) at the same time;
• Ability for clients to upload new revisions to the server;
• Appropriate tracking of the time of each change, so that history may be replayed correctly when multiple clients are active on the same file;
• Conflict detection and mitigation (ideally, conflicts should be reported and the user should be in charge of resolving them by accessing both source files and performing the changes necessary);

Secondary requirements:

• Use of HTTP/HTTPS

  Desktop application

The desktop application is used to sync project files to a specific location on the local filesystem, and to track and upload any revisions made by the user.

The client will support syncing to a single folder. This folder will be treated as the root of all projects. Since projects are treated separately of each other by the server, the application may in the future support tracking different projects in different directories. This constraint will not be present in the beginning in order to simplify the design.

In order to set up the client, the user will have to specify the address at which the Koral server is installed, together with their authentication credentials (every user has an account). Once the address is provided, the client will attempt to authenticate, and begin fetching the projects and files to the directory specified (by default, /home/Koral).

[gabrielecirulli]

@iDuuck, @sebmck, @filp please feel free to share any of your thoughts. The document is just a draft and literally everything is up for discussion!

[iduuck]

I like. I removed the choice for io.js in the document, since it's been merged into node.js.

Edit: What about technical things? Precompiling through middleware/gulp and so on.

[gabrielecirulli]

Good call about io.js. For the tooling part, I'd reckon that's more of an implementation detail than a determining factor (compared to choices like using JavaScript vs another language), so maybe although we can mention it I think it shouldn't necessarily take a lot of precedence on purely architectural choices. What do you think?

[gabrielecirulli]

On second thought it's quite a blurry line between what matters and what is secondary, in terms of tooling. I'd say we can just approach it pragmatically: if the need for a discussion arises, we can handle it at that point.

[gabrielecirulli]

@iDuuck I've added a bunch of things (syncing protocol, desktop application, high level feature overview). I think we might have enough to work off of already. Any thoughts on this before we proceed?

[iduuck]

What about the idea of splitting everything up into little pieces of Node.js packages and just requiring, when they are needed. So, you have the koral package with the core and little extra packages named something like koral-s3, koral-ghlfs and so on. So the concept would be, that the user only needs the additional stuff when they really wanna go with it.

[filp]

Breaking up those sort of components into separate packages is worth it purely for the organizational benefits - also makes a lot of lines clearer (what to test where, for example). :+1:

[iduuck]

To continue this discussion: The guys over at Thoughtbot are doing something like this for middleman (just using ruby).

[gabrielecirulli]

Sorry for the delay (woah, where did time go), and yes, I think splitting the packages would be good too. Especially because it would enforce the need for a standardized API between those and the core.