Improve UX with frontend for managing multiple groups and other resource types

[chungers]

Currently the CLI is very developer-facing and a free-for-all in that it reflects all the http endpoint for all the plugin types. It's handy for developers to develop and test the building blocks but confusing to end users / ops.

The Issue here tracks the design and development of a separate frontend (and likely a separate CLI) that will provide a unified UX that will be much better suited for end-users of this toolkit and not just for developers.

Features

• Support configurations for multiple groups
• Support configurations for resource types
• Use file system for organizing namespaces for different groups and plugins. All configuration JSONs will be placed in appropriate locations in the filesystem. This means no more stdin via the cli tool.

This "frontend" will likely to provide means to simplify the start up for plugins -- by examining the local filesystem holding the configs and determine the plugins that need to be run. Ability to do this will be environment specific (e.g. running the plugins as unix daemons or as docker containers or whatever), so a base implementation and mechanisms for naming/ typing of plugins will have to be worked out as part of this.

[wfarner]

Can you elaborate on the UX? Doing so before breaking ground on code will be extremely helpful for others to collaborate during development, and have a shared vision of where this takes the project.

[chungers]

Motivation

• We want to have a scalable way of managing multiple groups (each with its own group config JSON) as well as other resource types (e.g. load balancers) that are coming to InfraKit.
• We want a single command that will take care of all the group watching and updating, for all groups and resources. No separate invocations of infrakit group watch <file> or infrakit group update <file>.

Design

The proposal here is that we use directory hierarchy / filesystem to manage a large collection of configuration files (in JSON). There will be a well-defined hierarchy of files and directories and the "frontend" (aka demux) will traverse and invoke the plugins as watch or update. The reason for using a directory hierarchy:

• Small config JSONs. The config file in a directory stays small as it applies only to the plugin (e.g. a Group plugin) it configures.
• Easy to edit and make changes. Bad formatting is easily isolated and fixed
• Integrates well with third-party config files that are already stored in directories -- for example, Terraform config files (or ansible) already have their own folders. Everything can be in managed under one filesystem hierarchy.
• Easy / natural backup / storage -- tar ball, git, S3 or Keywhiz backed filesystems

There are also other nice properties such filesystem events / file hashes that provide enough signal to simplify interaction with plugins such as Group, which has distinct verbs like watch and update. These verbs can be deduced from the filesystem events and hashes or commit triggers.

Initially we will only support groups. So under a top level directory, we will have something like this, for managers, workders-db, workers-large, and workers-small groups.

infrakit/
└── groups
    ├── .trash
    ├── managers
    │   └── config.json
    ├── workers-db
    │   ├── config.json
    │   └── terraform_files
    │       ├── main.tf
    │       └── variables.tf
    ├── workers-large
    │   └── config.json
    ├── workers-small
    │   └── config.json
    └── zookeeper
        └── config.json

8 directories, 7 files

/tmp$ cat infrakit/groups/zookeeper/config.json 
{
    "ID": "zk",
    "Properties": {
        "Instance" : {
            "Plugin": "instance-vagrant",
            "Properties": {
                "Box": "bento/ubuntu-16.04"
            }
        },
        "Flavor" : {
            "Plugin": "flavor-zookeeper",
            "Properties": {
                "type": "member",
                "IPs": ["192.168.1.200", "192.168.1.201", "192.168.1.202"]
            }
        }
    }
}
/tmp$ 

Under the top level directory infrakit, a groups directory indicates that all members of this folder are configurations for plugins that conform to spi.Group. In each subdirectory under groups such as managers, there is a file config.json, which is the group config JSON as they are currently defined. Note it's possible to nest folders that are appropriate for the plugin -- e.g. the terraform_files directory for config files used by the Terraform plugin.

At the moment, we assume there are no dependencies. Each group / folder is independent on one of another.

Removing (unwatch) a group (see Group plugin api) is as simple as moving a directory into the .trash folder under groups. For example, unwatching the workers-small group looks like

/tmp$ mv infrakit/groups/workers-small infrakit/groups/.trash/
/tmp$ tree -a infrakit/
infrakit/
└── groups
    ├── .trash
    │   └── workers-small
    │       └── config.json
    ├── managers
    │   └── config.json
    ├── workers-db
    │   ├── config.json
    │   └── terraform_files
    │       ├── main.tf
    │       └── variables.tf
    ├── workers-large
    │   └── config.json
    └── zookeeper
        └── config.json

8 directories, 7 files

Alternatively, when integrated with backends that have full revision history, such as git, the removal of a group can be determined from change log.

UX

Given the directory structure above, the UX / workflow is as follows:

1. Create or edit a JSON config, add new folders as necessary. Or move folders to .trash for unwatch.
2. At the command line, do

$infrakit commit ./infrakit   # assumes infrakit is the name of the top level directory.

This command invocation will tell the frontend to start traversing the directories and depending on the state of the config files (new or edits), call the group plugins watch or update endpoints accordingly. The commit can also be tied to a commit trigger for a VCS (e.g. git).

Plugin Activation

As part of processing this hierarchy of files, the frontend will activate the plugin if it's not already running. This will relieve the user from having to start the plugins manually.

We may implement pruning (deactivating) plugins if they are no longer referenced / needed in the configs. That's for a later phase.

Other Concerns

• We will build integrations for git or other backends in the future
• Backup and replication can be accomplished using existing tooling such as S3FS, Infinit volumes

[wfarner]

The proposal above leaves me unclear on the UX problem being addressed. I was anticipating that this effort would focus on relieving the user of managing startup of plugins for their configs. If i'm reading correctly, this seems to be more of a configuration management effort.

I find the config approach described above more cumbersome than the single JSON document we use today. The single document stands on its own, is atomically shareable, and it is easy to see in one place how the pieces fit together. I'm not seeing benefits of splitting it up that outweigh the added complexity of interacting with the system.

[chungers]

Addressed the point on automating plugin activation. It's implicit but was not called out explicitly in the doc.

I don't think this is any more cumbersome than a single JSON document. A single JSON document is not significantly more shareable than a tarball of the entire directory.

• Smaller JSONs are easier to work with. The directory structure is easily visualized using tools like File Explorer or tree. For counter example -- see large Cloudformation JSONs and how third party tools like troposphere tries to solve by reusable snippets of python that are used to stitch together a final, large document.
• Discrete JSON files make transition from plugin documentation to a larger system easier -- just drop the examples to a folder and edit.
• Plugin implementations may have their own configs that are not always exposed in the config JSON but nevertheless are required -- for example, Terraform's main.tf files, etc. This provides a convenient mechanism to have a central place to manage all things related to the config.

Are there specific examples of 'added complexities' you are referring to?

[chungers]

Updated the doc to make clear that a config.json is the whole config, as seen in the documentation of the plugins. I am not proposing to split up fields or chunks within a logical, atomic config into separate files.

[wfarner]

Are there specific examples of 'added complexities' you are referring to?

I gave myself the impression that config files were splitting apart each plugin configuration into separate files. The fact that configs are 1:1 with Groups alleviates a bunch of my concern.

Other than that, I find the implicit side-effects of changing files to be surprising and difficult to work with (e.g. move to .trash directory has side-effects, sounds like you may have more similar side-effects in mind).

The biggest benefit i can see is storing auxiliary files next to the relevant configuration, but my hunch is that is not big enough of a problem to be engineered away.

[chungers]

A PR #283 is a start to address this. The manager should be the entity that handles this, in terms of roles and responsibility.

[chungers]

PR #283 is merged -- this is the start of a single frontend. Note its data model at manager/spec.go has the notions of multiple groups.