Remote Execution

[rphillips]

Bus system for agents (Maybe)

Remote Execution Agent

• Shell TTY
• One off command
• Audit - timestamp, who ran the command
• Streaming bidirectional protocol
• Plugins - One off commands / bash + lua / scheduled commands
  • Repository for support engineers to keep and share scripts
  • Customers would have access to these scripts as well

    Server Endpoint

• Design subprotocol in websockets for the various streams we need (TTY, Multiplexing, RPC)
  • MsgPack
• Repose for Auth
• REST Interface / Websocket
• SFTP would be interesting to browse / edit files + ACLs
• Authorize connections with a Technical Contact
• Webhook: can run a one off command to inject info into alarm emails

  Demo

  1. Customer logs into the customer side of the website
  2. Support logs into the support side of the website + requests a terminal
  3. Customer gets a dialog to request permission to make a terminal
     • Both people see the terminal
     • Customer's view is read-only
     • Support side is read/write
  4. Session is done - Session is recorded and can be played back. Results + Metadata
  5. Viewable list of audit logs

[creationix]

Here is my overview of the system. Some of this is repeated/reworded from the initial meeting notes in the issue description, much is new.

---

Programs

There are 3 main programs in this system. Also other front-end systems can consume the APIs

Agent

• self-contained static binary for easy of deployment and maximum portability.
• connects to agent endpoint and authenticates self and server.
• Can spawn TTY sessions with remote control.
• Also exposes filesystem API (like sftp)
• Can port forward (for remote debugging/testing of services)
• Can run one-off shell commands
• Can run one-off script functions
• Can also schedule shell commands or script functions.
• Script plugins are run by hash and downloaded/cached and synced on-demand.
  • agent downloads missing objects from repository through endpoint automatically.

Script Repository

This is basically like an updated version of lit.luvit.io where we learned from our mistakes and using Rackspace authentication.

• Users can publish and browse scripts / libraries.
• Uses git internally to revision scripts and to enable simple syncing using lit protocol.
• Single namespace (but deep folders allowed) for scripts, first-come, first-serve. Owner can add more owners to a script, stored in commit.
• metadata for visibility, tags, etc.
• Various levels of visibility depending on role (support racker, customer, agent team member, etc)
• Exposes APIs through REST and websocket.

Agent Endpoint

• Public facing API with dual REST/websocket APIs
• some APIs are websocket-only such as attaching to a TTY session stream.
• Everything is audited/recorded. Customers can see everything related to their servers, Rackers can see everything they've done, managers can see everything their employees have done.
• proxies repository downloads for syncing script data.
• one agent API will be the ability to send off metrics of other bulk data using the AEP as a proxy.
• Maybe use repose as authentication layer.
• APIs for managing/deploying agents.
• Per agent chat rooms and maybe WebRTC signaling. Also recorded and audited.

Support Website

• Customers deploy agents through web interface
• Customers can request support through web interface
• Support rackers can receive support requests
• Support rackers can see machine/agent history as well as request special permissions like a TTY session.
• Customers with paranoid settings can authorize security requests via web
  interface.
• Customers can watch actions live as well as review past items.
• Support rackers can browse filesystem, change files, schedule or run commands or scripts, all through website.
• Customers who want to can also access support tools for audited self-service.

User Stories

The user stories will help guide implementation order and priority. We can implement the minimal functionality for each story, but see the future stories to plan ahead when architecting code.

Support TTY

This is a pretty big first step, but should make for a very impressive demo.

• Customer logs into support site and requests support for a machine.
• A support racker will take the case and see the question.
• A text chat and optionally video chat will open.
• Support racker clicks on open TTY button to request a terminal into the box.
• Customer receives confirmation prompt which they accept.
• Customer sees live tty session (read-only) while racker has interactive session.
• Everything that anyone does through the agent is recorded and updated live on the screens of anyone looking at the agent's page.
• Past actions are also listed and are viewable.

Add in FileSystem Browsing / Editing

A second step would be adding in functionality for a racker to browse the files on a box. The audit system will record nearly everything including what folders they open, what files they view, and any changes made. This is also shown in real-time on both sides and can be played back at a later date.

We can even go so far as to show the text editor as a racker edits a config file complete with cursor. The long-term record only needs remember the final state of the file when saved.

Automation - Shell Commands

In addition to giving more power to support rackers, we want to reduce the load on support rackers. This step will add one-off commands that can be done purely through APIs. They will also be recorded and audited live of course.

Automation - Scripts

Shell commands are limited to what's on the machine shell and form a more automated version of the tty shell. But scripts can run standalone logic within the agent using the APIs provided in the sandbox.

Scheduled Commands

In order to take over for the old monitoring system, we need the ability to run commands or scripts on a schedule so that the agent can automatically do things without first being told to do it every time by the AEP.

[creationix]

Note that this system is very useful for tasks other than support. The filesystem / tty combo, for example is more than enough for a developer to work on a remote machine via a web browser. I built something much like this at Cloud9 IDE.

It's the integrated auditing of everything and rackspace authentication that makes this a very useful rackspace tool, but we could easily open source bits of this to gain community support if desired.

[creationix]

Design TODOs:

Define sandbox API for scripts running in agent.

We need to decide what APIs the sandbox will provide to scripts and probably have several sets of permissions so that some scripts can be considered safer than others.

Choose Scripting Language.

We want something that people know, is very fast, and doesn't incur too much memory overhead.

LuaJIT (Lua)

Pros:

• LuaJIT is a great engine on the performance side.
• Well written Lua code can be faster than optimized C for some workloads.
• LuaJIT's memory overhead isn't too bad for a scripting language.
• We already have Luvit which contains much of what this project would need.
• We are the Luvit development team and can make changes if needed easily.
• FFI is built in for when you need to access system APIs

Cons:

• The language itself, while a pretty good language, is not well known or popular.

V8 (JavaScript)

Pros:

• V8 is also very fast.
• JavaScript is super popular and was a large reason node.js took off and Luvit did not.
• We can probably use node.js.
• Has ES6 features like generators.

Cons:

• V8 is a memory hog compared to LuaJIT.
• Binary addons in node are a royal pain, there is no FFI built into node.
• We have very little control over the direction of node.js and npm.

DukTape (JavaScript)

Pros:

• Super low memory overhead. Much better than even luajit.
• We already have libuv bindings in https://github.com/creationix/dukluv
• Uses super popular JavaScript language
• Has native lua style coroutines in addition to ES5 features.
• We can easily shape the future of dukluv since there are few users.

Cons:

• Is not JIT class performance, it's a simple interpreter.
• Dukluv is not as mature as node.js
• Does not have generators yet.

PyPy (Python)

Pros:

• Claims to be fast, I don't know how fast compared to the others.
• Python is fairly popular within rackspace thanks to openstack and general popularity among Linux system admins.
• The main maintainer of libuv is also the creator of pyuv so libuv support should be first-class with python.

Cons:

• Don't know if it's actually fast compared to others
• No idea on memory overhead in comparison.
• Language is somewhat fragmented since 3.0 broke a lot of compatibility.

Jack

Personally I'd love to design a new language from scratch purpose made for this system.

Pros:

• Make @creationix very happy.
• Language can be custom fit to needs.
• Memory overhead will be tiny.

Cons:

• Making it JIT fast is a serious investment in engineering work!
• Nobody knows the language since it doesn't exist! (not that that stopped us from creating esper)

[creationix]

another TODO is to design the API interface for AEP and script repository.

[creationix]

Steps to get started with initial prototype in luvit.

• Implement external interface to AEP using REST and websocket
• Add API to generate agent tokens.
• TLS Server for AEP with custom server cert
• Implement agent client that connects and verified cert (connection is websocket), authenticates with agent token.
• ...
• Implement sample client webapp that connects to AEP's API.