Script publishing system

[creationix]

We need a simple system for publishing scripts and a whitelist option in the agent config to limit to predefined scripts.

[creationix]

For an initial list, let's deploy a server without any authentication. You just publish a valid script and it gives you back a hash. (eventually users will auth with email)

A script is a msgpack object that references files by hash.

The system itself will be a public CAS (content addressable storage) using SHA256

The agent can download from this public CAS over public internet. The hash can be computed locally to confirm nothing was changes in transit.

[creationix]

Scripts need a schema-rpc style type signature embedded along with the lua script and any other metadata (such as author, tags, published date, description, homepage, etc).

[creationix]

Inter-dependencies of scripts are published as direct hashes, The system can tell you if there are updates and what kind based on the reverse graph.

Each module will have:

• Owners - List of users authorized to publish updates to this module
• Signature - signature of message used to authenticate.
• Parent - Optional parent module this supersedes
  • Changes - List of changes since parent
  • Level - level of change (0-metadata-only, 1-backwards compatible, 2-breaking change)
• Salt - Arbitrary data generated by server to prevent hash collision attacks.
• version - automatically generated by system in x.y format.
• Dependencies - Modules this depends on. These are aliased locally to short names for require calls.
• Name - A short symbolic name of this module
• Description - Single-line description of module
• Signature - Fully typed signature of module exports
• Docs - Markdown document documenting how to use module
• Tests - Unit tests as code that can be run automatically
  • Mocks - Optional mock replacements for system calls needed by tests.
• License - Optional license
• Code - Actual source code.

Note that parents never link to children. The system will need to walk the database periodically or on publish and create the reverse links so that updates can be suggested and discovered.

[creationix]

So there are some interesting aspects to this system:

• There is no global namespace other than the hashes themselves. If we go with something like sha256, that's a freaking huge namespace.
• Module updates are non-linear. Any module can have n descendants, projects can fork and go different directions.
• Things like public type signatures, docs, tests, are required, automated and unified.
• Version numbers are automatic. Semver-like qualities can mostly be verified through the unit tests and public signatures.
• Authentication is completely decentralized.
• Unauthenticated forks of modules can be made, but are marked as such.

[creationix]

Transient data that can be built from the data store:

• Name to hashes with that name
• User to published modules
• Module hash to child modules
• Module to dependent modules

Using this calculated data, useful queries can be made on this opaque data structure.

[creationix]

So I think we'll just use email alone for authentication. The system is simple. A user logs in using a cookie (or equivalent for CLI apps). If they are on a new machine, we send them an email with activation link. I've had great experiences as a user using this style on https://asciinema.org/.

[creationix]

Also for the initial prototype, we'll use Redis for storage. It's quick and simple. Most the data is immutable, the only thing that changes over time is the reverse indexes, but they are basically in the key -> list of hashes shape that the LPUSH command does nicely.

Once we're past proof-of-concept stage, we can look at servers with better persistence properties if needed.

[creationix]

With minimal additional work, we can add a field describing the intended platform of the script and allow publishing code for any kind of system (super-agent sandbox, powershell, node, luvit, browser, jack, uscript, etc..) Then this project might be useful to people outside super-agent. I know I would love to try it on some projects.

[creationix]

So let's stick with websocket protocol. This way we have a session concept. Users can upload hashes, query for missing hashes (essentially lit graph sync protocol) all without needing to authorize the uploads. Then they can publish the package that depends on them and the server can cull any uploads not directly related to the publish.

[creationix]

We could also use a plain REST style interface, but add a TTL for uploaded assets that auto-prunes them if they aren't anchored after a certain timeout.

Any valid automated client should be done in seconds, a human doing it manually should be done in under a minute. We could start with a 5 minutes window to be safe.

[creationix]

This is deployed to https://cas.luvit.io for now.