Emergence Roadmap (Automaton Runtime, Node Substrate, Integration into Relay and Ochestrator)

[CMCDragonkai]

This roadmap deals with the development of Emergence core modules. This roadmap is not final, subject to edits and change. I encourage all to expand subcomponents and write down more details development and exploration continues.

Emergence

Container Runtime Bindings

In order to orchestrate automatons and to be compatible with the leading application packaging system, we need a way to bind into container primitives from the programming language that we are writing our orchestrator in. Since our orchestrator will be in Haskell, we need a container library written in Haskell. These bindings need to be made compatible with the standard docker image format, now documented as part of OCI standard.

1. Learn about what the container image format is.
2. Document all of the container isolation primitives and investigate the kernel API for each isolation primitive.
3. Practice creating composition of primitives using low level C, shell or Haskell instead of expecting Docker or otherwise to just create it for you.
4. Investigate competing runtimes: Docker, LXC, LXD, Vagga, Openvz and compare and pick the best of all worlds features.
5. Find out what parts of a container runtime can be replaced or substituted for our own systems (such as for example its overlay/vpn network to support our relay implementation and overwatch/qos constraints).
6. Starting creating a haskell library that covers the abstractions that we want and expect this library to be utilised by the Matrix orchestrator and Forge system (talk to Roger about CI/CD systems)

Networking Experiments

1. Create example sandboxes for performance experiments.

CI/CD

1. Dat v.s. Centralised package registries
2. Global namespacing (pollution) vs local naming and aliasing
3. Continuous Integration
4. Node Creation
5. Architect Expressions
6. Docker registry vs Nixpkgs
7. Using context addresses, why they matter.
8. Object vs functional composition
9. Offline nodes
10. Identify CI stages - sequence diagram
11. Building a continuous Build system Haskell binding

For each sub component create separate issues that you can comment on the progress and then close the issue when they are done.

[mokuki082]

Emergence Roadmap update

Past

Over the last few months, we have researched multiple topics that will aid the development of Emergence. All details of the research are in this repository, but mainly:

• OCI Image format
• OCI Runtime spec
• Container runtime API design
• Nix Derivation
• Haskell -> Go binding
• Monadic IO
• Dockertools experiment

Research progress can be found in the issues tab or the repository.

Next

Here are the main deliverables that are needed for MatrixAI Artifacts at the current time.

Artifact Bundle

• Produce a Nix derivation function capable of generating Artifacts (OCI images) fullfilling user stories (user stories may need to compiled into appropriate requirements or engineering tasks). DockerTools is a good place to start.
• Create a haskell function capable of translating OCI images into OCI bundles (fs + config.json).
• Identify CI/CD stages, research possible tools (Hydra)
• Research ways of creating OCI container registry (preferably with Nix integration to support sharable resources), implement push/pull operations.
• Create a registry (or multiple peer-to-peer registries) for context addressed artifacts.
• Integrate CI/CD with created registry

Automaton Runtime

• Design the staging of Automaton runtime, and the time when it happens. For example:
  1. Produce OCI bundle from Artifact expression
  2. Add other Architect specs into the bundle's config before automaton deployment.
  3. Create the automaton
  4. Start the automaton
  5. Test the automaton (readiness/liveness probe)
  6. Link the automaton with the rest of the archiecture.
• Create a haskell library capable of calling runc to deploy the OCI bundle locally, API may be similar to runc's API or OCI container operations.
• Implement runtime staging

Architect expression

• Design the Artifact constructor capable of referencing/representing the Nix function for creating OCI images (and possibly Dockerfiles).
• Create a parser for the Artifact expression

[CMCDragonkai]

@ramwan Is currently looking into the netlink interface (for having a programmatic interface to the iproute2 package (ip link... etc commands)). We discovered that runc already does some work regarding netlink. In fact they have a dependency on: https://github.com/opencontainers/runc/tree/master/vendor/github.com/vishvananda/netlink While runc deals with networking inside a container, and this is probably mediated via a command line parameter to the runc reference implementation or as a configuration to runc using OCI specified configuration, the networking between containers and the networking between hosts between containers is not runc's job. That will be under Relay. We should investigate to what kinds of compatibility decisions we have to make between Relay and Runc.

[CMCDragonkai]

@mokuki082 The highest priority for now would be to start on the Haskell library using hnix (or the other haskell nix libraries) to deal with a hypothetical artifact expression and deploy to an OCI container. As you mentioned docker tools is a good place to start (and seeing how that might integrate with the rest of the Nix ecosystem). Note that Nix mostly just relies on external builders (those builders are usually represented as shell scripts). But this is not always true, in fact part of being able to have buildInputs or nativeBuildInputs is that these can be dependencies that required to build the current derivation. So we can do something where we bring our library/runtime to execute and build an OCI image/bundle.

From there we need to inspect how to have a p2p distributed content addressable nix store that also works with Architect expressions. And I'm hoping that @ll-aashwin-ll can help out here once he has the ball rolling with Polykey. I'll be working on developing the theory around all of this in the Architect language.