Configuration Graph Nodes

[nzhang-zh]

List up all the configuration parameters that should be present in the configuration graph. Consider this from the perspective of each Matrix system component.

[nzhang-zh]

Emergence would need to refer to the configuration graph to build Artefacts and configure and deploy Automatons.

Artefacts

We could have build dependencies represented in the configuration graph. In container based Artefact, this would be multi-layered image referencing blobs of layers. In Nix based Artefact, this would be package level dependencies similar to Nix's dependency graph.

• Artefact node: a fully specified Artefact which can be unpacked into a valid directory structure
• Layer node & Package node: build dependency of an Artefact

These nodes would be created/written by Forge and read by Emergence.

Runtime configs

Most runtime configurations are optional at Artefact building stage but are required to create or start an Automaton. So they should be represented in the configuration graph external to Artefact nodes.

• Runtime configuration node: parameters required for Automaton deployment and execution. Each parameter could be its own node, with a central node grouping them together as a single configuration.
  • Environment variables
  • Entrypoint, CWD
  • Storage volume mounts
  • Namespaces
  • Seccomp
  • Resource constraint, cgroup, rlimits
  • Hooks
  • Liveness/Readiness probes

The configuration node could be related to Artefacts and/or Automatons.

Automaton

The Automaton itself should be present in the configuration graph. We need to distinguish between the abstract Automaton as created by the operator and actual running Automaton instances. For example, from an operator's perspective, an Automaton could be simply a webserver. But in matrix network, there could several instances of the webserver running in different Automatons based on number of user requests, latency, location, etc.

For lack of better words,

• Automaton node
• Automaton instance node

[nzhang-zh]

Since we can separate nix based images from their closures https://github.com/MatrixAI/Emergence/issues/46#issuecomment-446100144, we need the capability to create a closure from an artefact's dependency nodes.

Nix has this capability natively.

[nzhang-zh]

Emergence Configurations

Notations

Each configuration is in one of the following catagories:

• Encapsulated: Value is defined from within container/artifact runtime.
• Optional: Value can be derived from within container/artifact runtime if not specified.
• Required: Value must be read from configuration graph.

Artefact Configuration

Nix Artefact

• drv :: string, Required
  • The nix derivation to build the artefact

OCI Artefact

• imageName :: string, Required
• imageDigest :: string, Required
• sha256 :: string, Required,
  • Hash of the image's unpacked root filesystem.

Automaton Configuration

• node :: string, Encapsulated, Need default
  • Identifier of the node to deploy to
• artefact :: string, Required
  • Identifier of the artefact being deployed

Runtime Configuration

Refer to OCI runtime spec.

Spec Version

• ociVersion :: string: Encapsulated, "1.0.0"

Root Filesystem

• root :: object, Optional
  • path :: string, Encapsulated
  • Root filesystem path can be fixed during artefact creation.
  • readonly :: bool, Optional
  • Default to false, or
  • Default to false if no state volume mounted and true otherwise

Mounts

• mounts :: [object], Optional
  • System mounts, /proc /sys /dev/pts etc, Encapsulated
  • Nix store mount, Encapsulated
  • Volume mounts, Optional derived from State Specification
  • Others, Optional

Process

• process :: object, Required
  • terminal :: bool, Encapsulated, false or true if debug mode
  • consoleSize :: object, Encapuslated, omitted
  • cwd :: string, Optional, default to "/"
  • env :: [string], Optional, default to "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
  • args :: [string], Required
  • rlimits :: [object], Optional, default to omitted
  • Available to Optimization but not to Operator
  • apparmorProfile :: string, Encapsulated, omitted
  • capabilities :: string, Optional, Need Default
  • noNewPrivileges :: bool, Optional, default to true
  • oomScoreAdj :: int, Encapsulated, omitted
  • selinuxLabel :: string, Encapsulated/Optional, Need Default
  • user :: object, Encapsulated, uid:0 gid:0

Hostname

• hostname :: string, Optional, default to artefact name

Hooks

• hooks :: object, Optional, default to omitted
  • prestart :: [object], Optional, default to omitted
  • poststart :: [object], Optional, default to omitted
  • poststop :: [object], Optional, default to omitted

Linux Specific Configs

• namespaces :: [object], Encapsulated, all

Annotations

• annotations :: object, Encapsulated, omitted

[CMCDragonkai]

Encapsulated: Value is defined from within Emergence.

I think you mean the container/artifact runtime. The emergence module involves the config graph as well.

[nzhang-zh]

@CMCDragonkai

Our release.nix should have a debug flag that allows one to add a small busybox shell into the container builds to allow a debuggable container.

Process.terminal is still encapsulated but changed to true in debug mode

[CMCDragonkai]

Oh that's not related to each other. I'm talking about release.nix for building any our applications and making them debuggable. Nothing to do with OCI.

[CMCDragonkai]

Actually I don't even know what does Process.terminal does?

[nzhang-zh]

Actually I don't even know what does Process.terminal does?

From oci runtime spec,

terminal (bool, OPTIONAL) specifies whether a terminal is attached to the process, defaults to false. As an example, if set to true on Linux a pseudoterminal pair is allocated for the process and the pseudoterminal slave is duplicated on the process's standard streams.

[nzhang-zh]

Also need to represent available hosts and their available resources as nodes in the configuration graph.