Rationalize testing frameworks

[richburdon]

Gravity should contain a set of low-level testing modules and a SMALL number of non-redundant testing frameworks (or harnesses/testbenches).

Types of test

• System testing modules that handle set-up of client/peers (e.g., common construction of Client using SDK)
• Agent-based: user code that uses the full stack using the bot API
  • Single process (in-memory, fake transport)
  • Spawned processes on same machine using webrtc
  • Simulated in headless browser
  • Multiple machine testing via KUBE bot-framework

Use cases

• App/Model testing (e.g., text, object consistency)
• Stress-testing stack/pipeline to determine throughput/latency
• Simulation of network topologies
• Strict in-process tests that exercise contrived boundary and race conditions

Notes

• We should consolidate "agent" testing with the bot framework since the BotFactory-Bot management mechanism is essentially similar to the current spawn-testing framework (incl. RPC).
• Agents typically run autonomously using parties/items/model via feed replication. However, the Bot API should be extended so that Bots can receive "events" from other sources. E.g., for testing the bot/agent could receive "commands". Bots also need this functionality (e.g., sync gmail account). This requires a bot event abstraction.

Design issues

• [ ] Unify spawn/bot-factory life-cycle for agents/bots
• [ ] Event abstraction for agent/bots
• [ ] Testing/agent orchestrator to use agent/bot events
• [ ] Snapshot models for deep comparison

Candidate repos to merge:

• echo-environmental-test (rename; currently text-model depends on this).
• spawn-testing
• feed-network-replicator

Naming

In the design and code let’s use the word node consistently instead of agent. Node orchestrator, factory, etc. you spawn a node. INSIDE the node runs a bot or agent. The bot/agent is user code that may be instantiated via ipfs etc.

Nodes are peers of each other. Bots are a kind if agent. They create a Client object to interact with parties and manipulate data. But other summer kinds of agents can be instantiated within a node. Eg a random number agent that just logs numbers. Or a logging agent that just logs events.

Related: https://github.com/dxos/gravity/pull/4

[dmaretskyi]

Agent testing

Goals

• Run tests in local process, another process or different machine
• Run tests on node, chrome, firefox & safari
• Same API for all platforms
• Control network conditions (see #8)
• Allow creating custom agents that don't use existing stack components
• Reuse existing components (e.g. bot-factory for spawning)

Architecture

Remote testing mechanism

1. Orchestrator spawns centralized infrastructure: IPFS node, signaling server
2. Bundles for remote execution are created and distributed (either with node or/and browser)
   • All machines connect to central IPFS node to mitigate long IPFS sync times
   • IPFS bundles should be split into multiple files:
     • Archive with agent sources
     • Archive with node_modules
     • Extra binaries: node, browser, webrtc
     • Manifest that describes how to unpack the bundle
   • This bundle format is designed to capitalise on IPFS file caching and also combine small JS files into a single archive for performance reasons
3. Remote bot factories are launched in test mode and allow spawning agents by their IPFS hash & IPFS node URL
4. An RPC channel is created from orchestrator to each agent
   • This can reuse our existing network swarming solutions provided we can achieve a star topology around orchestrator

[richburdon]

Discussion

• The core mechanism "spawns" agents, which MAY be peers on a network.
• In principle this is similar (and will converge with) the bot factory.
• The agent provides the "runtime environment" for the nodes and mainly deals with the lifecycle of the agents.
• The orchestrator provides this environment (with arbitrary components: e.g., feedstore, transport, party-manager) to an user-provided initializer function that is called by the agent.
• The orchestrator can create agents: in memory, in headless browsers, across multiple machines.
• Agents:
  • Are concrete classes (no inheritance).
  • Request objects from the environment; e.g. the agent's initializer function can create a client from the environment.
  • Can post "metrics" to a common component
  • Can consume events that contain protocol buffer (or JSON) objects
  • Can write snapshots which can be map-reduced by the orchestrator (say, after being triggered by an event)

Eventually

• The agent orchestrator replaces the bot container

Example

const nodeGenerator = (env) => {
  const { feedStore } = env;
  return new Client({ feedStore });
}