Testing Framework

[richburdon]

Test-runner app (or script?) to spawn signal server and spawn N clients (as separate processes) that connect to a party and replicate.

Spec

• [x] Spawn signal server
• [x] Spawn clients (pass configurable options object to each client)
• [x] Orchestrator commands one client to create a party and get the discovery key; invites others (through spawn stream as we do in bots today) [@egorgripasov]
• [x] Configure one client to create a party and send invites
• [x] Orchestrator passes party discovery key and pin to client
• [x] Other processes accept invites (shared PIN)
• [x] Client create or mutate Views with the ObjectModel (or other models)
• [ ] All clients randomly create, update, delete objects (agent/generator)
• [ ] Configure to run for N seconds before stopping
• [ ] Parse logs to detect state

Tests

• [x] Check all clients started-up and ran for the time period (logging failures)
• [x] Test convergence of data
• [x] One agent with DefaultModel pushing as much messages as possible
• [x] One agent with DefaultModel writing and another one reading
• [ ] Multiple agents with ObjectModel randomly creating, mutating
• [ ] Test maximum throughput
• [ ] Test catch-up time
• [ ] Sync time for a batch

Next

• [ ] Randomly connect/disconnect to the swarm (go online/offline)
• [x] Different models
• [x] Run processes using browser-runner
• [ ] Spawn bots
• [ ] Logs

[tinchoz49]

related: https://github.com/dxos/sdk/pull/12

[tinchoz49]

related: https://github.com/dxos/sdk/pull/27

[dboreham]

Notes on running:

Test:

1. This test creates 10 peers. Each peer then creates 1000 ObjectModel "items" (total = 10 * 1000).
2. The test framework waits for all mutations to quiesce across the network.
3. Elapsed time to reach quiescence is reported (e.g. 8430ms in the run below == 843us average per object mutation).

Setup:

$ cd sdk
$ yarn
$ yarn build

Run:

$ cd testing/spawn-testing
$ yarn example
yarn run v1.22.4
$ npm run build && DEBUG="dxos:spawn-testing*" node example.js

> @dxos/spawn-testing@1.0.0-beta.121 build /home/david/projects/wireline/sdk/testing/spawn-testing
> npm run clean && babel ./src --out-dir ./dist/es --ignore "**/*.test.js" --source-maps inline

> @dxos/spawn-testing@1.0.0-beta.121 clean /home/david/projects/wireline/sdk/testing/spawn-testing
> rm -rf dist

Successfully compiled 8 files with Babel (536ms).
  dxos:spawn-testing:signal discovery-swarm-webrtc running on 4000
  dxos:spawn-testing:signal  +0ms
  dxos:spawn-testing:example > signal started +0ms
  dxos:spawn-testing:example > peer0 created +516ms
  dxos:spawn-testing:example > party created 2f6c2edaa2873e9c0eb4a377b9449b82932a2eb84ed0d173006cda719f12d545 +33ms
  dxos:spawn-testing:example > peer1 created +527ms
  dxos:spawn-testing:example > invitation created +3ms
  dxos:spawn-testing:example > peer1 joined to the party +214ms
  dxos:spawn-testing:example > peer2 created +545ms
  dxos:spawn-testing:example > invitation created +4ms
.
.
.

Results:

  dxos:spawn-testing:peer  +15ms
  dxos:spawn-testing:peer model-update client=7d57ff messages=106 total=1000
  dxos:spawn-testing:peer  +6ms
  dxos:spawn-testing:example > sync successful +8s
sync: 8429.868ms
Done in 17.61s.

[dmaretskyi]

Testing sync times for small message batches

Sync time, ms / batch size, messages

Test methodology

Spawn two agents in chrome. One writes a bunch of messages. Wait for the other agent to process all of them. Measure the time elapsed. Repeat with bigger and bigger batches

Logs pastebin: https://pastebin.com/7DS8rAmC

Notice how after we pass ~10 messages we see the issue with only one message being delivered at first, and then the rest. This also coincides with around 2x jump in sync time.

Close-up on the lower ranges: