make lab environment to reliably test hyper-* scenarios

[serapath]

@todo

• [x] turn little tests and experiments which involve multiple "nodes" into proper tests inside a lab
• [ ] use dat and substrate testing frameworks systematically

---

1. I would recommend a lab/ folder with multiple named lab/<scenarioNameFolder>/s inside.
2. each one (e.g. lab/supporter-encoder-seeder/) would have one js file per "node", e.g.:
   • supporter.js
   • encoder.js
   • seeder.js
   • and one server called server.js which would later be - of course - replaced by the substrate node

To give one example for the "brotli encoding scenario" you were already working on, it might be possible to add the server.js using something like the example code below:

server.js

const http = require('http')

const PORT = 8989
const server = http.createServer(handler)
server.listen(PORT, () => console.log('listening on  http://localhost:' + PORT)

const originalFeed, encodedFeed
function handler (request, response) {
  const [ url, params ] = request.url.split('?')
  // request.url === '/feed1?dat=3j239fj23jtw093jtw90tjjtw9'
  if (url === '/feed1') {
    originalFeed = params.split('=')[1]
    response.end()
  } else if (url === '/feed2') {
    if (!originalFeed) throw new Error('did not receive feed address from SUPPORTER yet')
    encodedFeed = params.split('=')[1]
    response.end(originalFeed)
  } else {
    if (!encodedFeed) throw new Error('did not receive feed address from ENCODER yet')
    response.end(encodedFeed)    
  }
}

seeder/encoder/supporter.js

// use the code you already have for each of those "nodes", but add colors
// each different node should use a different color
// so that when they run in the same terminal and log their output
// a developer can differentiate by the color which node logged the message
var colors = require('colors/safe');

// to have each of the three processes log in different color, maybe prefix with
console.log(__dirname.split('/').pop())

console.log(colors.green('hello')); // outputs green text
console.log(colors.red.underline('i like cake and pies')) // outputs red underlined text
console.log(colors.inverse('inverse the color')); // inverses the color
console.log(colors.rainbow('OMG Rainbows!')); // rainbow
console.log(colors.trap('Run the trap')); // Drops the bass

package.json for npm start

{
  "scripts": {
    "start": "node server.js & npm run feed1 & npm run feed2 & npm run feed3",
    "feed1": "node ./supporter.js",
    "feed2": "node ./encoder.js",
    "feed3": "node ./seeder.js"
  }
}

...i hope that makes the idea of all of this more clear :-)

[serapath]

@ninabreznik could you check out https://github.com/mafintosh/hypercore/issues/242 and try to do the stuff that @RangerMauve suggested? Maybe we can give feedback to him and @mafintosh :-)

[telamon]

Hey i stumbled upon this thread by accident. Recently I've been building something to answer similar questions, and reading this thread i got curious if it would be of help to you too :)

hyper-simulator

It allows you to spawn peers with different behaviour and run scenarios in a (not yet fully deterministic) simulated environment sampling metrics and statistics for each simulation step.

[serapath]

oh @telamon thank you soooo much :-) it looks very interesting.

our project actually is kind of a bridge between two complex ecosystems.

1. dat ecosystem (or hyper-* ecosystem)
2. substrate ecosystem (or "polkadot blockchain networks")

...but for the dat part it looks very useful :-)

It doesn't mention in detail how it conceptually works. I could see from https://github.com/decentstack/hyper-simulator/blob/master/scenarios/hypercore-swarm.js and also from the README.md example how I can use it.

Especially

// ...
  const {
    // 1.
    id,      // Uniquely generated id
    name,    // Name of peer as specified by launch()
    storage, // An instance of random-access-file,
    swarm,   // An instance of simulated hyperswarm
    opts,    // Copy of opts given to launch()
     // 2.
    signal,  // Function signal(eventName, payload) (more on this below)
    ontick,  // Lets you register a handler for peer.tick event: ontick(myHandlerFun)
    simulator // reference to the simulator.
  } = context
// ...

1. are all those using the original libraries?

   • how are they augmented?

2. what's simulator, peer.tick and the details behind signal? :-)

[telamon]

@serapath Ah I see, yeah it only makes-sense for dat-stack simulations.

But as for how it works is pretty simple, the whole simulation runs offline and in-memory. It features a simulated swarm and discovery process compatible with the hyperswarm API.

Each connection generated during simulation will yield a ThrottledStream which gives us complete control over bandwidth.

So the simulator runs in iterations where each call to sim.process(deltaTime) generates a heartbeat/"tick"

During each tick; we animate the simulated swarm, firing simulated peer and connection events, and then pump all active ThrottledStreams providing them with bandwidth calculated by the amount of bandwidth remaining between the two peers. (the QoS could use some improvement.)

Each such process iteration generates multiple ndjson messages. The signal method is a helper to let peers in your scenario report their own ndjson messages during the run. :) (your messages automatically get stamped with the peer-id nothing special)

The resulting logfile can be analyzed offline or in realtime but given some custom signals it can tell you how your application will behave given a network of N-Peers.

The project arose as an alternative to organize 30 people to run your_application@version and ask them to shout when the desired objective is reached. :)

Anyway, let me know if you give it a test-run :heart: I've only used it to debug scaling issues in my own apps so far.

P.S. The storage is convenient random-access-file wrapper that subpaths everything as $PWD/_cache/${peer.id}/${requested_path}, the _cache folder gets deleted before and after each run.

[serapath]

wow, that's something you might want to add to your readme. I'll definitely try it sooner or later. Awesome to know that this exists, so we don't have to re-invent something like it :-) Thx

[telamon]

@serapath yes, thank you for asking inspiring questions :)

[serapath]

@telamon btw. Karissa told us you might have some time in between here and there and we could need a helping hand with some of our tasks. If that's true and you are interested, we could pay a couple of hours every week to solve issues :-) Not sure if you use the gitter chat, because I wrote you a PM a long time ago, but I guess it didn't get through. We also have a telegram group and otherwise our public gitter channel next to github.

[serapath]

• https://github.com/decentstack/hyper-simulator