Closed serapath closed 4 years ago
@ninabreznik could you check out https://github.com/mafintosh/hypercore/issues/242 and try to do the stuff that @RangerMauve suggested? Maybe we can give feedback to him and @mafintosh :-)
Hey i stumbled upon this thread by accident. Recently I've been building something to answer similar questions, and reading this thread i got curious if it would be of help to you too :)
It allows you to spawn peers with different behaviour and run scenarios in a (not yet fully deterministic) simulated environment sampling metrics and statistics for each simulation step.
oh @telamon thank you soooo much :-) it looks very interesting.
our project actually is kind of a bridge between two complex ecosystems.
...but for the dat part it looks very useful :-)
It doesn't mention in detail how it conceptually works.
I could see from https://github.com/decentstack/hyper-simulator/blob/master/scenarios/hypercore-swarm.js and also from the README.md
example how I can use it.
Especially
// ...
const {
// 1.
id, // Uniquely generated id
name, // Name of peer as specified by launch()
storage, // An instance of random-access-file,
swarm, // An instance of simulated hyperswarm
opts, // Copy of opts given to launch()
// 2.
signal, // Function signal(eventName, payload) (more on this below)
ontick, // Lets you register a handler for peer.tick event: ontick(myHandlerFun)
simulator // reference to the simulator.
} = context
// ...
are all those using the original libraries?
what's simulator
, peer.tick
and the details behind signal
? :-)
@serapath Ah I see, yeah it only makes-sense for dat-stack simulations.
But as for how it works is pretty simple, the whole simulation runs offline and in-memory.
It features a simulated swarm and discovery process compatible with the hyperswarm
API.
Each connection generated during simulation will yield a ThrottledStream which gives us complete control over bandwidth.
So the simulator runs in iterations where each call to sim.process(deltaTime)
generates a heartbeat/"tick"
During each tick; we animate the simulated swarm, firing simulated peer
and connection
events, and then pump all active ThrottledStreams providing them with bandwidth calculated by the amount of bandwidth remaining between the two peers. (the QoS
could use some improvement.)
Each such process iteration generates multiple ndjson
messages.
The signal
method is a helper to let peers in your scenario report their own ndjson
messages during the run. :) (your messages automatically get stamped with the peer-id nothing special)
The resulting logfile can be analyzed offline or in realtime but given some custom signals it can tell you how your application will behave given a network of N-Peers.
The project arose as an alternative to organize 30 people to run your_application@version
and ask them to shout when the desired objective is reached. :)
Anyway, let me know if you give it a test-run :heart: I've only used it to debug scaling issues in my own apps so far.
P.S.
The storage
is convenient random-access-file
wrapper that subpaths everything as $PWD/_cache/${peer.id}/${requested_path}
, the _cache
folder gets deleted before and after each run.
wow, that's something you might want to add to your readme. I'll definitely try it sooner or later. Awesome to know that this exists, so we don't have to re-invent something like it :-) Thx
@serapath yes, thank you for asking inspiring questions :)
@telamon btw. Karissa told us you might have some time in between here and there and we could need a helping hand with some of our tasks. If that's true and you are interested, we could pay a couple of hours every week to solve issues :-) Not sure if you use the gitter chat, because I wrote you a PM a long time ago, but I guess it didn't get through. We also have a telegram group and otherwise our public gitter channel next to github.
@todo
lab/
folder with multiple namedlab/<scenarioNameFolder>/
s inside.lab/supporter-encoder-seeder/
) would have one js file per "node", e.g.:supporter.js
encoder.js
seeder.js
server.js
which would later be - of course - replaced by the substrate nodeTo give one example for the
"brotli encoding scenario"
you were already working on, it might be possible to add theserver.js
using something like the example code below:server.js
seeder/encoder/supporter.js
package.json
fornpm start
...i hope that makes the idea of all of this more clear :-)