Closed simonduq closed 6 years ago
I think we should convert 'testbed' to 'test environment'. In my opinion, we should open up the benchmark to be run also on simulators, or even ad-hoc networks ('on-the-desk' experiments, or non-open testbeds).
Agree. Although I'd like to come up with a better name, because "environment/setup/configuration" might be a bit unclear.
Definitely. I'll try to think about this and make a proposal
Okay, here is a proposal to feed the discussion. I suggest to adapt the data scheme we currently have in the repo as follows.
There are 4 categories of things:
Plus optional information relevant to report with your results, for example
From there, the main open challenge remaining (in my opinion) is the definition of the profiles, and in particular how to define the output metrics: should we specify the mesurand/metric (e.g., average energy consumption) or the physical phenomenon (e.g., power draw per node).
See the discussion there: https://github.com/iot-benchmark/iot-benchmark.github.io/issues/4
Looks good, only thing I'm unsure about is if we want so many different config fields for each run, as opposed to adding one more structure that describes a configuration (node IDs used, protocol config etc.). And maybe even this goes into "experimental setup"?
If runs could simply point to a config, we'll be able to easily look at all iterations of any particular config.
I think I see what you mean: nodeIDs used, protocol config etc. are field of a "result", not of a specific "run". Correct?
If so, that's also what I had in mind, but I realize the layout was a bit confusing. I would not put it in the 'experimental setup' though, because a setup here is rather general (aka 'not made by the user') while the 'results' contains everything which is 'produced' by the user.
If not.. then I did not get your point.
That's what I meant yes, and I realize only now that's also what you proposed initially. I was confused, thinking your last bullet list was a per-run thing, but it's part of "result".
So in terms of modification of the current data schema, it's mostly:
That would be my proposal.
a side-effect is that all results become a set, as a field of "result", might be a lot less flexible than having one separate element per run, each pointing to something that describes the full configuration
Both are not exclusive I think. I would be in favor of keeping a 'run' structure, similar to what you proposed (containing essentially the metrics, a timestamp and the 'result_id'). Then, the 'result' structure would contain a list of 'runs': run1, run2, runx... all related to the same profile_id, protocol_id and setup_id.
OK, that works for me. Let's update the data schema and jekyll implem. Feel free to start with the schema on the wiki :)
We can have fields for both per-node and end-to-end fields and let the users choose to fill in the fields depending on the environment they run the experiment in.
We have converged on something that seems to make sense for now. Let's give it a try and start contributing results. We will see if we need to adapt the structure.
So, based on the last call, we should try to find something else than 'result'. I don't remember everything that was said... What about this:
Rename: 'Experimental setup' -> 'Setup' 'Result'-> 'Experiment'
That would mean an 'Experiment' becomes the association of
With this, we still have the problem that was mentioning Om: where do I put the experiment specific things ?
key parameters (e.g., LWB_ROUND_PERIOD = 1), testbed configuration file (e.g., Flocklab XML template), complete list of node IDs (only relevant for testbeds) etc.
I would be against having it in the 'Setup', as it would lead to have essentially one 'setup' per 'experiment'. I would prefer keeping a limited amount of setups. Ideally, I think it would be good that uploading results to the repo requires as few files as possible. If we rename 'Result'-> 'Experiment', it would make sense to have the protocol parameters etc. in there. or?
My proposal was:
Okay, that makes sense. Let's go with your proposal. I will update the data schema accordingly asap (hopefully today).
Thanks! Please give a heads up when the schema is updated :)
A clarification, so a specification or setup includes {profile, protocols, environment}. For each setup, we can many runs, each run has additional values for output and observed metrics. Make sense to me.
Done
In the current data schema, runs refer to a setup, which in turns consists in a profile, testbed, protocol and configuration. Needs discussing more.