giuseppefioccola
commented
10 months ago I think it must be detailed. Let's say that what we have now is a very initial version. To give you an idea on how a Benchmarking Methodology draft may look like you can see RFC 2544 (and RFC 1242 for the terminology), which are the base documents for IETF BMWG. I fully agree with the two objectives you mentioned, and I think are both very important. If you look at the IETF BMWG Charter (https://datatracker.ietf.org/wg/bmwg/about/), the benchmarking assessment fits better our initial scope, but I find the modeling of the power drawn really interesting. In this regard, I guess you also verified the model accuracy with benchmarking measurements. We can be ambitious and try to address both at the first stage. Consider that in IETF the work is normally done step by step. We do not necessarily need to include everything in the -00 version of the draft, and we can incrementally improve our document during its progress. Then if it has a too large scope, we may split it into two documents.
romain-jacob
cpignata
How deep and detailed should this document eventually go? Right now, it's way too high level and does not help compared to the past drafts and papers on the topic.
From my work on the topic, it became clear there are two related but importantly different objective to "benchmarking"
Which one do we aim to address? 1. is easier, as it is somewhat arbitrary; so you cannot really "get it wrong" but it is not easy to define test scenarios that are useful/applicable to a large set of devices. 2. is harder IMO, because one must make choices about "what to include in the model" and this will change depending on the class of devices that we consider.
Concretely, I've done 2. with a student here. We end up with a power model which is essentially the sum of:
What we worked on is a well-defined procedure to put numbers on each of those parts. Is that what we want this draft to be about?