Open romain-jacob opened 7 months ago
I think it must be detailed. Let's say that what we have now is a very initial version. To give you an idea on how a Benchmarking Methodology draft may look like you can see RFC 2544 (and RFC 1242 for the terminology), which are the base documents for IETF BMWG. I fully agree with the two objectives you mentioned, and I think are both very important. If you look at the IETF BMWG Charter (https://datatracker.ietf.org/wg/bmwg/about/), the benchmarking assessment fits better our initial scope, but I find the modeling of the power drawn really interesting. In this regard, I guess you also verified the model accuracy with benchmarking measurements. We can be ambitious and try to address both at the first stage. Consider that in IETF the work is normally done step by step. We do not necessarily need to include everything in the -00 version of the draft, and we can incrementally improve our document during its progress. Then if it has a too large scope, we may split it into two documents.
I would propose to scope the current draft as focusing on 1. "Assessing "who performs best" over a set of well-defined scenario" with a clear separation with objective 2. I think it would make things too complex/messy to try integrating both in a first draft.
If the feedback we get goes towards "please include 2" as well, we can see then. I feel it would make progress less daunting if we start off with problem 1 only for now.
Thoughts?
@romain-jacob I agree with you. It is better to focus on the bechmarking assessment at first and then let's see. I have proposed some new text and I have opened a PR (https://github.com/cpignata/powerbench/pull/4). In particular I added the energy efficiency definition given by ETSI (https://www.etsi.org/deliver/etsi_es/203100_203199/203136/01.02.00_50/es_203136v010200m.pdf) and I referred to this work in this new revision. It is just to start discussing how to perform benchmarking tests. Please have a look and feel free to edit.
Hi, Giuseppe, Romain,
I also fully agree with the strong demarcation on scope on (1) and leaving (2) out. As a refresher:
My sense is that (1) is a very useful BMWG document, and we will need to look into the variables to define the scenarios (ports, util %, packets distro, feat, etc). However (2) is a non-RFC paper.
I’ll build some time early next week to go over the commits and issues.
On Jan 30, 2024, at 1:50 PM, Giuseppe Fioccola @.***> wrote:
@romain-jacob https://github.com/romain-jacob I agree with you. It is better to focus on the bechmarking assessment at first and then let's see. I have proposed some new text and I have opened a PR (#4 https://github.com/cpignata/powerbench/pull/4). In particular I added the energy efficiency definition given by ETSI ( https://www.etsi.org/deliver/etsi_es/203100_203199/203136/01.02.00_50/es_203136v010200m.pdf) and I referred to this work in this new revision. It is just to start discussing how to perform benchmarking tests. Please have a look and feel free to edit.
— Reply to this email directly, view it on GitHub https://github.com/cpignata/powerbench/issues/1#issuecomment-1916777650, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOP5KYZIQ2DVB5IYCMUHUTYRDUBLAVCNFSM6AAAAABCIZTG42VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJWG43TONRVGA . You are receiving this because you are subscribed to this thread.Message ID: @.***>
How deep and detailed should this document eventually go? Right now, it's way too high level and does not help compared to the past drafts and papers on the topic.
From my work on the topic, it became clear there are two related but importantly different objective to "benchmarking"
Which one do we aim to address? 1. is easier, as it is somewhat arbitrary; so you cannot really "get it wrong" but it is not easy to define test scenarios that are useful/applicable to a large set of devices. 2. is harder IMO, because one must make choices about "what to include in the model" and this will change depending on the class of devices that we consider.
Concretely, I've done 2. with a student here. We end up with a power model which is essentially the sum of:
What we worked on is a well-defined procedure to put numbers on each of those parts. Is that what we want this draft to be about?