Unclarity about the evaluation environment

[zaheduzzaman]

While section 5 expects to do well-round of evaluations, it does not specify where or how to do the evaluation. Does the congestion control proposals provide the evaluation results in a simulator or in a real lab setup or in a controlled part of the Internet or over the Internet? While it is understandable that an accurate description on the evaluation environment is tough to define but the specification should at least mention what is good enough environment to evaluation on, or state it does not really matter.

[gorryfair]

See below:On 27 May 2024, at 10:50, Zaheduzzaman Sarker @.***> wrote: While section 5 expects to do well-round of evaluations, it does not specify where or how to do the evaluation. Does the congestion control proposals provide the evaluation results in a simulator or in a real lab setup or in a controlled part of the Internet or over the Internet? While it is understandable that an accurate description on the evaluation environment is tough to define but the specification should at least mention what is good enough environment to evaluation on, or state it does not really matter.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***> I am not sure how we add guidance … surely all of these method can be appropriate (more so for some of the evaluations) when bringing new work to the IETF. When last calling a draft - it would be good to see more than one approach where possible. Simulation ought to be better in understanding stability and identifying worst case impact; real lab or controlled tested could be good for understanding typical interactions with other traffic. I still wonder what we can best advise?Gorry

Postscript: Reading again: my comment is about testing specific new CC mechanisms, if we're talking about a new CC algorithm (not just a new technique), then I agree with Martin below.

[martinduke]

This is covered in Section 3, which is becoming Section 2:

Congestion control algorithms without empirical evidence of Internet-scale deployment SHOULD seek Experimental status.

Congestion control algorithms with a record of measured Internet-scale deployment MAY directly seek the Standards Track if there is solid data that reflects that it is safe, and the design is stable, guided by the considerations in Section 6. However, the existence of this data does not waive the other considerations in this document.

Is that good enough?

[zaheduzzaman]

I have noticed that section. I think it is great that we are somewhat defining the criteria for a PS. But still we are not giving any hint on - say, if simulaiton results are good enough to be considered as experimental or shall we at least want to see some controlled testbed evaluations, or put it like this - what is considered as bare minimum empirical evidence? I would be great if we can put some clarity here.

[gorryfair]

I think we ought to expect some experiment results ... but how detailed, rather depends on what is being proposed. Having some experiments (even at large scale) and no simulation of theory (e.g. on how a method treats overload) would also seem very dangerous to me. These have always been the case in the past - i.e. TSVWG has pushed back to ICCRG to see experimental results AND some simulation/analysis when it comes to CC.

[zaheduzzaman]

@gorryfair are you saying we are just fine with some expection on evidence of experiments without specifying what is the minimum required environ to test in? we do have test cases defined for new algorithm testing, can't we at least say get inspiration or use them in the experiments?

[gorryfair]

Aha - I think you're suggesting we provide examples (rather than requirements) and we could refer to RFCs? If so, I'll try a PR.