Similar to how we have conformance tests, where a single set of tests validate functionality against common behavior of components, there should also be performance tests against components and have those as part of the certification process. In other words, stable components should also pass the basic set of performance tests.
Because performance can be impacted by limitations of each component, the performance tests should have a threshold per component and not a global setting.
Describe the proposal
Similar to how we have conformance tests, where a single set of tests validate functionality against common behavior of components, there should also be performance tests against components and have those as part of the certification process. In other words, stable components should also pass the basic set of performance tests.
Because performance can be impacted by limitations of each component, the performance tests should have a threshold per component and not a global setting.