trevgc
commented
4 years ago CNTT objectives with respect to RC workstream includes driving a conformance (badging) program for Cloud Native Infrastructure and CNFs. The conformance program (OVP2.0) specifies three classes of conformance testing.
- CNCF has proposed a CNF conformance program in collaboration with OVP2.0. CNCF CNF conformance aims to provide criteria for "CNF cloud nativeness"
- ONAP is evolving VNF conformance (part of OVP 1.0) to CNF conformance. ONAP CNF conformance aims to provide criteria for "CNF onboarding"
- CNTT RC-2 is developing cloud infrastructure conformance criteria for Cloud Platform adherence to RM and RA-2 requirements. CNTT does not currently specify a model or architectures for CNFs and there are no CNTT specific requirements available for CNF capabilities or features.
RC-2 is the workstream where CNTT develops …
- conformance test strategy
- specification for conformance test infrastructure (lab environment and tools)
- conformance test method specifications
- conformance test-cases which map to RA and RM requirements
- Specifies conformance test criteria that proves a certain capability or feature of the platform exists and behaves as expected
RC-2 requirements are derived from RM and RA-2 which specify infrastructure capabilities which include compute, memory, storage resource capabilities, performance optimization capabilities, monitoring capabilities. Many capabilities/features specified in the RM and RAs impact performance and indeed are included specifically because of their relevance wrt performance and hence TCO (e.g. huge pages). The only way to test certain capabilities are present and working as expected is to actually measure performance even if the threshold to pass is relatively low. Therefore both functional and performance types of testing are needed
Conformance Criteria
Must/shall conformance criteria are testable as pass/fail and ensure requirement are met to ensure minimum thresholds of functional operability and/or performance behavior. This is the focus of RC-2 since this is what will drive a commercially significant badging program.
Should/may conformance criteria may be testable or non-testable and provide recommendations or best-practices for functional operability and/or performance behavior. These criteria and associated tests can be very useful for developing, evaluating or deploying a cloud infrastructure but are not critical to a commercially significant badging program.
Formal definition of conformance: From https://www.nist.gov/itl/ssd/information-systems-group/what-thing-called-conformance "ISO/IEC Guide 2[2] defines conformance (also referred to as conformity) as the fulfillment of a product, process, or service of specified requirements. These requirements are specified in a standard or specification as either part of a conformance clause or in the body of the specification. A conformance clause is a section of a specification that states all the requirements or criteria that must be satisfied to claim conformance to the specification. Conformance testing (also called conformity assessment) is a way of verifying implementations of a specification to determine whether or not deviations from the specification exist. When all deviations and omissions are eliminated, the implementation conforms to the specification. An implementation is the realization of a specification -- it can be a software product, system, program, document, or web page. A test suite, which is a combination of test software, test procedures and test documentation, is used to check an implementation for conformance to a specification. The test software, consisting of a set of test files (i.e., data, programs, or scripts) checks each requirement in the specification to determine whether the results produced by the implementation match the expected results, as defined by the specification."
Functional Testing: Most but not all the existing tests currently being considered for RC-2 are functional tests. Reuse of existing test suites is encouraged however OVP 2.0 is not aiming to replace programs such as Certified Kubernetes which is a "CNCF software conformance program that ensures a vendor’s version of Kubernetes supports the required APIs, as do open source community version". Testing to confirm specific versions of APIs or software components is out of scope of conformance sine this is an unambiguous part of a product specification. The focus of functional testing in conformance is the actual behavior of the system wrt a capability specified in the RM or RA-2.
Performance Testing: The main objective of CNTT performance testing is to understand if a potential BOM is functioning correctly to get expected performance. It requires a set of performance testing tools (open source) that help with the dimensioning of a solution by an operator. This testing is a comparison between the SUT and a CNTT reference implementation with (pass/fail) result. Performance testing for comparisons between commercial implementations is not a goal of CNTT. Performance testing relies on existing benchmarks.
Benchmarking: Benchmarking assessments do not define acceptance criteria or numerical performance requirement. Benchmark testing and Conformance testing intersect when a specific requirement in the software specification is very important, such as a frequently-executed function. Correct execution with the expected result constitutes conformance. The completion time for a single conforming execution, or the number of conforming executions per second are potential Benchmarks. Benchmarks assess a key aspect of the computing environment in its role as the infrastructure for cloud-native network functions. The benchmarks (and related metrics) have been agreed by the Industry and documented in publications of an accredited standards body. As a result, benchmarks are a sub-set of all possible performance metrics. Examples benchmarks include data rate, latency, and loss ratio of various components of the environment, expressed in quantitative units to allow direct comparison between different systems treated as a black box (vendor-independence). Because the demands on a particular system may vary from deployment to deployment, Benchmarking assessments do not define acceptance criteria or numerical performance requirements.
tomkivlin
xmulligan
ulikleber
Notes from April virtual meeting: Platform performance We need a baseline/control implementation to use as the comparison that all other deployments of the RI compare against.
One principle is the RI is self-contained, so can't compare against each other. RI1 performance test wasn't intended to compare to a control deploy, but to understand a potential BOM to get a particular performance. To characterise a particular configuration and use that to estimate the BOM. Performance testing/benchmarking/dimensioning is valuable - should we include in a conformance activity though? RC2 could have two parts a) conformance for badging in OVP, and b) a set of performance testing/benchmarking tools (open source) to help with the dimensioning / BOM of a solution by an operator This is good, but unlikely to be a day-one thing (i.e. do the open source tools exist?) Important to make the distinction between benchmarking (performance of infra/components against well defined metrics - no pass/fail thresholds) and testing/characterisation Need to be clear on terminology used (i.e. benchmarking/performance testing/characterisation/dimensioning)