[RC1] Types of test cases needed

[rabi-abdel]

For each test case, we need to clearly specify if it a:

• Must (has to pass to get a badge)
• Should (supposed to pass, but a failure will be OK for this release - as it relates to an exception)
• May (Optional to meet this requirement).

we need to be very specific of how the test output should be as to reflect on CNTT requirements.

[rabi-abdel]

@lylavoie Please feel free to add more details

[collivier]

@rabi-abdel : I don't understand this topic which echoes with latest RI/RC meeting. It misleads between Iaas Verification and CNTT Compliance.

All tests have been selected and tuned according to RA1 and then are only covering MUST requirements (mostly RA1 chapter 5). All single test covering an optional service (should or may) or an optional service capability has been removed in CNTT RC (see regex) and then all Functest CNTT containers.

If you plan to add the classical Functest containers in CNTT, we should create an additional chapter covering IaaS verifcation. From my understanding it's nothing specific to CNTT (or OVP) and it has driven Functest for years.

As already mentioned, there are a few error between the requirements in RM and RA. If a service becomes optional for any reason, it would be automatically removed from RC. No additional tag is requested to do so, it's about classical bugs in collaborative work.

This topic may lead lots of side effects towards OPNFV and it will be great to quickly dive into RC. It makes sens to discuss in TSC meeting. All sparsed mails or meetings seem not successful here.

[rabi-abdel]

Hi @collivier I have a feeling there is miss-understanding. I will send a calendar invite to talk about this. I hope early next week will be OK for it.

[trevgc]

There are two types of NFV Infrastructure testing to consider. These comments do not pertain to VNF testing which is a third category of testing. Certain test-cases may be applicable to both types of tests however other test-cases may only be appropriate to one or the other test category.

1) Testing to validate an RI (i.e. CNTT instantiation of an RA). The testing approach is similar to that which would be undertaken when developing a commercial NFVi. The goal is to provide high confidence that the RI is a "good reference" i.e. it implements all the RA requirements correctly and optimally. This means maximum test coverage using appropriate existing test cases and developing new tests where there are gaps. The tests should be part of a devops CI environment and results interpreted by developers. The tests may be useful for vendors to reuse when developing commercial NFVI products. We can consider this as testing from a bottoms up perspective i.e. tests are designed with a good understanding of the underlying hardware technologies and software stack implementation. We know from OPNFV experience that this type of testing is huge and can take considerable time (hours, days or even weeks).

2) Testing for NFVI conformance (i.e. as part of a badging program with defined set of pass/fail criteria). The testing approach here is similar to that which would be undertaken for purpose of preparing to deploy a VNF. The goal is to provide high confidence that the RI will support performance and capacity requirements of a VNF i.e. provides sufficient resources and capabilities to make forecasting performance/capacity possible and to avoid unpleasant surprises of VNF behavior once deployed. The tests will typically be run by an operator who wants to check the infrastructure meets TCO or performance/capacity claims for supporting a VNF. We can consider this as testing from a tops down perspective i.e. tests are designed without needing to understand the underlying hardware technologies and software stack implementation. What really matters here is that 1) the test methodology replicates as closely as possible the actual deployment of a VNF and 2) measures from the perspective of the VNF (for example ... can the infrastructure support xyz capability (say) packet throughput without any regard to how this is achieved "under the hood"?).

[trevgc]

@acmacm @LucProvoost @opensource-tnbt @lylavoie ... comments will be appreciated.

[trevgc]

With respect to Testing for NFVI conformance. By definition NFVI testing must be independent of using a real VNF as part of the SUT. The way an infrastructure is consumed will be different for each VNF e.g. many "small" VNFs or one very "large" VNF may in the end consume similar resources and provide similar performance/capacity (or not). What we are trying to do is test the overall capacity of the underlying infrastructure in as realistic a way as is possible. It will not be perfect but we have to start somewhere and could do worse than to select a few configurations to see if the behavior / capacity changes drastically. Perhaps we start with a mid-size VM that consumes (say) 4 cores and then load the system with (say) 1 VNF, 10 VNFs, 20 VNFs. Each VNF (or VNF pair) does exactly the same thing. For RA1 a VNF is a VM running some appropriate workload … some may call this a "golden VNF". We will need at least one golden VNF for each profile i.e. Network intensive, Compute intensive, …

[pgoyal01]

@trevgc "We will need at least one golden VNF for each profile i.e. Network intensive, Compute intensive, …" or a VNF with VNFCs that require different profiles"

[rabi-abdel]

based on the discussion during the meeting on 23rd March 2020, this will be moved to backlog and it will be implemented when we start having exceptions. the technical approach of doing it can either be a tool to post-process or manually by reviewers.

[trevgc]

@rabi-abdel please clarify ... what is moved to the backlog? If the meeting referenced on 23rd was RA-1, the minutes don't clarify. AFAIK this issue is to discuss the types of test cases needed to guide RC and that is needed.