[RC1 Ch03] NFVI Test Case Requirements

rabi-abdel commented 5 years ago

This issue is to create test cases requirements for NFVI testing.

gkunz commented 5 years ago

I'd like to get this started by volunteering to prepare a first draft (a list) of test case requierements for discussion and alignment. So, if you like, assign this issue to me.

michaelfix commented 5 years ago

@gkunz - #411 RC1 Ch03 assigned to you Georg. Note, we (Mark S., Kyle G., myself) did a lot of work in this space for NFVI test case/requirements. Our work can be found in the Ref Model (RM) chapter 8 and chapter 8 annex. Please review those two sections to see what can be moved to the RC1 Ch03. Please include @markshostak (Mark) @kagreenwell (Kyle) and myself @mf4716 in all emails, or to discuss. We can use GitHub to communicate too. Again, thank you! -Mike

michaelfix commented 5 years ago

For convenience, links to RM Ch8 and RM Ch8 Annex.

gkunz commented 5 years ago

@mf4716 thanks. Based on your comment, I feel there might be a slightly different interpretation of the term "test case requirements". So, just to avoid misunderstandings: I read "test case requirements" in the context of the Reference Compliance 1 document as requirements a test case must fulfill to be usable as a test case in a compliance program. This is a different perspective than "test scope requirements", i.e. what should be tested. The existing documentation (e.g. RM/c8) focuses mostly on the latter.

Anyway, I'll get this started by pushing a simple list of my interpretation of test case requirements for review.

michaelfix commented 5 years ago

perfect @gkunz - thank you

gkunz commented 5 years ago

After detailing my list of test case requirements I believe that those requirements are more suited for RM Ch08. Currently, this chapter is missing a concrete list of requirements which every test case needs to meet in order to be selectable for RC Ch03. In addition, these requirements are generic and mostly independent of NVFI or VNF scope.

In conclusion, I think we should include those in RM Ch08 instead. Your opinion?

Ref #	sub-category	Description
`req.nfvi.test.01`	Test case implementation	All test cases must be implemented in an open source project and publicly available.
`req.nfvi.test.02`	Test case implementation	All test cases must be documented, comprising target reference architecture, test preconditions, basic test flow execution, test assertions, and pass/fail criteria.
`req.nfvi.test.03`	Test case implementation	All test case documentation must be publicly available.
`req.nfvi.test.04`	Test case implementation	At runtime, a test case must not depend on connectivity to external systems or resources (e.g.,of OPNFV, LFN or other).
`req.nfvi.test.05`	Test case implementation	Test cases and test projects should be mature to ensure availability and maintenance of test case implementations over an extended period of time. Criteria for maturity include the release history of the project, a healthy contributer base, (to be extended)
`req.nfvi.test.06`	Test case implementation	Test cases must generate machine and human-readable results (to be specified further in Chapter ??).
`req.nfvi.test.07`	Test case implementation	The execution of a test case must be idem-potent, i.e., test cases handle setup and teardown of all required resources,
`req.nfvi.test.08`	Test case implementation	Test cases must run against a fully deployed and operational system under test, i.e., not isolated individual components.
`req.nfvi.test.09`	Test case implementation	Tests must not require un-merged patches to the relevant upstream projects.
`req.nfvi.test.10`	Test case implementation	If a test case is part of a larger test suite, selection of individual tests for execution must be possible.
`req.nfvi.test.11`	Test case implementation	Tests and test tools must run independently of the method of NFVi platform installation.
`req.nfvi.test.12`	Test case implementation	Tests and test tools must run independently of specific platform components not specified in the corresponding reference archiecture, i.e., allowing different backend implementations such as storage backends or SDN controllers.
`req.nfvi.test.13`	Test case implementation	All test cases must pass on all reference implementations of a particular reference architecture in the OPNFV CI/CD pipeline.
`req.nfvi.test.14`	Test case selection	All test cases must be traceable to a requirement based on the CNTT reference model or a reference architecture.

michaelfix commented 5 years ago

Agree @gkunz , as these are best suited for RM Ch08 as they're covering any RM for compliance. Therefore, the RI-1 Ch03 requirements would be specific to the RI-1 specifications / requirements from RA-1.

gkunz commented 5 years ago

good, I'll create a corresponding issue and a PR for RM ch08.

michaelfix commented 5 years ago

@deebhatia @gkunz - please review/recommend changes to this proposed chapter outline. I'll create a PR Tuesday with any input you provide.

Assumptions: Automatable, Integrated with CICD tool chain
Type of requirement: Bare metal, API, etc
Table showing Profile Catalog
Identify SW Reference
Identify HW Reference
Options Available / Configured
Extensions Available / Configured

deebhatia commented 5 years ago

@mf4716 I don't think I fully understand what we want to target in this chapter. Generic test case selection requirements are already part of RM § 8.2.4. RA1 requirements are getting traced in RC § 4.3. Is this about everything else which is not specified in RA and coming from RI, making it mandatory to be verified for deterministic results? Testing the baremetal is one such requirement.

michaelfix commented 5 years ago

hi @deebhatia - you're correct, in that we include generic criteria in RM Ch 8 which must be met for any Compliance program. For RI1-Ch3, this is about documenting what requirements are needed for NFVI certification Openstack specifically, what assumptions we have about the environment, Openstack, automation, tool chaining, and also profile and HW/SW options and extensions. More specifically, what type of requirement is being certified, for example, Baremetal validations which you highlighted. I'll send you an offline email with the entire RC document outline for context. Its a work in progress, so please recommend changes.