Methodology proposal for the QA test plan

[j-rivero]

Together with @mabelzhang we have been working on establish the methodology for implement the QA test plan. The proposal here is a based on three concepts: areas of testing, risks associated and tests to cover the check for all the risks. There is an example at the end of how it should work with just one case. Part of #23

Note: Areas section has an initial work on defining the real areas, probably help to understand the whole methodology

Feedback or comments are more than welcome.

QA Test Plan Methodology

State: draft/brainstorm Methodology: identify the different areas for testing that compose the whole demo, identify the risks associated with the different areas and finally create a series of tests that can cover all the risks detected in the steps before.

1. Areas of testing:

Try to divide the global demo into different areas so it is easier to check if checks are covering all expected features.

(initial brainstorm)

• Setup
  • S1 Login in RoboMaker
  • S2 Launch the simulation in RoboMaker
  • S3 Spawn robot onto Orchard level
  • S4 Camera view is third person
  • S5 Launch nav2 stack
  • …
• Navigation
  • N1 Have a map of gathering points in world poses (position and orientation)
  • N2 Navigate the robot to a gathering pose (position and orientation) using nav2 framework via RVIZ
  • …
• Manipulation
  • M1 Generate a list of all reachable apples
  • M2 Trigger apple picking; camera view should change (if desired)
  • M3 Move the manipulator to a picking pose (position and orientation) popped from the queue
  • M4 Attempt to pick apple until successful, or timeout
  • M5 Spawn crate if needed
  • M6 Move the manipulator to the storage position and orientation)
  • M7 Store apple into crate
  • M8 All reachable apples have been attempted or gathered
  • …
• Demo logic
  • D1 Global state machine
  • D2 Navigation orchestration
  • D3 Apple gathering state machine
  • D4 Count of apples picked / crate count
  • …
• Global
  • G1 Networking bandwidth
  • G2 Simulation performance
  • G3 Communications between ROS and simulation
  • G4 GUI buttons
  • …
• …

2. Risks associated:

Analysis of the risks associated with every area of testing.

(template)

Field	Descripton
ID:	R- + unique identifier
Areas of testing associated	Which areas of testing are impacted by the risk. Could be one or more
Description	Short description of the risk detected
Type of risk	Determine the consequences of the risk: critical, medium, low
Impacted by changes in	What changes in the demo setup and/or development can affect the possibility of the risk
Mitigation measures	Which possible actions can be taken to mitigate or solve the possible problem derived by the risk

3. Tests to cover the check for all the risks:

Set of tests that describe the actions needed to check if any of the possible risks have derived in an issue or a problem.

(template)

Field	Descripton
Name and ID	Descriptive short name and “T + unique identifier”
Risk associated	One or more risk identifiers that the test is trying to check
Type	What kind of testing is expected by the test: smoke testing (quick check), …
Purpose of the testing	Describe what the test is trying to accomplish
Duration of the testing	Approximated in minutes
Assumptions	Initial considerations and expected initial conditions at the beginning of the test
Scope of the testing	What the test is covering and what is not
Execution of the test	Inputs: step by step inputs expected to run the test
	Expected outputs: what is the expected outputs from each of the steps (if any), what is the criteria to consider a problem,..

---

Example of risk and test associated:

A brief example with a random area, a given risk to that area and a test to cover partially that risk.

Areas:

• Global
  • G1 Networking bandwidth
  • G2 Simulation performance
  • … ...

Risks analysis:

Field	Descripton
ID	R-SIM-PERFORMANCE
Areas of testing associated	G2 simulation performance
Description	Impacts on the simulation performance can be critical for the successful run of the demo
Type of risk	critical
Impacted by changes in	3D models: scene design, vehicle design.
	Software changes: new code introduced and changes in current code.
	Changes in hardware: changes in any component of the computer + screen, display resolution, etc.
Mitigation measures	Revert the changes in code or models, change in the hardware or the setup of the hardware, etc.

Test:

Field	Descripton
Name and ID	Quick performance check (T-QUICK-SIM-PERF)
Risk associated	R-SIM-PERFORMANCE
Type	smoke-test
Purpose of the testing	Really fast check that the performance of the whole simulation is the expected
Duration of the testing	5 min
Assumptions	Robomaker is up and prepared to spawn the simulation
Scope of the testing	Check O3DE FPS throughput under expected initial conditions. Consider pure simulation performance not the user experience
Execution of the test	Step 1: Start the simulator, wait for the simulation to begin, wait extra 20 seconds.
	:white_large_square: FPS in o3de should display no less than X FPS
	Step 2: Move the vehicle to the first nav point (elaborate)
	:white_large_square: During the movement of the vehicle FPS in o3de should display no less than X FPS

[j-rivero]

No problems found during last week review.