Metamorphic Testing: Testing the Untestable

[SigmaJahan]

Metamorphic testing is not determined by checking an individual concrete output, but by applying a transformation to a test/source input and observing how the program output “morphs” (transforms) into a different one as a result, rather than as mappings of specific inputs to specific outputs. The main idea is to test a software system by checking whether each identified metamorphic relation (MR) holds among several executions. Metamorphic relations are derived from the innate characteristics of the software under test, which can help generate test data and verify the correctness of the test result without the need for a test oracle. MR should be diverse and involve different input parameters and input constraints to exercise the program under test as thoroughly as possible. A given source test case can be transformed into one or more follow-up test cases such that the metamorphic relation can be checked Three steps for MT:

1. Identifying Metamorphic relation- requires creativity and domain knowledge. There are two common approaches: One is Input driven approach- involves thinking about how certain changes in the program’s inputs may be expected to produce certain changes in the program’s outputs, another one is Output-Driven Approach- starting from possible relations among the outputs, typically found in the target domain and then thinking about what kind of changes in the program’s inputs would lead to satisfaction of the expected relation among outputs.
2. Implementation (source test case and follow-up test case)- Each metamorphic relation can be typically instantiated into many metamorphic tests, by using different test data.
3. Generating automated test case- generating inputs as well as outputs for full test automation Application of MT in Compiler: GraphicsFuzz for testing graphics drivers by image Application of MT in AI: DeepTest autonomous driving using images Limitations: -MR cannot be used to tell whether or not the output of a program is the expected one thus it is not suitable for testing critical systems that require the correctness checking of each output
   • Identifying MR is mostly a manual process ( automated MR only for numerical programs)
   • the number of MR is potentially huge hence the selection is difficult as well Future work: New application domains and automated inference of metamorphic relations, tools, and integration with other testing techniques

[SigmaJahan]

https://towardsdatascience.com/metamorphic-testing-of-machine-learning-based-systems-e1fe13baf048

• Better for ML-based testing which is more probabilistic and black-box in nature -A more advanced approach to generate input transformations is to apply different ML-based techniques to build the transformations themselves:
  1. GAN-based Transformations with MMT: GAN (Generative Adversarial Networks) for image augmentation ( eg: Autonomous Vehicle- transforming one normal car image to generate snow/smoke/foggy etc images)
  2. LIDAR Transformation: adding noise around lidar cloud point for transformation out ROI area
  3. Metamorphic Testing of Machine Translation: automated language translations with ML-based NLP techniques using Bert for google translation Test selection in MT is important, One approach is greedy search strategy- iteration of three steps with defined thresholds/ or number of experiments for the measuring neuron activation coverage