Guide on Continuous Testing

[yunks128]

Purpose

• The purpose of this pull request (PR) is to introduce and document a comprehensive guide on Continuous Testing. The content covers an overview of continuous testing, a ready-to-use continuous testing plan template, and addresses valuable feedback from the community. The aim is to enhance understanding and implementation of continuous testing in modern software development.
• In this guide, we propose novel and high-impact concepts such as AI generation of tests to save considerable developer time.

Proposed Changes

• [ADD] Added detailed sections on Continuous Testing, including an introduction, phases, and a plan template.
• [ADD] Included a workflow diagram and a table for test specifications to assist in visualizing and customizing the testing approach.
• [ADD] Integrated an example Continuous Testing Plan for better understanding and application.
• [ADD] Incorporated community feedback on usability testing, tool selection, project maturity, priority phases, licensing, and iterative implementation.

Issues

• pull-112 testing -frameworks pull requested

Testing

The changes in this PR have been reviewed and validated through the following steps:

• Reviewed the content for clarity, completeness, and adherence to the proposed format.
• Cross-referenced links to relevant issues to confirm proper linking.
• Checked the workflow diagram for coherence and correctness.

See live preview of changes: https://yunks128.github.io/slim/docs/guides/software-lifecycle/continuous-testing/

[riverma]

Nice progress @yunks128 - some thoughts:

• Organization of Section 2 - can we be more clear about Robot framework application to Unit or System Tests? I think merging the content in 2.2 into 2.1.2 would make sense, and perhaps just having the following tree structure:

  Writing Tests
  - Unit Tests
  - System Tests

• The LLM landscape is evolving rapidly. Can we write Section 2 in a more general way that ensures it stays up-to-date easily? Some ideas:
  • Make recommendations to use LLMs but let the user pick the LLM model that's best suited to their use case.
  • Where do non-open models like ChatGPT fit in? People are going to ask this. I think for open source code, they could be useful. Perhaps you mention or link to a table of non-open models for open source code bases.
• Let's finish up Section 3 and 4 before making this an open PR

[riverma]

@yunks128 - let's work with @drewm-jpl on the unity-sps repository to test out your latest guide and see where we can add value.

I'm thinking at a minimum, we could work through your guide and propose a PR with:

• A TESTING.md file that outlines the testing approach
• Augmentation / additions to existing test cases using your recommended techniques
• Validation that the team is using our best recommended automation strategies from your guide

[yunks128]

@yunks128 Next steps for unity-sps use case for continuous testing:

1. Follow the continuous testing best practice guideline to generate TESTING.md test plan, unit test & system test code, and pre-commit hooks. Gather information from existing test code (https://github.com/yunks128/unity-sps/tree/develop/unity-test)
2. Gather high-level testing descriptions from @drewm-jpl and generate tests using continuous testing guide recommendations (i.e. LLMs) and compare results against @drewm-jpl's hand-written tests for validation.
3. Gather testing automation specifics (i.e. pre-commit hooks, gh actions) @drewm-jpl has implemented for unity-sps, and infuse recommendations back into continuous testing guide where applicable

Refer to: unity-sps unity-sps api test

[sjlewis-jpl]

I would suggest explicitly mentioning docstrings for test functions, with guidance on what to include. In my experience, these are often neglected in test code, but are no less important. Here's some suggestions for what to include in test function docstrings:

• Describe the purpose of each test (rather than relying on interpreting the test function's name, or reading through the test code itself). For example: """Test for when the direction vector has zero magnitude."""
• Say where to find the function being tested (might be more appropriate for a test module docstring, or class docstring). Not always necessary, but can be useful, especially if there's inherited methods, or if tests for multiple modules are combined into a single test file. For example:

  
  class ProjectPoint(unittest.TestCase):
  """
  Unit tests for the `projectPoint` function in the `computeTime.py` module.
  """
  def test_projectPoint_2D(self):
      """Test with 2D vectors."""

- When related to a bug fix, call this out (along with a ticket ID, if applicable).  See example below.
    - Other relevant IDs can be included, such as for Change Requests, Requirements (if this test is used as a verification), Anomaly Reports, etc.

    """
    Tests to prove a bug fix in the function (ref BUG-REPORT-1234).
    This function should raise an error if the two input vectors are the same.
    Previously, it judged this by summing the elements of the difference vector.
    Now the function computes the magnitude of the difference vector, and 
    only raises an error if that magnitude is zero.
    This test calls the function with vectors that used to erroneously return errors,
    and will demonstrate that the errors are no longer raised.
    """

- 

[riverma]

I would suggest explicitly mentioning docstrings for test functions, with guidance on what to include. In my experience, these are often neglected in test code, but are no less important. Here's some suggestions for what to include in test function docstrings:

• Describe the purpose of each test (rather than relying on interpreting the test function's name, or reading through the test code itself). For example: """Test for when the direction vector has zero magnitude."""
• Say where to find the function being tested (might be more appropriate for a test module docstring, or class docstring). Not always necessary, but can be useful, especially if there's inherited methods, or if tests for multiple modules are combined into a single test file. For example:

class ProjectPoint(unittest.TestCase):
    """
    Unit tests for the `projectPoint` function in the `computeTime.py` module.
    """
    def test_projectPoint_2D(self):
        """Test with 2D vectors."""

• When related to a bug fix, call this out (along with a ticket ID, if applicable). See example below.

  • Other relevant IDs can be included, such as for Change Requests, Requirements (if this test is used as a verification), Anomaly Reports, etc.

        """
        Tests to prove a bug fix in the function (ref BUG-REPORT-1234).
        This function should raise an error if the two input vectors are the same.
        Previously, it judged this by summing the elements of the difference vector.
        Now the function computes the magnitude of the difference vector, and 
        only raises an error if that magnitude is zero.
        This test calls the function with vectors that used to erroneously return errors,
        and will demonstrate that the errors are no longer raised.
        """

This is a great suggestion @sjlewis-jpl. Good practice to help understand the test written beyond just having it. Definitely critical to assess if tests cover the intended scope.

@yunks128 - what do you think? Could we add modifications to prompts or a blurb about inline comments in tests?

[riverma]

@yunks128 some comments / questions from the recent talk you gave on this PR:

• How can we validate the security aspects of generated test code?
• Is the code checked for quality? How can the quality be improved?
• How can the LLM test code generation be interrupted and improved if its writing tests in the wrong way?

[yunks128]

I would suggest explicitly mentioning docstrings for test functions, with guidance on what to include. In my experience, these are often neglected in test code, but are no less important. Here's some suggestions for what to include in test function docstrings:

• Describe the purpose of each test (rather than relying on interpreting the test function's name, or reading through the test code itself). For example: """Test for when the direction vector has zero magnitude."""
• Say where to find the function being tested (might be more appropriate for a test module docstring, or class docstring). Not always necessary, but can be useful, especially if there's inherited methods, or if tests for multiple modules are combined into a single test file. For example:

class ProjectPoint(unittest.TestCase):
    """
    Unit tests for the `projectPoint` function in the `computeTime.py` module.
    """
    def test_projectPoint_2D(self):
        """Test with 2D vectors."""

• When related to a bug fix, call this out (along with a ticket ID, if applicable). See example below.
• Other relevant IDs can be included, such as for Change Requests, Requirements (if this test is used as a verification), Anomaly Reports, etc.

        """
        Tests to prove a bug fix in the function (ref BUG-REPORT-1234).
        This function should raise an error if the two input vectors are the same.
        Previously, it judged this by summing the elements of the difference vector.
        Now the function computes the magnitude of the difference vector, and 
        only raises an error if that magnitude is zero.
        This test calls the function with vectors that used to erroneously return errors,
        and will demonstrate that the errors are no longer raised.
        """

This is a great suggestion @sjlewis-jpl. Good practice to help understand the test written beyond just having it. Definitely critical to assess if tests cover the intended scope.

@yunks128 - what do you think? Could we add modifications to prompts or a blurb about inline comments in tests?

@sjlewis-jpl I appreciate your suggestion! I will do my best to include inline comments in the test codes for the continuous testing guide. @riverma

[yunks128]

I would suggest explicitly mentioning docstrings for test functions, with guidance on what to include. In my experience, these are often neglected in test code, but are no less important. Here's some suggestions for what to include in test function docstrings:

• Describe the purpose of each test (rather than relying on interpreting the test function's name, or reading through the test code itself). For example: """Test for when the direction vector has zero magnitude."""
• Say where to find the function being tested (might be more appropriate for a test module docstring, or class docstring). Not always necessary, but can be useful, especially if there's inherited methods, or if tests for multiple modules are combined into a single test file. For example:

class ProjectPoint(unittest.TestCase):
    """
    Unit tests for the `projectPoint` function in the `computeTime.py` module.
    """
    def test_projectPoint_2D(self):
        """Test with 2D vectors."""

• When related to a bug fix, call this out (along with a ticket ID, if applicable). See example below.
• Other relevant IDs can be included, such as for Change Requests, Requirements (if this test is used as a verification), Anomaly Reports, etc.

        """
        Tests to prove a bug fix in the function (ref BUG-REPORT-1234).
        This function should raise an error if the two input vectors are the same.
        Previously, it judged this by summing the elements of the difference vector.
        Now the function computes the magnitude of the difference vector, and 
        only raises an error if that magnitude is zero.
        This test calls the function with vectors that used to erroneously return errors,
        and will demonstrate that the errors are no longer raised.
        """

This is a great suggestion @sjlewis-jpl. Good practice to help understand the test written beyond just having it. Definitely critical to assess if tests cover the intended scope. @yunks128 - what do you think? Could we add modifications to prompts or a blurb about inline comments in tests?

@sjlewis-jpl I appreciate your suggestion! I will do my best to include inline comments in the test codes for the continuous testing guide. @riverma

@sjlewis-jpl Thanks again! Added the following:

We recommend adding inline comments in your tests to clarify the purpose of each test. These comments should include details on the function being tested, the test type (e.g., bug fix, change request, requirements validation, anomaly reports), and any relevant context. For example:

import unittest
from computeTime import projectPoint  

class TestProjectPoint(unittest.TestCase):
    """
    Unit tests for the `projectPoint` function in the `computeTime.py` module.
    """
    def test_projectPoint_2D(self):
        """
        Purpose: Test the `projectPoint` function with 2D vectors to ensure it
        correctly handles the case where the input vectors are the same.

        Function: `projectPoint(vector1, vector2)`

        Test Type: Bug fix validation (ref BUG-REPORT-1234)

        Description: This test validates that the function no longer raises
        an error when the input vectors are the same. Previously, it incorrectly
        raised an error by summing the elements of the difference vector. The 
        function has been fixed to compute the magnitude of the difference vector 
        and only raise an error if that magnitude is zero. This test provides
        input vectors that used to erroneously return errors and checks that
        the errors are no longer raised.
        """

        # Test case with 2D vectors that are the same
        vector1 = [1, 2]
        vector2 = [1, 2]

        # Call the function and assert that it does not raise an error
        try:
            projectPoint(vector1, vector2)
        except ValueError:
            self.fail("projectPoint() raised ValueError unexpectedly with identical vectors")

        # Test case with 2D vectors that are different
        vector1 = [1, 2]
        vector2 = [2, 3]

        # Call the function and assert that it completes successfully
        try:
            projectPoint(vector1, vector2)
        except ValueError:
            self.fail("projectPoint() raised ValueError unexpectedly with different vectors")

if __name__ == '__main__':
    unittest.main()

[yunks128]

@riverma This is quite strange! The link is not broken! The link works well online as well as locally. I updated the docusaurus.config.js file to set the onBrokenLinks configuration to 'warn' to bypass that error. Would this be ok? https://yunks128.github.io/slim/docs/guides/software-lifecycle/continuous-testing/testing-frameworks/#for-test-code-generation

[riverma]

@riverma This is quite strange! The link is not broken! The link works well online as well as locally. I updated the docusaurus.config.js file to set the onBrokenLinks configuration to 'warn' to bypass that error. Would this be ok? https://yunks128.github.io/slim/docs/guides/software-lifecycle/continuous-testing/testing-frameworks/#for-test-code-generation

Hey @yunks128 - we did in fact have a broken link. Search for the "Here's our recommended approach to deciding the right model for you to use (see our full list of recommended code generation models):" text within the guide, as below.

I think we want to keep broken hyperlinks as build failures. Otherwise the SLIM website will be littered with bad links. Could you revert your change to docusaurus.config.js?