search

ApexAI / apex_rostest

Framework for ROS2 Integration Testing
5 stars 6 forks source link

Improve handling of skipped tests #41

Closed pbaughman closed 5 years ago

pbaughman commented 5 years ago

Resolves issue https://github.com/ApexAI/apex_rostest/issues/40

Merge Order

Fixes the junit XML to match the XML generated by pytest more closely. Specifically, using 'classname' and 'name' attributes in TestCase and setting the 'message' attribute in the skip element for skipped tests.

This PR also introduces a feature allowing the @unittest.skip() decorator to be put on a generate_launch_description function to skip the whole launch.

dejanpan commented 5 years ago

@pbaughman how do I test it? do I install ros2 from master?

pbaughman commented 5 years ago

@dejanpan Just replace our internal apex_launchtest package with the github version and build

dejanpan commented 5 years ago

I tested this as follows:

apex_launchtest tests:

  1. colcon build against ros2 crystal
  2. colcon test --packages-select apex_launchtest => check

using skip message in good_proc.test.py DUT:

  1. apex_launchtest src/test/apex_rostest/apex_launchtest/examples/good_proc.test.py --junit-xml result.xml

  2. xmllint --format result.xml:

     <?xml version="1.0" encoding="us-ascii"?>
<testsuites errors="0" failures="0" name="apex_launchtest" tests="4">
  <testsuite errors="0" failures="0" name="good_proc.test.py" skipped="0" tests="4" time="4.079">
    <testcase classname="TestProcessOutput" name="test_out_of_order" time="0.001"/>
    <testcase classname="TestGoodProcess" name="test_count_to_four" time="4.078"/>
    <testcase classname="TestProcessOutput" name="test_exit_code" time="0.0"/>
    <testcase classname="TestProcessOutput" name="test_full_output" time="0.0"/>
  </testsuite>
</testsuites>

  3. Apply this patch; https://gist.github.com/dejanpan/ff25f18e4e889053d32e0d0d7d13d3db

  4. apex_launchtest src/test/apex_rostest/apex_launchtest/examples/good_proc.test.py --junit-xml result.xml

  5. xmllint --format result.xml:

    <?xml version="1.0" encoding="us-ascii"?>
<testsuites errors="0" failures="0" name="apex_launchtest" tests="0">
  <testsuite errors="0" failures="0" name="good_proc.test.py" skipped="1" tests="0" time="0">
    <testcase classname="__skipped_test_case" name="skipped_launch" time="0">
      <skipped message="Foo test test desc"/>
    </testcase>
  </testsuite>
</testsuites>