RIOT-OS / RobotFW-tests

Includes tests for RIOT based on the Robot Framework
GNU Lesser General Public License v2.1
4 stars 13 forks source link

dist/robotframework: Expose ROBOT_EXTRA_ARGS in build system #109

Closed MrKevinWeiss closed 3 years ago

MrKevinWeiss commented 3 years ago

Add ROBOT_EXTRA_ARGS env variable to have control over robot args. This allows things like specific test selection when running. The ROBOT_ARGS will remain the same and only add args to the default call. Note that setting ROBOT_ARGS will no longer include ROBOT_EXTRA_ARGS.

Testing

A plain build should remain the same

make robot-clean robot-test -C dist/tools/output_to_xunit/
``` make: Entering directory '/home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit' python3 -m robot.run \ --name "example" --settag "APP_example" --settag "BOARD_host" --metadata RIOT-Version: --metadata RIOT-Board:host --metadata RIOT-Application:example -P "/tests:/home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit/../../../dist/robotframework/lib:/home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit/../../../dist/robotframework/res" -l NONE -o /home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit/../../../build/robot/host/example/output.xml -r NONE \ tests/ ============================================================================== example ============================================================================== example.Example Test ============================================================================== Passed Testcase | PASS | TEST PASSED ------------------------------------------------------------------------------ Failed Testcase | FAIL | TEST FAILED ------------------------------------------------------------------------------ Fail With Nice Message | FAIL | This is my failure message, much better that True==True ------------------------------------------------------------------------------ Record Empty value | PASS | ------------------------------------------------------------------------------ Record String value | PASS | ------------------------------------------------------------------------------ Record Long List | PASS | ------------------------------------------------------------------------------ Record Long Dict | PASS | ------------------------------------------------------------------------------ Record using keyword | PASS | ------------------------------------------------------------------------------ [ WARN ] Executing keyword 'Fail' failed: It is OK to, this will show PASS and have a WARN tag here or there Assert Test Fails Warning | PASS | ------------------------------------------------------------------------------ Skip Test | SKIP | I am skipping this test ------------------------------------------------------------------------------ Skip If Test | SKIP | I am skipping this test ------------------------------------------------------------------------------ Do Not Skip Test | PASS | ------------------------------------------------------------------------------ example.Example Test | FAIL | 12 tests, 8 passed, 2 failed, 2 skipped ============================================================================== example.Pass :: Simple robot test that passes. ============================================================================== Assert Test Passes | PASS | ------------------------------------------------------------------------------ Assert Test Passes Again | PASS | ------------------------------------------------------------------------------ example.Pass :: Simple robot test that passes. | PASS | 2 tests, 2 passed, 0 failed ============================================================================== example.Error Msg :: Simple robot test that passes. ============================================================================== Should Mock Success | PASS | ------------------------------------------------------------------------------ Should Not Mock Success | FAIL | Mock Return Success expected FAIL but received Success ------------------------------------------------------------------------------ Should Not Mock Key Error | FAIL | Mock Key Error does not have a result, return = {'foo': 'Success'} ------------------------------------------------------------------------------ Should Not Mock Type Error | FAIL | Mock Type Error does not have a result, return = Success ------------------------------------------------------------------------------ Should API Call Repeat on Timeout Success | PASS | ------------------------------------------------------------------------------ Should API Call Repeat on Timeout Error | FAIL | Mock Return Timeout failed after 16 times with {'result': 'Timeout'} ------------------------------------------------------------------------------ example.Error Msg :: Simple robot test that passes. | FAIL | 6 tests, 2 passed, 4 failed ============================================================================== example | FAIL | 20 tests, 12 passed, 6 failed, 2 skipped ============================================================================== Output: /home/weiss/wd/RobotFW-tests/build/robot/host/example/output.xml make: [/home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit/../../../dist/robotframework/Makefile.include:21: /home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit/../../../build/robot/host/example/output.xml] Error 6 (ignored) python3 /home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit/../../../dist/tools/output_to_xunit/output_to_xunit.py \ --output /home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit/../../../build/robot/host/example/xunit.xml \ /home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit/../../../build/robot/host/example/output.xml make: Leaving directory '/home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit' ```

Adding an arg allows us to only run one test suite

ROBOT_EXTRA_ARGS="-s pass" make robot-clean robot-test -C dist/tools/output_to_xunit/
``` make: Entering directory '/home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit' python3 -m robot.run \ --name "example" --settag "APP_example" --settag "BOARD_host" --metadata RIOT-Version: --metadata RIOT-Board:host --metadata RIOT-Application:example -P "/tests:/home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit/../../../dist/robotframework/lib:/home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit/../../../dist/robotframework/res" -l NONE -o /home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit/../../../build/robot/host/example/output.xml -r NONE -s pass \ tests/ ============================================================================== example ============================================================================== example.Pass :: Simple robot test that passes. ============================================================================== Assert Test Passes | PASS | ------------------------------------------------------------------------------ Assert Test Passes Again | PASS | ------------------------------------------------------------------------------ example.Pass :: Simple robot test that passes. | PASS | 2 tests, 2 passed, 0 failed ============================================================================== example | PASS | 2 tests, 2 passed, 0 failed ============================================================================== Output: /home/weiss/wd/RobotFW-tests/build/robot/host/example/output.xml python3 /home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit/../../../dist/tools/output_to_xunit/output_to_xunit.py \ --output /home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit/../../../build/robot/host/example/xunit.xml \ /home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit/../../../build/robot/host/example/output.xml make: Leaving directory '/home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit' ```

Overwriting args ignores extra args and only runs the set args (this should fail)

ROBOT_EXTRA_ARGS="-s pass" ROBOT_ARGS="--dryrun" make robot-clean robot-test -C dist/tools/output_to_xunit/
``` ROBOT_EXTRA_ARGS="-s pass" ROBOT_ARGS="--dryrun" make robot-clean robot-test -C dist/tools/output_to_xunit/ make: Entering directory '/home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit' python3 -m robot.run \ --dryrun \ tests/ ============================================================================== Tests ============================================================================== [ ERROR ] Error in file '/home/weiss/wd/RobotFW-tests/dist/tools/output_to_xunit/tests/01__example_test.robot' on line 4: Resource file 'util.keywords.txt' does not exist. ... ```