CallopEO.py testing

[rzbrk]

Currently, there is no straight forward way to test CalliopEO.py. Unfortunately, the script or functions are not object-oriented which makes the usage of tools like unittest a little tricky.

I would recommend to establish a bash script for testing, which we could name testing.sh. This script itself can source a couple of files in a subdirectory. The very first file to source could be a config file (00_config.sh) to set up any bash variable needed for testing. For every test case, we can establish a dedicated file NN_testcase_name.sh, where NN is a number defining the order in which the test cases are executed and testcase_name can be replaced by a short name for the test case.

The test cases can be:

• Run script without Calliope Mini
• Run script with Calliope Mini attached but with no ZIP archive in folder
• Run script with Calliope Mini attached but with invalid ZIP file
• ...

Each bash file for a test case should contain the following:

• Instructions for the test person to establish a proper set-up for the test (e.g. connect or disconnect the Calliope Mini, copy ZIP file to directory). This instructions are basically echo commands.
• Bash instructions to set-up the rest of the environment
• Bash instructions to execute the CalliopEO.py script
• Bash instructions to test the results
• Bash instructions to create a small report for the test case, most important the result for the particular test case

What are your thoughts?

[rzbrk]

Testing is still work in progress. I do development in the feature branch testing of my fork.

[Amerlander]

I just did a single test file with the following cases in one file:

• Run Script without Calliope => Error no Calliope found
• Run Script without Zip Files => Error no Zip files found
• Run with only corrupted Zip file (and so no Hex file at all) => renamed to *.failed & Error no files found
• Run with the following files
  • Corrupted Zip File => will be renamed to *.failed (and overwrite the existing one)
  • Zip file with .garbage File inside => The last flashed script persists on the mini and will be logged again / if 'no-start.hex" was the last one it will behave like that.
  • Zip file with invalid .hex File => the last flashed script persists on the mini and will be logged again / if 'no-start.hex" was the last one it will behave like that
  • Hex file which runs 10s and then stops => regular execution
  • Hex file which runs 900s and then stops => result in a timeout in this test
  • Hex file which logs a lot of data => run into the filesize limit in this test
  • Hex file which send serial Data, but does not send a start or end tag. => This file will be reflashed 5 times and then skipped

I made the tests at first for checking the testcases and improve the manual tests on the astropi, so unlike in your tests in your fork there is no automated proof and it is all in a single test file.

[Amerlander]

I just had one issue with the testing.sh: It doesnt read the testing.conf and also in line 24 for testcase in ${tc_folder%/}/*.*; do, it fails to read this as phat and results in /*.* instead of ./testcases/*.*. So hard coding it to for testcase in ./testcases/*.*; do worked.

[rzbrk]

Mmh, strange ...

This only happens on my system, if executing testing.sh from outside the directory testing.sh is in. Is this the case for you? I made a change to ensure the script is executed from the right directory, see #26.

[rzbrk]

https://github.com/rzbrk/CalliopEO_AstroPi/blob/testing/testcases/testcases.md

[Amerlander]

Is now working in my test on the astro pi.