The cFS Test Framework (CTF) provides cFS projects with the capability to develop and run automated test and verification scripts. The CTF tool parses and executes JSON-based test scripts containing test instructions, while logging and reporting the results. CTF utilizes a plugin-based architecture to allow developers to extend CTF with new test instructions, external interfaces, and custom functionality.
CTF is currently in active development, and is available under the NASA Open Source Agreement license (NOSA). See the license section for more information.
To get started, clone the CTF repository using the following:
git clone https://github.com/nasa/CTF
For more detailed information on CTF usage, see the CTF Software User's Guide (CTF_SUG.pdf)
in the CTF Documentation directory of the CTF releases' Assets (https://github.com/nasa/CTF/releases).
11/20/2023
CTF Core Changes
Add an optional logging configuration csv_tlm_log. If True, CTF creates an additional tlm CSV log file.
Minor improvements and bug fixes.
CTF Plugins Changes
Updates to CFS Plugin
Display full data attributes of nested structures in tlm logs.
Resolve user-defined variables for all test instructions.
Fix a regression with shutdown and restart of CFS targets in the same test script.
Change CFS shutdown behavior to only shut down targets automatically if the target was started by StartCfs.
09/15/2023
StartCfs.09/27/2023
CTF Core Changes
Fix the timing mismatch between CTF run time and computer calendar time.
Fix the issue of EVS/telemetry log files not always being readable during runtime.
Minor improvements and bug fixes.
CTF Plugins Changes
Add a new Trick Plugin
Add three instructions FreezeTrickSim, SetTrickVariable and CheckTrickVariable to interface to a Trick variable server.
Add a new trick section in CTF config files to configure the Trick variable server interface.
Updates to CFS Plugin
Fix a regression in the CFS Plugin that caused CTF to not communicate with CFS targets which were not started via StartCfs.
Allow CheckEvent args to accept literal values in functions.
Log additional timing information in tlm log file.
Add optional payload args to SetUserVariableFromTlmHeader to filter telemetry packets
CCSDS Plugin
Fix CCSDS v1 message header issue.
Fix a regression issue of CCDD JSON reader not reusing known data types referenced by name.
CTF Tool and Scripts Changes
Consolidate unit tests for open source release and internal release.
Improve CTF test environment installation.
Update test scripts and sample cFS workspace.
Update CTF documents.
01/24/2023
CTF Core Changes
Improve label resolution in conditional branching and looping instructions.
Add a new test_variable section in config file to initialize user-defined variables.
Fix a bug in importing OS environment variables when reading config files.
Fix an error when disabling the last instruction in a test.
Minor improvements and bug fixes.
CTF Plugins Changes
Updates to CFS Plugin
Log telemetry packet errors and the raw data to the tlm log file.
Fix killing CFS processes for SSH targets.
Fix capturing CFS standard output file in test results for SSH targets.
CCSDS Plugin
Add detection of inconsistent data type definitions in CCDD JSON files.
Add support for arrays of multi-byte primitives.
Updates to Variable Plugin
variable_type to SetUserVariableFromTlm and SetUserVariableFromTlmHeader instructions for explicit type conversion. CTF Editor Changes
CheckTlmValue instruction.CTF Tool and Scripts Changes
Update CTF build environment: replace Anaconda with PyEnv.
Provide a script tools/re_order.py to order test scripts attributes consistently.
Update example test scripts.
Update sample cFS workspace.
Update CTF documents.
09/26/2022
CTF Core Changes
Allow user-defined variable in wait attribute of instructions.
Minor improvements and bug fixes.
CTF Plugins Changes
Updates to Validation Plugin
Allow SearchStr instruction to search text string in binary file.
Remove SaveFileAsText instruction from the plugin.
Add CheckFileExists instruction, which checks whether a file or folder exists on the host file system.
Updates to Variable Plugin
SetUserVariable instruction, such as "%", "//", "**", "<<", ">>", "&", "|", "^". Updates to CFS Plugin
Change cFS shutdown behavior for Linux and SSH targets to more reliably detect and kill processes.
Improve SendCfsCommand instruction: the header argument could be any header field name defined by CCSDS packet header.
Log tlm header's sequence_count field for debug purpose.
Minor improvements and bug fixes.
CTF Tool and Scripts Changes
Update configs and scripts to work with the current sample cFS workspace.
Consolidate and clean up example config files.
07/20/2022
CTF Core Changes
Update test script naming conventions and log formatting to be consistent.
Various logging improvements.
Update source code for the new sample cFS workspace.
Minor improvements and bug fixes.
CTF Plugins Changes
Updates to Validation Plugin
Add a new instruction InsertUserComment to allow testers to insert comments into test logs for post-test analysis.
Modify SearchStr and SearchNoStr instructions to support regex search.
Updates to Variable Plugin
Add target as an argument to SetUserVariableFromTlm and SetUserVariableFromTlmHeader instructions.
Add an optional argument variable_type to SetUserVariable instruction for explicit type conversion.
Minor improvements and bug fixes: convert bytes array to string in SetUserVariableFromTlm; allow variable to be array index in instruction arguments.
Updates to CFS Plugin
Allow SendCfsCommandWithRawPayload to work with any MID and command code.
Minor improvements and bug fixes.
CTF Tool and Scripts Changes
Update setup_ctf_env.sh to fix the issue of incompatible packages for Anaconda3 installation.
Update CTF Editor (GUI) to highlight invalid json scripts.
Improve documents and test scripts to certify CTF.
Modify the upgrade script in ctf/tools to automatically update CTF config files and test scripts for v1.6 compatibility.
03/31/2022
CTF Core Changes
Consolidate CheckEvent verification timeout configuration using the name verify_timeout in test scripts.
Minor improvements and bug fixes.
CTF Plugins Changes
Add a new plugin: Validation Plugin.
Updates to CFS Plugin
Fix a bug that caused missing logs and telemetry after the first test cases of a run.
Support hex value arguments (of the form "0x1F") in SendCfsCommand, CheckTlmValue, CheckEvent instructions.
Change SendCfsCommand to use the configured target's endianness for bitfield values.
Add new instruction SetUserVariableFromTlmHeader to Variable Plugin.
Add support in CCSDS Plugin for data structures defined in their own files independently of MIDs.
Remove the outdated instruction SetLabel.
Minor improvements and bug fixes.
CTF Tool and Scripts Changes
Integrate jsonlint tool to scan json test scripts before executing tests.
Modify the upgrade script in ctf/tools to automatically update CTF config files and test scripts for v1.4 / v1.5 compatibility.
Update CTF documents and FAQ.
Update CTF unit tests.
02/28/2022
CTF Core Changes
Support local variable evaluation in test instruction arguments with the format $variable$.
Update additional_plugins_path attribute in INI file to accept a list of (multiple) plugin paths.
Remove instruction's default 1 second wait time.
Minor improvements and bug fixes.
CTF Plugins Changes
SendCfsCommandWithRawPayload, which specifies a variable length payload in the form of a hex string.SendInvalidLengthCfsCommand to SendCfsCommandWithPayloadLength.SendCfsCommandWithPayloadLength now includes any serialized args in the payload, up to the length specified.CheckEvent and CheckNoEvent to allow a list of events to be checked simultaneously.SendCfsCommand instruction.Add conditional branching if and else instructions in Control-Flow Plugin. See plugin README for more information.
Clear ram drive on Linux target if cFS instance is starting without -RPR argument.
Minor improvements and bug fixes.
CTF Editor Changes
Fix bug of SetUserVariableFromTlm instruction for child element.
Fix bug of saving instruction argument 0 as string type instead of int type.
Remove backward-compatibility fields such as telemetry_watch_list, command_watch_list.
Fix bug of editor failing to save the modified script in some corner cases.
Support multiple plugin infor folders for editor.
Support new CheckEvent syntax.
CTF Tool and Scripts Changes
Add new test scripts for CTF certification test cases.
Minor improvements to unit test coverage, logging, and example scripts.
Add an upgrade script in ctf/tools to automatically update CTF config files and test scripts for v1.4 compatibility.
Update CTF documents and FAQ.
10/19/2021
CTF Plugins Changes
CFS shutdown on Linux logs an error, but does not fail, if no such process is found.
CFS Plugin macros now require the format #MACRO# instead of #MACRO.
Fix the 'disable' attribute in function definition.
CTF Documentation Changes
Updated JSON Test Script Guide
Add Troubleshooting & FAQ Guide section.
08/26/2021
CTF Core Changes
Logging improvements
colorlog is installed. Use pip install colorlog to enable.Allow test scripts' import paths to include environment variables, including the configured workspace_dir
Allow FTP upload to a directory that already exists.
Minor improvements and bug fixes.
CTF Plugins Changes
Add two new plugins: Variable Plugin and Control-Flow Plugin.
Sort CCDD files before parsing so the files are always processed in alphabetical order.
Allow CCDD macros for non-numeric string literals.
Support custom field arguments for CCSDS headers in SendCfsCommand.
Support the default fill value of arrays in SendCfsCommand args, if the array name is given instead of an index.
Add two new instructions CheckTlmPacket and CheckNoTlmPacket in cFS plugin.
Allow CheckTlmValue to check elements in arrays.
Remove start_cfs_on_init and auto_run configuration options for cFS targets.
Improve the mechanism to shut down cFS.
sysvinit-tools package to enable the pidof command.Minor improvements and bug fixes.
CTF Editor Changes
Add Editor support for new Control Flow instructions.
Improve Editor response while running 2 cFS instances.
Improve handling of large arrays such as in MM_LOAD_MEM_WID_CC by truncating inputs.
Minor improvements and bug fixes.
CTF Tool and Scripts Changes
Expand unit test coverage.
Purge test scripts from sample_cfs_workspace.
Purge start_cfs_on_init and auto_run from INI files.
Purge command & telemetry watch lists from CTF test scripts.
05/20/2021
SendCfsCommand.05/06/2021
CTF Core Changes
CFS Plugin Changes
Fix the use of CCDD macros in test scripts.
Improve argument type checking in SendCfsCommand instruction.
Improve validation of Remote CFS configs. RegisterCfs will fail if any errors are found.
Change ValidateCcsdsData parameter from name to target to match CFS instructions.
Minor improvements and bug fixes.
CTF Editor Changes
Fix the bug of deleting arguments for CheckTlmValue and CheckTlmContinuous instructions.
Minor improvements and bug fixes.
CTF Documentation and Tool Changes
Add document generation by doxygen tool.
Add pylint static code analysis.
Add pytest unit tests for plugins and library source files.
Update open source file header.
Split CI pipeline into two stages.
12/18/2020
CTF Core Changes
Add support for a disable field to CTF instructions within a test script to temporarily disable that instruction.
Add support for a description field to CTF instructions within a test script to capture comments.
Add end_test_on_fail_commands attribute to the Plugin class, allowing plugins to define a list of critical commands that end the current test script on failure.
Minor improvements and bug fixes.
CFS Plugin Changes
Add the RegisterCfs and StartCfs instructions to the end_test_on_fail_commands such that test execution is halted if registering or starting cFS fails.
Add support for Short Event Messages by setting the cfs:evs_short_event_mid_name field in the config to match the MID name of the CFE_EVS_ShortEventTlm_t within the CCSDS JSON definitions.
Add the CheckNoEvent instruction to check that an event message was not sent during the verification timeout.
Ensure args do not get malformed while sending a command to multiple cFS targets
Resolve CheckTlmValue passing if one or more variables fails to be evaluated from the telemetry packet.
Minor improvements and bug fixes.
CTF Editor Changes
Add support for disabling/enabling test instructions or test cases
Add support for viewing/editing descriptions (comments) within the test script
Add the ability to rename folders or test scripts within the editor
Minor improvements and bug fixes.
10/30/2020
Major Release
This is the first open source release for CTF. Note that CTF remains in active development.
Existing users should review the change logs below and ensure current configurations and scripts are updated.
Release Additions
external/ directory with a self contained example CFS project. This project can be configured to utilize CTF out of the box. Refer to external/README.md for more information.CTF Core Changes
Update test script JSON schema with more accurate field names
commands array to instructionscommand object to instructionAdd an option to provide an additional plugin path within the configuration file. Custom plugins are loaded from that directory, in addition to CTF default plugins.
additional_plugin_path = <path_to_custom_plugins_dir> field to the config ini.plugins, lib, etc...), so as to not shadow any modules within the CTF repo.Add an option to ignore specific CTF instructions within the configuration file. This is useful for CI or specific-configurations that may not have the ability to run certain instructions.
ignored_instructions = <instruction_1>, <instruction_2>, ... field to the config ini. Specify the instructions to ignore (comma-seperated).Minor improvements and bug-fixes.
CCSDS Plugin Changes
Update to CCSDS Definition Schema (JSON) to support multiple MID values for the same message definitions
Change config field evs_event_msg_mid to evs_event_mid_name which should now be set to the MID name for the Event Messages (for example CFE_EVS_LONG_EVENT_MSG_MID).
CFS Plugin Changes
Update name field to target
name field is used for all cFS Plugin Instructions. This field is now renamed to target for clarity, and specifies the cFS target to apply the instruction to.
"name": to "target" within existing test scripts. Example scripts have been updated as part of the release.No longer validate cfs_run_dir on registration. Previously, if the cFS instance was not built (i.e the run directory does not exist), validation would fail.
Rename set_length to payload_length for the SendInvalidLengthCfsCommand test instruction.
User IO Plugin (New)
WaitForUserInput test instruction (with a prompt message shown to the user). This is useful when testing safety-critical functionality, or need to pause the test until the user confirms to proceed.Example Plugin Changes
CTF Editor Changes
Improvements to the Run Status View
Updates to support CTF and CCSDS definition changes.
2020-09-14
CCSDS Plugin Changes
CCSDS_header_path to set the appropriate python header module for CTF to usecFS Plugin Changes
Allow "RegisterCfs" instruction to be used without specifying a name. This uses the default [cfs] configuration section.
Add new field to "StartCfs" test instruction to support additional command-line arguments when starting the cFS instance.
{
"command": "StartCfs",
"data": {
"name": "",
"run_args": "-R PO"
},
"wait": 1
}cfs_run_args field defined in the configFix "SendInvalidLengthCfsCommand" instruction to send the actual length specified in the instruction as the payload length
payload_length + length of the CCSDS header(s)Fix command/telemetry sockets not sending/receiving packets when cFS restarts within a single script
Write cFS output log files in append mode such that logs are not overwritten when cFS restarts
CTF Editor Changes
Disable implicit padding of telemetry and command payload structures (assumes explicit padding in the C data structures, or disabled C compiler implicit padding).
Minor improvements and bug-fixes.
2020-08-06
Major backend updates to improve reliability/maintainability of CTF.
cFS Plugin Changes
evs_message_clear_after_time is added to keep events available in CTF telemetry
up to a specific time. This is needed since EVS messages are only output once.is_regex parameter.msg parameter is now optional. Leaving it blank will only verify the APP and ID fields of EVS messages. Logging
Config INI Changes
telem_verify_timeout to ctf_verification_timeouttelem_verify_poll_period to ctf_verification_poll_periodCCSDS_reader_script_path, CCSDS_reader_class, CCSDS_moduleevs_tlm_list_depthip to cfs_target_ipcfs_output_file, log_ccsds_imports, evs_messages_clear_after_timeCTF Editor
is_regex field to CheckEvent instruction.
scripts/example/test_advanced_example.json for reference.2020-08-05
2020-07-08
cFS Plugin Changes
General
CTF Editor
[] where the test author can insert either a hard-coded index, or a macro from the macro map.
CTF Core
2020-05-21
cFS Plugin Changes
CTF Core
CTF Editor
Other minor bug fixes and improvements
2020-04-24
cFS Plugin Changes
Initial Plugin Implementations For
Re-haul CTF Editor to support generic CCSDS JSON Exports, and multi/remote cFS.
Other bug-fixes and improvements
2019-11-22
Initial CTF Release
Initial Plugin Implementations For
Initial CTF Editor Release
MSC-26646-1, "Core Flight System Test Framework (CTF)"
Copyright (c) 2019-2023 United States Government as represented by the Administrator of the National Aeronautics and Space Administration. All Rights Reserved.
This software is governed by the NASA Open Source Agreement (NOSA) License and may be used, distributed and modified only pursuant to the terms of that agreement. See the License for the specific language governing permissions and limitations under the License at https://software.nasa.gov/.
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either expressed or implied.