alxndrsh / fpf

Frame Processing Framework
MIT License
17 stars 4 forks source link
ccsds

FPF - Frame Processing Framework

FPF is a flexible framework of objects that can be used to construct different processing algorithms for frame organized data streams.

It was initially designed and build targeting processing of raw satellite telemetry conforming CCSDS standards. Even more precisely, it was intended to build simple tools to conduct an inventory, quality control and initial processing steps of data downlinked from EOS missions (TERRA and AQUA satellites).

Development of FPF was kindly supported by the Space Science and Engineering Center (University of Wisconsin-Madison), RBC Signals, R&D Center ScanEx

FPF is free software. You may copy, distribute, and modify it under the terms of the License contained in the file LICENSE distributed with this source code and binary packages. This license is the same as the MIT/X Consortium license

Fast start

This is a very brief introduction for impatient who just need to try FPF tools to process TERRA (or AQUA) raw data up to Level-0.

Step 1. Download latest binary release package for your operating system (Linux or Windows) from Github page at https://github.com/alxndrsh/fpf/releases Unpack the archive in any folder where you want to keep the tools. No special installation procedure required for running.

Step 2. To process AQUA(and TERRA) DB raw telemetry stream (CADU sequence) you may use supplied configuration file raw2pds_terra_aqua.ini. This configuration fie build a full required processing chain using a number of FPF framework objects. To run if with your file (in the following example /data/terra/TERRA_000.raw) you should call it from command line (having the folder where you have placed the FPF as current):

./fpf -i raw2pds_terra_aqua.ini /data/terra/TERRA_000.raw

Output PDS file will be written with name /data/terra/TERRA_000.raw.pds (just appending .pds extension to the input file name).

Installation

You may install FPF using either ready to run binaries or source code.

Binary distribution

For each milestone release binary distribution of the framework engine is provided for Linux-x64 and Windows operating systems. The binary packages, packed as zip or tar+gz archives, contain an engine executable file, a set of sample chain configuration ini-files and a few documenting text files. Deployment of the binary release is very trivial - just place the files anywhere you think it would be convenient for you to find them. You may test if everything is OK and the executable is compatible with your system simply running fpf command in you terminal window. Running without arguments it will print the version line and then will complain that no configuration file is given. Spend some time to look through this manual to get familiar with the system and learn how you may control the engine and construct usable processing chains.

Some FPF blocks depend on the cURL library, and thus the engine executable does too. The cURL libraries should be installed on you system. On Linuxes it is widely used and usually are available from distro repositories. If you have trouble when run time linking to cURL you may try a simplified version ( fpf_nocurl ) at a price of loosing some minor functionality.

Building from the source code

FPF source code is hosted on the Github. So you may always get access to the latest source code and documentation visiting Github project space. To build the framework you need GCC C++ compiler and GNU make utility (some other compilers may also work without or with minimal adjustments). The core of the framework and most of the node classes do not use anything except the plain C++ 98. But for some classes additional external libraries, specific language or compiler features may be required. If you have trouble with such classes you may remove them from the project usually without consequences for other parts (sure, at a price of loosing some functionality implemented by the dropped classes).

To build the source on Linux operating system open terminal window, step to a folder where you want to keep the build tree and fetch the latest source code from the Github with command

git clone https://github.com/alxndrsh/fpf.git

if you have no git tools installed you may download zipped source code archive from the [Github page] (https://github.com/alxndrsh/fpf), unzip it in any working folder. Step down to the fpf subfolder. Identify which of the provided makefiles better suits your needs:

Run the make utility using selected makefile version:

make -f Makefile

These makefiles build both debug and release(optimized) program versions in bin/Debug and bin/Release subfolders. Release version is recommended for generic use. Debug instrumentation may be helpful to diagnose some issues in case of suspicious or erroneous program behavior.

General framework structure

The framework is designed to support streamlined processing of data structured as a sequence of blocks. Such blocks may be called frames, packets, messages, the only important and required condition are that suck blocks should have finite size, originate as a whole, and come into processing chain strictly one after another. Through the full documentation and the code such blocks are called frames (even if for e.g. CCSDS "source packets" are processed).

FPF provides a number of distinct objects that let one to solve different frame processing tasks. Thanks to objects structure and following a few rules one may construct different processing chains for different tasks or different processing scenarios. At the top level of design each processing scenario should perform in cycle through three operations: read input bit stream, detect and select frames, perform different tasks with the frames one by one. To support such generic scheme FPF defines three main types of processing objects (actually implemented as abstract C++ classes, all others should inherit from one of these base ones).

To get better understanding what all these very generic words are about and what you may accomplish with all this staff, we encourage you to look further though the description of classes already implemented in the framework and try some sample chains built to solve real world tasks.

To glue these objects into a useful working program FPF project provides an engine executable (fpf on Linux or fpf.exe on Windows). This console program lets one to build ad execute a processing chain according to a given configuration file. Configuration file is formatted asa usual INI file with each section containing parameters of one FPF object. Configuration file is given as a value for -i command line argument. Last argument in the command line may be used as a substitution value for a placeholder string "$INPUTFILE$" used anywhere in the configuration file text.

Framework building blocks manual

The framework is a set of quite self consistent and independent classes. So the main meaningful description of what the program using FPF objects should be read from the description of the available classes.

Framer (Frame Source) classes

FrameSource_CADU

FrameSource_CADU class reads a bit stream from a InputStream. Then it identifies CCSDS transfer frames (CADU) based on known sync markers (ASM) and formats frame structures to be passed further to the processing chain. Object of this class is a core of the processing algorithms working with SSCDS CADUs.

Class parameters:

FrameSource_PDS

FrameSource_PDS class reads a stream of CCSDS source packets and generates FPF frame structures for each packet. So object of this class are required to read and process packet level data. Such data level in NASA terminology used for EOS (TERRA/AQUA) processing is called Level-0 (or PDS that stands for Production Data Set). In contrast to the CADU frame stream the packets are not required to have the same size. To coordinate their identification input packets are required to have valid packet headers conforming CCSDS standard and valid packet size field values inside this headers.

Class parameters:

Input Stream classes

InputStream_File

InputStream_File, as its name suggests, is used to read input bit stream from files.

Parameters:

Frame processing nodes

CNode_Counter

CNode_Counter is one of the simplest frame processing nodes. It does quite trivial task - counts number of frames passing though. No changes are made to the frame data or attributes. Recall that frame filters, blocking nodes and routers may be present in the processing chain. So placing several counter objects in different places may be reasonable to control how many frames passed each control point. The total number of counted packets are reported to the screen at the end of processing. Look at the reported name of the counter node if several object have been instantiated

Parameters.

CNode_Descrambler* -

CNode_Descrambler performs descambilng (derandomisation) of the frame content. This operation is performed as bitwise XOR of the frame data with given pseudorandom bit sequence.

Parameters:

CNode_RS

CNode_RS node class perform block error detection and correction using Reed-Solomon algorithm. Currently implemented algorithm type (225,223) with interleaving x4. At the end of processing the node reports total counters of passed frames, frames with corrected errors, number of frames with uncorrected errors.

Parameters:

CNode_PacketExtractor

CNode_PacketExtractor processing node performs extraction and reconstruction of CCSDS source packets from the transfer frames (CADU). Pay attention, that this node actually has two outputs. One is linked to the next node using common next_node parameter. All input frames are passed to this output link without modifications. This let one to continue processing frames in the chain even after CNode_PacketExtractor node (for example, to stack several packet extractors one after another to extract packets from different virtual channels). The second output is used to emit the source packets (recall, they also become frames in out terminology). Sizes of transfer frames and source packets may be not synchronized (moreover, one stream may contain a mix of packets with different sizes), so the packets are generated and passed via the second link not at every input frame arrival.

Parameters: