search

stcorp / procsim

Tool to simulate satellite data processor tasks
BSD 3-Clause "New" or "Revised" License
4 stars 1 forks source link

readme

Procsim

Procsim is a tool to simulate satellite data processor tasks.

In the Payload Data Ground Segment (PDGS) of a satellite mission, the data processing functionality is handled by at least one Processing Facility (PF) which integrates one or more processors. The PF is responsible of the processing orchestration, implementing the logic to: prepare the input data for the processing, trigger the processing (performed by the processors), collect the output data. The processor is the entity responsible of the data processing and is composed of one or more Tasks.

Tasks (executables) are called by the PF with a single command line argument: the file name of a JobOrder file. The JobOrder contains information for the Task, such as processing parameters, location of input data and which output data is expected.

This tool, Procsim, can be used to simulate such Tasks, in order to test the interface between the Processing Facility and the software implementing the Task.

Procsim does not do any 'real' processing, but reads and interprets the JobOrder, consumes resources (CPU, disk and memory) and generates output data with correct directory/file names and valid Main Product Headers.

During execution, logging is produced, with log levels according to those specified in the JobOrder. However, log levels can be overruled on the command line or in the configuration file. The program exits with an exit code which can be specified.

Supported missions

Procsim consists of a common core and mission-specific plugins containing the mission-specific code. Currently, the following missions are supported:

Installation instructions

To use procsim, you will need:

Procsim is distributed as a source distribution created using setuptools. It can be installed in several ways, for example using pip or by invoking setup.py manually. Note: installation using setup.py requires super user privileges in most cases.

Using setup.py:

tar xvf <procsim_package_name>.tgz
cd <procsim_package_name>
python3 setup.py install

If setup.py fails due to permission errors, use

sudo python3 setup.py install

Using pip:

pip install <procsim_pacakge>.tgz

Procsim is a command-line based tool. To test the installation, enter:

procsim --version

Usage

For every Task to be simulated, you need:

Shell script

The PF calls the processor with only one argument, the name of the JobOrder file. The shell script, used to redirect the PF's call, should call procsim with the following arguments:

The script will look like this:

#!/bin/sh
procsim -t $0 -j $1 <path_to_config/configfile>

Optionally, you can specify a specific scenario using -s, followed by the name of the scenario.

Scenario configuration

Procsim can act as a stub for all kind of processors. Its behavior is determined by a 'scenario'. A scenario specifies e.g. the amount of resources (CPU/memory/disk) to be used, the time procsim should sleep and the output products to be generated.

The scenarios are described in JSON configuration files. C-style comments and trailing comma's at the end of lists and objects are allowed. Date/time points should be specified as strings in ISO format, such as "2021-02-01T00:24:32.000Z". Time periods, such as the slice period, are in seconds with type float.

A configuration file can contain one or multiple scenarios. The scenario is selected by automatically using the combination of task file name (i.e. the name of the executable called by the PF) and the JobOrder contents, or manually using an additional command line parameter.

The configuration file is structured as following:

{
  "mission": "biomass",
  "log_level": "debug",
  "scenarios": [
      // Array with scenarios
  ]
}

A JSON editor with syntax checking and coloring, such as Visual Studio Code, is recommended to create and edit the configuration files. The configuration parameters are described below.

An example scenario:

    {
      // Task #5
      "name": "L0 step 5, Consolidation of External Calibration Standard products",

      "file_name": "level0_task5.sh",
      "processor_name": "l0_ec",
      "processor_version": "01.01",
      "task_name": "Step5",
      "task_version": "01.00",

      "log_level": "debug",
      "logging": [
        {
          "level": "info",
          "message": "Procsim log message"
        }
      ],

      // resources
      "processing_time": 60,   // in seconds
      "nr_progress_log_messages": 4,
      "nr_cpu": 1,
      "memory_usage": 100,     // in MB
      "disk_usage": 100,       // in MB

      "outputs": [
        {
          "type": "EC_RAW__0S",
          "size": 0,   // in MB
          "enable": true,
          "metadata_source": ".*EC_RAWP_0S.*"
        }
      ],
      "exit_code": 0
    },

The parameters are described below.

Metadata parameters

Most metadata will be copied from an input source. Some metadata is set by the output product generator, such as the product type, the processor name and the processor version (all read from the scenario) and the baseline version (read from the JobOrder). Some output product generators set additional fields as well, such as the 'slice number'.

Metadata parameter values can be specified in the scenario. If already read from the metadata_source, they will be overwritten. Parameter values can be placed in the scenario 'root' (common for all output products) or in a specific output section.

Example:

      "mission_phase": "Tomographic",
      "outputs": [
        {
          "type": "AC_RAW__0A",
          "metadata_source": ".*EC_RAW__0M.*",
          "swath": "AC",
          "operational_mode": "AC",
        },

This scenario sets the mission_phase to "Tomographic" for all output products, and the swath and operational mode of the AC_RAW__0A product to "AC".

Generator specific parameters

Some generators have specific parameters, such as enable_slicing for the RAWSxxx_10 generator. These parameters can be placed on either the scenario level or in a specific output section, as with the metadata parameters.

A list with all supported parameters for a specific output type can be retrieved using procsim -i [product_type].

Example:

$ procsim -i AC_RAW__0A

AC_RAW__0A product generator details:
-------------------------------------

    This class implements the ProductGeneratorBase and is responsible for
    generating Level-0 ancillary products.

    Inputs are a Sx_RAW__0M monitoring product and all RAWS022_10 products
    belonging to the same data take.
    The output takes the begin/end times of the monitoring product and adds the
    leading/trailing margins as specified in the job order or the scenario.
    (defaults is 16/0 seconds).

    The generator adjusts the following metadata:
    - wrsLatitudeGrid, aka the slice_frame_nr, to '___'
    - partialSlice, set to false
    - incompleteSlice, set to false
    - phenomenonTime (the acquisition begin/end times)
    - validTime

Supported scenario parameters for product type AC_RAW__0A are:
   - baseline (int)
   - begin_position (date)
   - end_position (date)
   - num_l0_lines (str)
   - num_l0_lines_corrupt (str)
   - num_l0_lines_missing (str)
   - swath (str)
   - operational_mode (str)
   - mission_phase (str)
   - data_take_id (int)
   - global_coverage_id (str)
   - major_cycle_id (str)
   - repeat_cycle_id (str)
   - track_nr (str)
   - output_path (str)
   - compact_creation_date_epoch (date)
   - creation_date (date)
   - zip_extension (str)
   - leading_margin (float)
   - trailing_margin (float)

Sample code

Directory examples contains examples of scenario configurations, job orders and scripts to demonstrate them.

Program flow

The next section describes the program flow during normal execution of procsim.

Init phase

Processing

Output generation

Program termination

Running in a container

A Dockerfile, to create a container image from source, is provided in the release package. The container wraps the procsim application and the Python interpreter, allowing the tool to run 'anywhere'. The container image can be build using Podman (provided by Red Hat Linux) or Docker. Use the following command in the directory where Dockerfile is located:

podman build -t procsim .

After that, you can use the Dockerized procsim container. You can run

podman run --rm procsim -v

to show the version number. The script examples/generate_L0_container.sh shows a usage example.

Note: replace podman with docker to use Docker instead of podman.