search

acep-uaf / camio-meter-streams

0 stars 0 forks source link

readme

SEL-735 Meter Event Data Pipeline

This repository contains a set of Bash scripts that make up a data pipeline, designed to automate the process of interacting with an SEL-735 meter. The pipeline is divided into two main executable scripts:

  1. data_pipeline.sh: Handles the first four steps:

    • Connecting to the meter via FTP
    • Downloading new files
    • Organizing and creating metadata
    • Compressing data

  2. archive_pipeline.sh: Handles the final step:

    • Archiving and transferring event data to the dedicated server.

Prerequisites

Ensure you have the following before running the pipeline:

Installation

  1. Clone the repository:

    git clone git@github.com:acep-uaf/camio-meter-streams.git
cd camio-meter-streams/cli_meter

    Note: You can check your SSH connection with ssh -T git@github.com

Configuration

General Configuration Steps

  1. Navigate to the config directory and copy the example configuration files to a new file:

    cd config
cp config.yml.example config.yml
cp archive_config.yml.example archive_config.yml

  2. Update the configuration files with the target details:

    • config.yml: Add the FTP server credentials and meter configuration data.
    • archive_config.yml: Add the source and destination directories and other relevant details.

  3. Secure the configuration files so that only the owner can read and write:

    chmod 600 config.yml
chmod 600 archive_config.yml

Execution

To run the data pipeline and then transfer data to the target server:

  1. Run the Data Pipeline

    Execute the data_pipeline script from the cli_meter directory. The script requires a configuration file specified via the -c/--config flag. If this is your first time running the pipeline, the initial download may take a few hours. To pause the download safely, see: How to Stop the Pipeline

    Command

    ./data_pipeline.sh -c config/config.yml

  2. Run the Archive Pipeline

    After the data_pipeline script completes, execute the archive_pipeline script from the cli_meter directory. The script requires a configuration file specified via the -c/--config flag.

    Command

    ./archive_pipeline.sh -c config/archive_config.yml

    Notes

    The rsync uses the --exclude flag to exclude the working directory to ensure only complete files are transfered.

  3. Run the Cleanup Process (Conditional)

    If the archive_pipeline script completes successfully and the enable_cleanup flag is set to true in the archive configuration file, the cleanup.sh script will be executed automatically. This script removes outdated event files based on the retention period specified in the configuration file.

    If the enable_cleanup flag is not enabled, you can run the cleanup manually by passing in the archive configuration file.

    Command

    ./cleanup.sh -c config/archive_config.yml

    Notes

    Ensure that the archive_config.yml file is properly configured with the retention periods for each directory in the cleanup process.

How to Stop the Pipeline

When you need to stop the pipeline:

Testing

This repository includes automated tests for the scripts using Bats (Bash Automated Testing System) along with helper libraries: bats-assert, bats-mock, and bats-support. The tests are located in the test directory and are automatically run on all pull requests using Github Actions to ensure code quality and functionality.

Prerequisites

Ensure you have cloned the repository with its required submodules, they should be located under the test and test/test_helper directories:

  1. Clone the repository with submodules:

    git clone --recurse-submodules git@github.com:acep-uaf/camio-meter-streams.git

    If you have already cloned the repository without submodules, you can initialize and update them with:

    git submodule update --init --recursive

Running the Tests

  1. Navigate to the project directory:

    cd /path/to/camio-meter-streams/cli_meter

  2. Run all the tests:

    bats test

Adding Tests

When making changes to the pipeline, it is essential to add or update tests to cover the new or modified functionality. Follow these steps to add tests:

  1. Locate the appropriate test file:

    Navigate to the test directory and identify the test file that corresponds to the functionality you're modifying. If no such file exists, create a new test file using the .bats extension (e.g., my_script_test.bats).

  2. Write your tests:

    Use bats-assert, bats-mock, and bats-support helper libraries to write comprehensive tests. Refer to the bats-core documentation.

    If your tests require shared variables or helper functions, define them in test/test-helper/commons.bash to ensure consistency and reusability across multiple test files. For example:

    # commons.bash
MY_VARIABLE="common value"
function my_helper_function {
    echo "This is a helper function"
}

    Example test structure:

    @test "description of the test case" {  
    # Arrange  
    # Set up any necessary environment or input data.  

    # Act  
    result=$(command-to-test)  

    # Assert  
    assert_success  
    assert_output "expected output"  
}

  3. Run your tests locally:

    Ensure your new tests pass locally by running bats test.

  4. Commit and push your changes:

    Include your test updates in the same pull request as the code changes.

Continuous Testing with GitHub Actions

All tests in the repository are automatically executed through GitHub Actions on every pull request. This ensures that all contributions meet quality and functionality standards before merging. Ensure your pull request passes all tests to avoid delays in the review process.