ETL Totesys Project -Various Artists

The project creates applications to Extract, Transform and Load data from a operational database for our client Terriffic Totes. The data is archived in a data lake, and is made available in a remodelled OLAP data warehouse hosted in AWS.

• ETL Totesys Project -Various Artists
  • Features
  • Technologies
  • Installation
  • Project requirements
  • Python dependencies
  • Setup
  • Usage
  • Unit tests
  • Pre-deployment checks
  • Deployment
  • Authors
  • Project status

Features

• Three PEP8 compliant python applications that interact with AWS and database infrastructure and manipulate data as required (details below)
• Use of Terraform for automated provisioning and managing of AWS infrastructure
• Use of GitHub Actions for continuous integration and deployment
• Monitoring and alerting through AWS SNS and Cloudwatch
• Extensive testing and documentation of main applications and utility functions

Technologies

The project uses a number of technologies and AWS serviuce

• Programming: Python, SQL, HCL
• Database: PostgreSQL
• AWS Services: IAM, RDS, Lambda, S3, Secrets Manager, Cloudwatch, SNS
• IaC: Terraform
• CI/CD: GitHub Actions
• Project management: GitHub Projects

Installation

The installation steps below make extensive use of the Makefile.

Project requirements

• Ensure you have Python 3.9 or greater installed
• Add your AWS credentials (with relevent permissions) to GitHub Secrets using the following keys:
  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY

Python dependencies

A full list of core python dependencies for production can be found in the requirements.txt, the main python dependencies include:

• boto3
• pg8000
• pandas
• awswrangler

Setup

After forking and cloning repo:

1. Setup development environment.

    make create-environemnt

2. Install development dependencies e.g bandit, safety, flake8 etc.

    make dev-setup

3. Install python libraries.

   make requirements

Usage

After setup, you can run each application locally.

<!-- Execute the extract application -->
make pyrun file=src/extract.py

Unit tests

After setup, you can run all tests locally.

<!-- Execute all unit tests -->
    make unit-tests

Pre-deployment checks

After setup, you can run all checks (bandit, safety, falke8 and coverage).

<!-- Run all pre-deployment checks -->
    make run-checks

Deployment

The project uses GitHub Actions for CI/CD and you can see more details in the test_and_deploy.yml file.

Note: The CI/CD process is set to trigger upon any push, or merge of a pull request, to the main branch.

Authors

• Atif Khan
• Dave Geddes
• Angus Hirst
• Louis Kelly
• Lucy Adams
• Weronika Falinska

Project status

Closed on 11th August 2023.