Set up CI/CD GitHub actions

[azmolmiah]

Overview: Make sure that before our code is pushed onto the main branch we're passing the tests unit and integration. Also, may want to initially test AWS infrastructure.

name: DataEngineering CI/CD

on: push: branches:

• main

jobs: build: name: Build and Test runs-on: ubuntu-latest

steps:
  - name: Checkout code
    uses: actions/checkout@v2

  - name: Set up Python
    uses: actions/setup-python@v2
    with:
      python-version: '3.x'

  - name: Install dependencies
    run: |
      python -m pip install --upgrade pip
      pip install -r requirements.txt

  - name: Run unit tests
    run: pytest

  - name: Run security tests
    run: |
      pip install safety bandit
      safety check
      bandit -r .

  - name: Upload test results
    uses: actions/upload-artifact@v2
    with:
      name: test-results
      path: test-reports

deploy: name: Deploy to AWS runs-on: ubuntu-latest needs: build if: github.ref == 'refs/heads/main'

steps:
  - name: Checkout code
    uses: actions/checkout@v2

  - name: Set up AWS credentials
    uses: aws-actions/configure-aws-credentials@v1
    with:
      aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
      aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      aws-region: 'your-aws-region'

  - name: Deploy Infrastructure with Terraform
    run: |
      terraform init
      terraform apply -auto-approve

[azmolmiah]

Github Actions Setup and Workflow:

[azmolmiah]

Going to need to test and check the terraform stuff with CI/CD using BIll's help