vfedotovs / sslv_web_scraper

ss.lv web scraping app helps automate information scraping and filtering from classifieds and emails results and stores scraped data in database
GNU General Public License v3.0
5 stars 3 forks source link

FEAT(ws): load env variables using bash script uing 1password cli #314

Open vfedotovs opened 4 days ago

vfedotovs commented 4 days ago
#!/bin/bash

# Sign in to 1Password (if session token is not already available)
eval $(op signin <your-signin-domain> <your-email> <your-secret-key>)

# Retrieve secrets and export them as environment variables
export API_KEY=$(op item get "My-API-Key" --field "apiKey")
export DB_USER=$(op item get "My-Database-Credentials" --field "username")
export DB_PASSWORD=$(op item get "My-Database-Credentials" --field "password")

chmod +x load_secrets.sh
source ./load_secrets.sh

TODO add op cli post deploy install as script

vfedotovs commented 4 days ago

Requires Store 1Password Credentials in GitHub Secrets First, you need to store some sensitive information in GitHub Secrets so that the pipeline can authenticate with 1Password without exposing secrets.

Go to your GitHub repository. Navigate to Settings > Secrets and Variables > Actions. Click on New repository secret. Add the following secrets: OP_SIGNIN_DOMAIN: Your 1Password sign-in domain (e.g., my.1password.com). OP_SECRET_KEY: Your 1Password secret key. OP_EMAIL: Your 1Password email address. OP_MASTER_PASSWORD: Your 1Password master password.

create GH action file

ci.yml

name: CI/CD Pipeline with 1Password Secrets

on:
  push:
    branches:
      - main

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      # Step 1: Checkout the repository
      - name: Checkout repository
        uses: actions/checkout@v3

      # Step 2: Install the 1Password CLI
      - name: Install 1Password CLI
        run: |
          curl -sS https://downloads.1password.com/linux/debian/amd64/stable/1password-cli-latest.tar.gz | tar -xz -C /usr/local/bin/
          chmod +x /usr/local/bin/op

      # Step 3: Sign in to 1Password
      - name: Sign in to 1Password
        env:
          OP_SIGNIN_DOMAIN: ${{ secrets.OP_SIGNIN_DOMAIN }}
          OP_EMAIL: ${{ secrets.OP_EMAIL }}
          OP_SECRET_KEY: ${{ secrets.OP_SECRET_KEY }}
          OP_MASTER_PASSWORD: ${{ secrets.OP_MASTER_PASSWORD }}
        run: |
          eval $(echo $OP_MASTER_PASSWORD | op signin $OP_SIGNIN_DOMAIN $OP_EMAIL $OP_SECRET_KEY --raw)

      # Step 4: Retrieve secrets from 1Password and export them
      - name: Retrieve and export secrets
        run: |
          export API_KEY=$(op item get "My-API-Key" --field "apiKey")
          export DB_USER=$(op item get "My-Database-Credentials" --field "username")
          export DB_PASSWORD=$(op item get "My-Database-Credentials" --field "password")
        shell: bash

      # Step 5: Use secrets in the build process
      - name: Build and test with secrets
        run: |
          echo "Using API_KEY: $API_KEY"
          echo "Using DB_USER: $DB_USER"
          # Run your build or test commands here, e.g., using the secrets
          # python myapp.py --db-user=$DB_USER --db-password=$DB_PASSWORD
        shell: bash
vfedotovs commented 3 days ago
cat create_db_ini.sh
#!/usr/bin/env bash

# Retrieve credentials from 1Password using `op`
DB_HOST=$(op item get "PG_DB_ITEM" --field "host")
DB_NAME=$(op item get "PG_DB_ITEM" --field "pg_db_name")
DB_USER=$(op item get "PG_DB_ITEM" --field "db_user")
DB_PASSWORD=$(op item get "PG_DB_ITEM" --field "db_password")

# Create the `database.ini` file with the content
cat <<EOF > database.ini
[postgresql]
host=$DB_HOST
database=$DB_NAME
user=$DB_USER
password=$DB_PASSWORD
EOF
vfedotovs commented 3 days ago

Usage source ./create_env_file.sh

cat create_env_file.sh
#!/usr/bin/env bash

# Retrieve credentials from 1Password using `op`
ACCESS_KEY=$(op item get "ENV_FILE_ITEM" --field "AWS_ACCESS_KEY_ID")
SECRET_ACCESS_KEY=$(op item get "ENV_FILE_ITEM" --field "AWS_SECRET_ACCESS_KEY")
DB_PW=$(op item get "ENV_FILE_ITEM" --field "POSTGRES_PASSWORD")
DB_NAME=$(op item get "ENV_FILE_ITEM" --field "DB_NAME")
DB_USER=$(op item get "ENV_FILE_ITEM" --field "DB_USER")
SRC_EMAIL=$(op item get "ENV_FILE_ITEM" --field "SRC_EMAIL")
DEST_EMAIL=$(op item get "ENV_FILE_ITEM" --field "DEST_EMAIL")
SENDGRID_API_KEY=$(op item get "ENV_FILE_ITEM" --field "SENDGRID_API_KEY")
S3_BACKUP_BUCKET=$(op item get "ENV_FILE_ITEM" --field "S3_BACKUP_BUCKET")
RELEASE_VERSION=$(op item get "ENV_FILE_ITEM" --field "RELEASE_VERSION")

export S3_BACKUP_BUCKET=$S3_BACKUP_BUCKET
export AWS_ACCESS_KEY_ID=$ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=$SECRET_ACCESS_KEY
export RELEASE_VERSION=$RELEASE_VERSION
export SENDGRID_API_KEY=$SENDGRID_API_KEY

# Create the `.env.prod` file with the content
cat <<EOF > .env.prod
AWS_ACCESS_KEY_ID=$ACCESS_KEY
AWS_SECRET_ACCESS_KEY=$SECRET_ACCESS_KEY

SRC_EMAIL=$SRC_EMAIL
DEST_EMAIL=$DEST_EMAIL
SENDGRID_API_KEY=$SENDGRID_API_KEY

DB_NAME=$DB_NAME
DB_USER=$DB_USER
POSTGRES_PASSWORD=$DB_PW
EOF