Scope: docker-bhasai to devops and migrate stateful services to k8s

[singhalkarun]

Move all the services which are not moved to K8S to use this repository (update volume names as needed), this should be done with minimum downtime. Plan the process and risks.

https://docs.google.com/spreadsheets/d/1Ojpea-LoP0055R9mzikwUuiF9r0ey2URYm7ONj5gxMc/edit?gid=0#gid=0

[Srijan-SS02]

Process of Migrating services

• Setting up BharatSahAIyak/devops on the machine with bhasai-devops-dev branch
• Setup docker and daemon on the machine
• Adding necessary env varibales in .env file
• Setting up SSH
• Stop the service to be migrated from the host machine
• Migrate named volume using script, by running it on the host machine.
• Update the IP and minio url wherever required
• Start the services

Downtime

• Migrating at time of low usage
• Incremental backup

[Srijan-SS02]

Process of Migrating the services

• Setting up BharatSahAIyak/devops repo(branch bhasai-internal) on the new machine.

• Setup docker and daemon using make target on the new machine

  • sudo apt install make
  • make install-docker
  • Exit out of VM and re-connect to the VM to reflect the latest user changes
  • make setup-daemon

• Setup ssh for migration script

  • create ssh key pair(if not already available) on the old machine ssh-keygen -t rsa -b 4096 -C "Migration Key"
  • copy the public key id_rsa.pub in new machine cat ~/.ssh/id_rsa.pub
  • Add the Public Key to Authorized Keys on the new machine

    cd ~/.ssh
    echo "paste_the_copied_public_key_here" >> ~/.ssh/authorized_keys
    chmod  600 ~/.ssh/authorized_keys

    • use the id_rsa(private key) on old machine for migration script

• Setup Samagra-Development/devops on host machine to use the migration script

• Test the script using a demo volume docker volume create demo_volume make migrate-volume

  • use demo volume here to test

• Add all the necessary environment variables to .env file on the target machine

  
  AI_SERVICES=
  DEFAULT_GITHUB_BRANCH=
  DOCKER_REGISTRY_URL=
  env=
  org=

db

POSTGRES_USER= POSTGRES_PASSWORD=

vault

VAULT_USERNAME= VAULT_PASSWORD=

redis

minio

MINIO_ROOT_USER= MINIO_ROOT_PASSWORD= MINIO_ACCESS_KEY= MINIO_SECRET_KEY=

telemetry

POSTGRES_USER=

POSTGRES_PASSWORD=

CLICKHOUSE_DB= CLICKHOUSE_USER= CLICKHOUSE_PASSWORD=

clickhouse

superset

SUPERSET_DATABASE_PASSWORD= SUPERSET_ENV= SUPERSET_SECRET_KEY= SUPERSET_ADMIN_PASSWORD= SUPERSET_TALISMAN_ENABLED= SUPERSET_EMBEDDING_DOMAIN= SUPERSET_ENABLE_CORS= SUPERSET_ENABLE_TEMPLATE_PROCESSING= SUPERSET_DASHBOARD_FILTERS_EXPOSE= SUPERSET_ENABLE_RBAC_ACCESS=

status(depricated)

- Stop the service on the host machine before migration of volume
      - `docker stop <container_id>`
- Run the script to migrate the volume, keep the name same(as per the project name **-p bhasai**)
     - ` make migrate-volume `
- deploy the service on the new machine after its volume is migrated
      - `make deploy [services=<service_name>]`
- Update the URL from the lb machine ( Karun)
- Update the IP and minio URL in valut wherever required

exmaple:
![image](https://github.com/user-attachments/assets/68a5da1e-1d3b-4665-ba87-5c4e47de231b)

### List of volumes
- bhasai_db
- bhasai_redis
- bhasai_redis_conf
- bhasai_minio
- bhasai_vault
- bhasai_clickhouse 
- fusionauth_es
- fusionauth_db

# Downtime
- Migrating at time of low usage
- Incremental backup