Deploy Develop and Stable Versions of Application

brettpalmberg commented 3 years ago

User Interface

[x] UI Branch stable --> Bucket: cwbi-apps-stable
[x] UI Branch develop --> Bucket cwbi-apps-develop
[x] All changes from branch new-develop --> develop
[x] Update workflows that use branch new-develop to use branch develop
[x] Delete branch new-develop
[x] Set Root URL for Midas API via Environment Variable at build-time. Similar to change made on another project here as an example

Data

[x] Copy Images from bucket corpsmap-data/instrumentation/ to cwbi-data-develop/midas
[x] Copy Images from bucket corpsmap-data/instrumentation/ to cwbi-data-stable/midas

Airflow

[x] Terraform - Create Airflow Connection MIDAS_STABLE
[x] Terraform - Create Airflow Connection MIDAS_DEVELOP
[x] Deploy updated Airflow scheduler containers
[x] Duplicate DAGs (support Stable and Develop versions)
- [x] Merge https://github.com/USACE/airflow-config/pull/21

Database

[x] Create new database midas_stable initialize tables
[x] Create new database midas_develop Initialize tables
[x] Browser "down for maintenance" page at apps.rsgis.dev/instrumentation
[x] Turn off data ingest into cwbi-geotech-instrumentation
[x] Turn off Airflow DAGs
[x] Turn off API
[x] Export Database cwbi-geotech-instrumentation
[x] Copy database export to midas_develop
[x] Copy database export to midas_stable

APIs and Data Acquisition

[x] Terraform - modularize midas-api
[x] Terraform - modularize midas-dcs-loader
[x] Terraform changes to account for bucket prefix change /instrumentation --> /midas
[x] Update Github Workflows for Develop / Stable branches instrumentation-api
[x] Update Github Workflows for Develop / Stable branches instrumentation-dcs-loader
[x] Deploy API develop at midas-api.rsgis.dev
[x] Deploy API stable at develop-midas-api.rsgis.dev
[x] Deploy midas-dcs-loader develop; 0 workers (so data loads into queue but not database)
[x] Deploy midas-dcs-loader stable; 0 workers
[x] Scale midas-dcs-loader develop to 2 workers
[x] Scale midas-dcs-loader stable to 2 workers
[x] Turn on Airflow DAG for AWARE Sensors into database MIDAS_DEVELOP
[x] Turn on Airflow DAG for AWARE Sensors into database MIDAS_STABLE
[x] Redirect page at apps.rsgis.dev/instrumentation --> midas.rsgis.dev

Cleanup after 1 week of validation

[x] Delete no-longer-used database, api, etc

brettpalmberg commented 3 years ago

Related to deployment approach documented in #90

brettpalmberg commented 3 years ago

Hello @KevinJJackson . @adamscarberry and @brettpalmberg are working on migration of existing MIDAS application to develop and stable instances. Can you jump-in and help us on the items under User Interface?

brettpalmberg commented 1 year ago

Removing myself from assignment; possible that this issue can be closed, as it appears to be substantially complete without additional comments for the past 2 years. Just doing some issue cleanup related to items in my "assigned" inbox.

KevinJJackson commented 1 year ago

@brettpalmberg - Yeah, taking a look into the backend as it seemed like we came across a few tables on a public DB that were causing some weird things to happen. @dennisgsmith - Feel free to close if you feel confident that the deployed DB is good where it is.

dennisgsmith commented 1 year ago

All data has been migrated to midas schema from public and old tables removed.

USACE / instrumentation