Redesign Build Deployment Process (External)

[MukuFlash03]

This PR serves as the implementation for this issue for redesigning our build and deployment processes across e-mission-server, join, admin-dash, public-dash primarily.

Collaborating on this along with @nataliejschultz .

All four redesign PRs for the external repos: E-mission-server: https://github.com/e-mission/e-mission-server/pull/961 Join: https://github.com/e-mission/nrel-openpath-join-page/pull/29 Admin-dash: https://github.com/e-mission/op-admin-dashboard/pull/113 Public-dash: https://github.com/e-mission/em-public-dashboard/pull/125

---

A rough plan can be:

1st phase

• Consolidating Differences
  • Minimize the differences in internal and external versions, cleanup unnecessary files, so on.
• Image_Build_Push.yml:
  • CI / CD via GitHub actions to build and push images to Dockerhub for all four repos based on the current process for e-mission-server.
  • Image build push only builds for that specific repo when merge is made to a specific branch.
  • Additionally, for e-mission-server, image-build-push will now also need to trigger the GitHub action in join, admin-dash, public-dash.

2nd phase:

• Multi-tier uniform repo structure in nrelopenpath
  • Need to change the internal repos to pull from the externally built images for each repo without duplicating the entire external code repository internally.

[MukuFlash03]

Checking in the first initial set of changes for consolidating the differences in the external and internal versions. These are based on the differences identified here

Currently included these changes:

1. Image_build_push.yml
   • Involves the image_build_push GitHub action automation to build and push images to Dockerhub whenever changes are merged to the specified branch.
   • I have used a local test branch (image-push-merge) which can be changed to master once changes are finalized.

---

Next are various changes to consolidate differences; made in this commit.

2. Loading OpenPATH logo from local image rather than URL link.
   • Changed to use a static local image to load the logo used in app_sidebar_collapsible.py

---

3. Changed sed to jq in start.sh.

   • Making it consistent to the internal repo version. TODO: Might not even need to do this, once we decide how to handle environment variables, and might get rid of using the respective conf files at all.

4. Changed dash version to latest v 2.16.1 Detailed information available in the differences table mentioned at the top.

   • Upgraded to the latest version so that we no longer need the workaround which involves using INDEX_STRING_NO_META.
   • Removed all uses of INDEX_STRING_NO_META.

---

4. Modified Dockerfile(s)
   • Removed assets/ from COPY command as only contents of assets/ were being copied to root app directory.
   • In COPY assets command, copy entire directory instead of just css file.

[MukuFlash03]

Testing admin-dashboard observations

Got steps from @jiji14 which are applicable on existing code on main branch (before redesign code changes). For prod like testing, use docker-compose-prod.yml file instead of docker-compose-dev.yml in the below commands.

# Copy config file
cp config-fake.py config.py

# Run build script
and containers
docker compose -f docker-compose-dev.yml build
docker compose -f docker-compose-dev.yml up -d

# Load data

./docker/load_mongodump.sh  tar_data_file (name of the file containing data)

# Change line 10 in start.sh 

sed "s-localhost-${DB_HOST}-" conf/storage/db.conf.sample > conf/storage/db.conf

# Change DB_HOST in docker-compose-dev.yml

DB_HOST=mongodb://db/openpath_stage

# Run docker containers
docker compose -f docker-compose-dev.yml up -d

# Go to localhost:8050 and login

username  = hello 

password  = world

---

Steps after redesign No need to copy config file or modify sed command as these are not used anymore in redesigned code changes.

# Run build script
and containers
docker compose -f docker-compose-dev.yml build
docker compose -f docker-compose-dev.yml up -d

# Load data

./docker/load_mongodump.sh  tar_data_file (name of the file containing data)

# Change DB_HOST in docker-compose-dev.yml

DB_HOST=mongodb://db/openpath_stage

# Run docker containers
docker compose -f docker-compose-dev.yml up -d

# Go to localhost:8050 and login

username  = hello 

password  = world

[MukuFlash03]

Tested external repo code both with redesign code changes. Admin-dash working with data and values can be seen in dashboard.

Screenshots available with me showing values being loaded: Data tab and Tokens tab

[nataliejschultz]

Testing build from Mukul's server code:

Changed the build image in the Dockerfile to the consolidate-differences image:

Ran $ docker build -t mukul-admindash . to build the image.

Had to turn off the VPN momentarily due to an HTTP request being blocked + keeping it from building.

Loaded cortezebikes snapshot tar.gz data into the database using:

./load_mongodump.sh ../data/openpath_prod_cortezebikes_*_*.tar.gz I did not want to completely rebuild a db container and rename, so I changed the name of the database in load_mongodump.sh to just be db:

Confirmed that the data was loaded in by using docker exec -it db /bin/bash and running:

mongo
show dbs

The data that I loaded was there:

Ran docker run:

$ docker run --name mukul-int-admin-dash-1 -d -e DASH_SERVER_PORT=8050 -e DB_HOST=mongodb://db/openpath_prod_cortezebikes -e WEB_SERVER_HOST=0.0.0.0 -e CONFIG_PATH="https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/" -e STUDY_CONFIG=cortezebikes -e DASH_SILENCE_ROUTES_LOGGING=False -e SERVER_BRANCH=master -e REACT_VERSION="18.2.0" -e AUTH_TYPE="basic" -p 8050:8050 --network emission mukul-admindash

Of note: I kept getting an error when the STUDY_CONFIG variable was enclosed in quotes. This is strange, because the CONFIG_PATH variable was fine to be enclosed in quotes, and the CONFIG_PATH is concatenated with STUDY_CONFIG after being read in as environment variables.

After waiting for a moment for the data to load, I was able to access the login screen at localhost:8050. I logged in with hello:world, and saw that the data were present!

(intentionally sharing the least consequential columns)

The sign-ups trend graph was also shown, I just don't want to show any actual data. I have the screenshot to show it if necessary.

[nataliejschultz]

Internally testing admin dash build:

Cd’ed into local version of internal nrelopenpath multi-tier branch . Changed the Dockerfile to build off of the local image I created in the last test:

Built the image with

docker build -t mukul-internal-admin-dash ./admin_dashboard

Ran the container with:

docker run --name mukul-int-admin-dash-2 -d -e DASH_SERVER_PORT=8050 -e DB_HOST=mongodb://db/openpath_prod_cortezebikes -e WEB_SERVER_HOST=0.0.0.0 -e CONFIG_PATH="https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/" -e STUDY_CONFIG=cortezebikes -e DASH_SILENCE_ROUTES_LOGGING=False -e SERVER_BRANCH=master -e REACT_VERSION="18.2.0" -e AUTH_TYPE="basic" -p 8050:8050 --network emission mukul-internal-admin-dash

Note that I again had to remove the quotations around the value for STUDY_CONFIG. After going to localhost:8050, everything looked exactly the same as before!

User numbers blocked out to protect user privacy.

[nataliejschultz]

Merged changes from the tag automation branch, so now going to cleanup + prepare for merge with main/implementation

[nataliejschultz]

To respond to this we ran into issues with the COPY function in the dockerfile. We can keep it as is, or I can change all of the docker composes to build off of the Dockerfile in the main directory. The two docker files are identical aside from their locations.

Other than that, I have merged the tag update functionality + prepared all of the files to access the correct branches and directories for the automated tag updating.

[shankari]

Squash merging again, with 90 commits for 13 files.