Procurement Concierge

The Procurement Concierge is a web application that enables Buyers to work with Program Staff to publish Requests for Information, and for Vendors to respond to them. In addition, Vendors can use this tool to publish their own ideas, enabling Program Staff to facilitate connections between Buyers and Vendors.

This document describes the project's developer environment, technical architecture and deployment infrastructure.

Table of Contents

• Project Organisation
  • Front-End (src/front-end)
  • Back-End (src/back-end)
  • CRUD Resources
  • Shared (src/shared)
  • Database Migrations (migrations)
• Contributing
• Development Environment
  • Dependencies
  • Quick Start
  • NPM Scripts
  • Environment Variables
• Deployment
  • Environments
  • Deployment Process
  • Running Database Migrations
  • Backups
  • Restoring from Backup
  • Replica Sets
• Team
• Credits

Project Organisation

The Concierge is a full-stack TypeScript web application that uses MongoDB for persistence. It is written in a functional and declarative style with the goal of maximising compile-time guarantees through type-safety.

The source code is split into four parts:

Front-End (src/front-end)

A TypeScript single-page application using React, Immutable.js, Bootstrap and SASS. The front-end's build system is executed by Grunt.

The front-end's state management framework (src/front-end/lib/framework.tsx) provides type-safe state management, and is heavily influenced by the Elm Architecture. If you've used Redux before, you will find this to be very similar since Redux is also based on the Elm Architecture. The main difference is that this project's framework derives greater inspiration from the Elm Architecture and it aims to be far more type-safe than Redux.

Back-End (src/back-end)

A TypeScript server that vends the front-end's build assets (src/back-end/lib/routers/front-end.ts) as well as a JSON CRUD API (src/back-end/lib/resources/**.ts) that performs business logic and persists data to a MongoDB database.

The server framework (src/back-end/lib/server/index.ts) provides type-safe abstractions for API development, and is executed by Express (src/back-end/lib/server/adapters.ts).

CRUD Resources

CRUD resources are created in a standardised, type-safe way in this project. CRUD abstractions are located in src/back-end/lib/crud.ts, and it is recommended to review this module prior to extending the API.

Shared (src/shared)

The src/shared folder contains modules that expose types and functions that are used across the entire stack: front-end and back-end.

Database Migrations (migrations)

All database migration logic is stored in the migrations folder. Migrations are managed by the migrate NPM module, and use the native MongoDB driver to execute them. The MongoDB-related environment variables described below are required to define the database to connect to.

You can create a migration using the following command:

npm run migrations:create -- <MIGRATION_NAME>

This command creates a migration file from a template and stores it at migrations/migrations/{TIMESTAMP}_{MIGRATION_NAME}.ts.

DO NOT delete or change committed migration files. Creating and executing migrations is a stateful process, and, unless you know what you are doing, running a database migration should be viewed as an irreversible process.

Contributing

Features should be implemented in feature branches. Create a pull request against the develop branch to have your work reviewed for subsequent deployment.

The develop branch contains all approved code.

The master branch contains work that has passed the Quality Assurance process and is ready to be deployed to production.

Hotfixes can be merged directly to master via a pull request, but should be merged back into the develop branch as well.

Development Environment

Dependencies

If you are using NixOS or the Nix package manager, running nix-shell will install all necessary dependencies, and drop you in a shell with them accessible in your $PATH.

If you are not using Nix, please ensure the following packages have been installed:

• Node.js 14.x
• SASS
• MongoDB 3.6.x
• Docker
• Docker Compose 3.x

Once complete, cd into this repository's root directory and proceed to install NPM dependencies:

npm install

Quick Start

Once you have installed all necessary dependencies, create a .env file and replace the placeholder values with your credentials. Refer to the "Environment Variables" section below for further information.

cp sample.env .env
# Open .env in your text editor.

Finally, open three terminals and run the following commands:

docker-compose up -d db # Terminal 1, starts a local MongoDB server.
npm run back-end:watch # Terminal 2, starts the back-end server.
npm run front-end:watch # Terminal 3, builds the front-end source code.

Then, visit the URL logged to your terminal to view the now locally-running web application.

You can stop the local MongoDB server by running docker-compose down.

You can wipe the local MongoDB server by removing the .db directory.

You can also run the application in a docker container locally, by using docker-compose up -d (do not specify the db service). This will build the front-end, and run the back-end as a service in the background.

NPM Scripts

It is recommended that developers use the following scripts defined in package.json to operate this web application:

# Usage
npm run <SCRIPT_NAME>

Environment Variables

Developers can configure the following environment variables to alter how the web application is built and/or run. All environment variables affect the back-end server's functionality, and are sanitised in src/back-end/config.ts. NODE_ENV is the only environment variable that affects how the front-end source code is built.

In development, developers can create a .env file in the repository's root directory to configure environment variables. As a convenience, developers can refer to sample.env as a guide.

Deployment

This project is deployed to the Government of British Columbia's own OpenShift infrastructure. NOTE The instructions below apply to deployment to the OpenShift 4 (OCP4) environment, and no longer apply to deployment to OCP3 environments.

Environments

We have three application environments:

The Development, Test and Staging environments are secured behind HTTP Basic Auth. Please contact a team member to access these credentials.

Deployment Process

The "1b91ec-tools" OpenShift project is used to trigger the deployment process for all environments.

To deploy to the Development environment, start a build for "app-concierge-dev", and OpenShift will build and deploy HEAD from the development branch into the Dev environment listed above.

To deploy to the Test environment, start a build for "app-concierge-test", and OpenShift will build and deploy HEAD from the master branch into the Test environment listed above.

To deploy to the Production environment, start a build for "app-concierge-prod", and OpenShift will build HEAD from the master branch. You will need to manually deploy the build to the Production environment listed above once it completes by deploying the "app-concierge-prod" deployment in the "1b91ec-prod" project.

For instructions on building images for each of the environments and setting up build and deployment configurations in OpenShift 4, please refer to the instructions in openshift/README.md.

Running Database Migrations

Using an environment's deployment shell, run npm run migrations:up or npm run migrations:down in the root of this repository's directory.

Backups

Automated backups of the MongoDB database are performed with the BC Government Backup Container. Full documentation for this tool can be found here: https://github.com/BCDevOps/backup-container. The schedule for automated backups is as follows:

• Every 6 hours with a retention of 4 backups
• Every 24 hours with a retention of 1 backup (daily)
• Every week with a retention of 4 backups (weekly)
• Every month with a retention of 1 backup (monthly)

A manual backup can be immediately performed by connecting to the backup container pod in OpenShift and running backup.sh -1.

Backup archives are stored in the same OpenShift project as the Procurement Concierge application, on a separate provisioned volume.

You can find instructions for building and deploying the Backup Container images to OpenShift 4 here.

Restoring from Backup

In the unfortunate event that you need to restore your data from a backup archive, the backup.sh script can be used to restore from the last backup file. Refer to https://github.com/BCDevOps/backup-container#restore for details on how to use this script.

Replica Sets

If you are deploying the Concierge application to OpenShift, we recommend running the MongoDB database as a replica set, or Stateful Set in OpenShift. Replica sets use redundant pods on separate nodes to minimize downtime and mitigate data loss. We have provided the necessary configuration and instructions for setting up the replica set in openshift/README.md.

Replica Sets

If you are deploying the Concierge application to OpenShift, we recommend running the MongoDB database as a replica set, or Stateful Set in OpenShift. Replica sets use redundant pods on separate nodes to minimize downtime and mitigate data loss. We have provided the necessary configuration and instructions for setting up the replica set and configuring the application below.

IMPORTANT: Create a backup of your existing database before migrating to a replica set. This is in case anything goes wrong, and for transferring over data to the replica set.

1. Create the replicaset

   • If you wish to change any of the defaults (i.e. database name, passwords, etc.), edit the file openshift/mongodb-replicaset and change the appropriate parameters values before performing the steps below
   • Run oc new-app openshift/mongodb-replicaset.yaml to create the replicaset from the openshift command line tools

2. Migrate existing data to the replicaset

   • Scale down the Concierge application using the existing MongoDB database OR set the SCHEDULED_MAINTENANCE environment variable in the Concierge application deployment to 1
   • Create a data dump of your existing database with mongodump -u admin -p <admin-password> --authenticationDatabase=admin
   • Use the mongo-restore tool to transfer the existing database data to the new replicaset mongorestore -u admin -p <admin-password> --authenticationDatabase=admin

3. Configure the Concierge application to use the replicaset

   • Update the environment variables for DATABASE_SERVICE_NAME, MONGODB_USER, MONGODB_PASSWORD, MONGODB_DATABASE_NAME
   • Add a new environment variable called MONGODB_REPLICA_NAME and assign it the name of the replicaset (by default this is rs0)
   • Scale the Concierge application back up / change SCHEDULED_MAINTENANCE environment variable to 0

4. Set the backup pod to point to the replicaset

   • Update the environment variable MONGODB_URI in the devexutils_backup deployment to reflect the new username, password, database service name, and database name.
   • Redeploy

5. Cleanup

   • Remove the old mongodb deployment
   • Remove the old mongodb service
   • Remove the old mongodb storage
   • Remove the old mongodb persistent volume

Team

The Procurement Concierge is currently operated by the BC Developers' Exchange within the Government of British Columbia.

This project was originally built by the digital product development team at Real Folk.

Credits

This project would not have been possible by the incredible work done by open source project maintainers. The licenses for open source projects used by the Procurement Concierge Program's web app are documented in OPEN_SOURCE_LICENSES.tx.

In addition, the hero image on the web app's landing page is provided by Cytonn Photography on Unsplash.