Welcome to the home of the OHS TTAHUB.
HHS | Ad Hoc |
---|---|
[![HHS](https://circleci.com/gh/HHS/Head-Start-TTADP.svg?style=shield)](https://app.circleci.com/pipelines/github/HHS/Head-Start-TTADP) | [![adhocteam](https://circleci.com/gh/adhocteam/Head-Start-TTADP.svg?style=shield)](https://app.circleci.com/pipelines/github/adhocteam/Head-Start-TTADP) |
For the latest on our product mission, goals, initiatives, and KPIs, see the Product Planning page.
Warning when using Docker to run either the full app or the backend services, PostgreSQL (5432) and Redis (6379) are both configured to bind to their well-known ports. This will fail if any other instances of those services are already running on your machine.
docker ps
..env.example
to .env
.AUTH_CLIENT_ID
and AUTH_CLIENT_SECRET
variables to to values found in the team Keybase account. If you don't have access to Keybase, please ask in the acf-head-start-eng slack channel for access.CURRENT_USER
to your current user's uid:gid. This will cause files created by docker compose to be owned by your user instead of root.yarn docker:reset
. This builds the frontend and backend, installs dependencies, then runs database migrations and seeders. If this returns errors that the version of nodejs is incorrect, you may have older versions of the containers built. Delete those images and it should rebuild them. If you are using a newer Mac with the Apple Silicon chipset, puppeteer install fails with the message: "The chromium binary is not available for arm64"
. See the section immediately following this one, entitled "Apple Silicon & Chromium" for instructions on how to proceed.yarn docker:start
to start the application. The frontend will be available on localhost:3000
and the backend will run on localhost:8080
, API documentation will run on localhost:5003
, and minio will run on localhost:9000
.yarn docker:stop
to stop the servers and remove the docker containers.The frontend proxies requests to paths it doesn't recognize to the backend.
Api documentation uses Redoc to serve documentation files. These files can be found in the docs/openapi
folder. Api documentation should be split into separate files when appropriate to prevent huge hard to grasp yaml files.
Make sure you have access to all the necessary spaces on Cloud.gov
On a Mac
cf login -a api.fr.cloud.gov --sso
.bash ./bin/latest_backup.sh -d
(file will be placed in current directory).psql
(if not brew install libpq
).psql postgresql://username:password@127.0.0.1:5432/postgres < ./bounce.sql && psql postgresql://username:password@127.0.0.1:5432/ttasmarthub < db.sql
(Where username:password are replaced with credentials from .env and db.sql is the file you downloaded and unzipped).yarn docker:db:migrate
bounce.sql
select pg_terminate_backend(pid) from pg_stat_activity where datname='ttasmarthub';
drop database ttasmarthub;
create database ttasmarthub;
On Windows TBD
On a Mac with Apple Silicon, puppeteer install fails with the message:
"The chromium binary is not available for arm64"
See docker-compose.override.yml and uncomment the relevant lines to skip downloading chromium and use the host's binary instead.
You will need to have chromium installed (you probably do not). The recommended installation method is to use brew: brew install chromium --no-quarantine
To ~/.zshrc (or your particular shell config), you'll need to add:
export PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
export PUPPETEER_EXECUTABLE_PATH=`which chromium`
On a Mac with Brew installed Docker, yarn commands may fail due to the absence of docker-compose
(vs docker compose
). To resolve:
brew install docker-compose
You can also run build commands directly on your host (without docker). Make sure you install dependencies when changing execution method. You could see some odd errors if you install dependencies for docker and then run yarn commands directly on the host, especially if you are developing on windows. If you want to use the host yarn commands be sure to run yarn deps:local
before any other yarn commands. Likewise if you want to use docker make sure you run yarn docker:deps
.
You must also install and run minio locally to use the file upload functionality. Please comment out S3_ENDPOINT=http://minio:9000
and uncomment S3_ENDPOINT=http://localhost:9000
in your .env file.
Our CI will fail if code is committed that doesn't pass our linter (eslint). This repository contains a pre-commit hook that runs eslint's built in "fix" command on all staged javascript files so that any autofixable errors will be fixed. The precommit hook, in .gihooks/pre-commit, also contains code to auto-format our terraform files, which you can read more about here.
If you are not using your own custom pre-commit hooks:
start from repo root directory
make the pre-commit file executable chmod 755 .githooks/pre-commit
change your default hooks directory to .githooks
.
git config core.hooksPath .githooks
If you are already using git hooks, add the .githooks/pre-commit contents to your hooks directory or current pre-commit hook. Remember to make the file executable.
It's important that our tests fully clean up after themselves if they interact with the database. This way, tests do not conflict when run on the CI and remain as deterministic as possible.The best way to do this is to run them locally in an isolated environment and confirm that they are sanitary.
With that in mind, there a few "gotchas" to remember to help write sanitary tests.
Grant.destroy
needs to run with individualHooks: true
or the related GrantNumberLink model prevents delete.Model.destroy
you should be adding individualHooks: true
to the Sequelize options. Often this is required for proper cleanup. There may be times when this is undesirable; this should be indicated with a comment.testUtils.js
. Be aware that they take a scorched earth approach to cleanup. For example, when debugging a flaky test, it was discovered that destroyReport
was removing a commonly used region.src/lib/programmaticTransaction.ts
, which make maintaining a clean database state when writing tests a breeze.The guidance is on using the captureSnapshot
and rollbackToSnapshot
functions from src/lib/programmaticTransaction.ts
to manage database state during automated testing with Jest. These functions ensure that each test is executed in a clean state, preventing tests from affecting each other and improving test reliability.
captureSnapshot()
: Captures the current state of the database, specifically the maximum IDs from specified tables, which is used to detect and revert changes.rollbackToSnapshot(snapshot: MaxIdRecord[])
: Uses the snapshot taken by captureSnapshot()
to revert the database to its state at the time of the snapshot. This is crucial for cleaning up after tests that alter the database.beforeAll
and afterAll
In this example, captureSnapshot
and rollbackToSnapshot
are used at the Jest suite level to manage database states before and after all tests run. This is useful when tests are not independent or when setup/teardown for each test would be too costly.
describe('Database State Management', () => {
let snapshot;
beforeAll(async () => {
// Capture the initial database state before any tests run
snapshot = await transactionModule.captureSnapshot();
});
afterAll(async () => {
// Roll back to the initial state after all tests have completed
await transactionModule.rollbackToSnapshot(snapshot);
});
it('Test Case 1', async () => {
// Test actions that modify the database
});
it('Test Case 2', async () => {
// Further test actions that modify the database
});
});
This approach uses captureSnapshot
and rollbackToSnapshot
at the start and end of each individual test. It is most effective when tests are meant to run independently, ensuring no residual data affects subsequent tests.
describe('Individual Test Isolation', () => {
it('Test Case 1', async () => {
const snapshot = await transactionModule.captureSnapshot();
// Actions modifying the database
await transactionModule.rollbackToSnapshot(snapshot);
});
it('Test Case 2', async () => {
const snapshot = await transactionModule.captureSnapshot();
// More actions modifying the database
await transactionModule.rollbackToSnapshot(snapshot);
});
});
If switching branches for code review, run yarn docker:reset
before running your tests.
Run yarn docker:test
to run all tests for the frontend and backend.
To only run the frontend tests run yarn docker:test frontend
.
To only run the backend tests run yarn docker:test backend
.
Migrations and seeding of the test db occurs within the script run by the docker:test
command.
To run eslint run yarn lint:all
or yarn lint:fix:all
to have eslint attempt to fix linting problems.
You may run into some issues running the docker commands on Windows:
Permission Denied
errors see this issueOn the frontend, the lcov and HTML files are generated as normal, however on the backend, the folders are tested separately. The command yarn coverage:backend
will concatenate the lcov files and also generate an HTML file. However, this provess requires lcov
to be installed on a user's computer. On Apple, you can use Homebrew - brew install lcov
. On a Windows machine, your path may vary, but two options include WSL and this chocolatey package.
Another important note for running tests on the backend - we specifically exclude files on the backend that follow the *CLI.js
naming convention (for example, adminToolsCLI.js
) from test coverage. This is meant to exclude files intended to be run in the shell. Any functionality in theses files should be imported from a file that is tested. The src/tools folder
is where these files have usually lived and there are lots of great examples of the desired pattern in that folder.
Docker Command | Description | Host Command | Local only Command | |
---|---|---|---|---|
yarn docker:deps |
Install dependencies for the frontend and backend | yarn deps |
yarn deps:local |
|
yarn docker:start |
Starts the backend and frontend | yarn start:local |
||
yarn docker:stop |
Stops the backend and frontend | |||
yarn docker:dbs:start |
Start only the supporting services | |||
yarn docker:dbs:stop |
Stop only the supporting services | |||
yarn docker:test |
Runs tests for the frontend and backend | |||
yarn docker:lint |
Runs the linter for the frontend and backend | |||
yarn docker:db:migrate |
Run migrations in docker containers | yarn db:migrate |
||
yarn docker:db:migrate:undo |
Undo migrations in docker containers | yarn db:migrate:undo |
||
yarn docker:db:seed |
Run all seeders located in src/seeders |
yarn db:seed |
||
yarn docker:db:seed:undo |
Undo all seeders located in src/seeders |
yarn db:seed:undo |
||
Starts the backend web process | yarn start:web |
yarn server |
||
Starts the worker process | yarn start:worker |
yarn worker |
||
Start the frontend | yarn client |
|||
Run tests for only the backend | yarn test |
|||
Run tests for the backend with coverage and output results to xml files | yarn test:ci |
|||
Run yarn test:ci for both the frontend and backend |
yarn test:all |
|||
Run the linter only for the backend | yarn lint |
|||
Run the linter for the the backend with results output to xml files | yarn lint:ci |
|||
Run yarn lint:ci for both the frontend and backend |
yarn lint:all |
|||
Host the open api 3 spec using redoc at localhost:5003 |
yarn docs:serve |
|||
Run cucumber tests | yarn cucumber |
|||
Collect backend coverage report | yarn coverage:backend |
The infrastructure used to run this application can be categorized into two distinct types: ephemeral and persistent.
Linting, unit tests, test coverage analysis, and an accessibility scan are all run automatically on each push to the Ad Hoc fork of HHS/Head-Start-TTADP repo and the HHS/Head-Start-TTADP repo. In the Ad Hoc repository, merges to the main branch are blocked if the CI tests do not pass. The continuous integration pipeline is configured via CircleCi. The bulk of CI configurations can be found in this repo's .circleci/config.yml file. For more information on the security audit and scan tools used in the continuous integration pipeline see ADR 0009.
In order for CircleCi to correctly pull the latest code from Github, we need to create and apply a SSH token to both Github and CircleCi.
The following links outline the steps to take: https://circleci.com/docs/github-integration/#create-a-github-deploy-key https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent
Steps to create and apply deploy token:
This application consists of three deployment environments: development/dev, staging, and production/prod. The CD pipeline is configured via CircleCi. The bulk of CD configurations can be found in this repo's .circleci/config.yml file, the application manifest and the environment specific deployment_config variable files.
Commits to specific branches in github repositories start automatic deployments as follows:
The application is deployed to the following URLs:
Environment | URL |
---|---|
sandbox | https://tta-smarthub-sandbox.app.cloud.gov/ |
dev | https://tta-smarthub-dev.app.cloud.gov/ |
staging | https://tta-smarthub-staging.app.cloud.gov/ |
prod | https://ttahub.ohs.acf.hhs.gov |
Sandbox Environment
An additional deployment environment called "sandbox" is available to developers for testing and development on feature branches prior to making a commit to Ad Hoc's main branch and deploying to "dev". The current .circleci/config.yml assumes deployments to this space are being made from the Ad Hoc repository. Deployments are pushed to the ohstta-sandbox cloud.gov space. To conserve resources, running application instances pushed to this space should be deleted as soon as they are no longer needed. Running application instances can be deleted by logging into cloud.gov, and then selecting and deleting the application.
Secret Management
CircleCI's project-based "environment variables" are used for secret management. These secrets include:
AUTH_CLIENT_ID
and AUTH_CLIENT_SECRET
.SESSION_SECRET
.JWT_SECRET
.NEW_RELIC_LICENSE_KEY
Exception:
Adding environment variables to an application
If you need to add a variable that is both public and does not change between environments, simply add it under env:
in manifest.yml. See NODE_ENV
as an example.
If your env variable is secret or the value is dependent on the deployment environment follow these directions.
If secret, add your variable to CircleCI
If the variable value you want to add needs to remain secret, you will need to add it as a project-based "environment variable" in CircleCI. Ad Hoc engineers can use this link to navigate to the Environment Variables page for our forked repository. Add your environment variables here. If you need different values for sandbox and dev make sure to make two variables, one for each environment.
For example, if you needed to add an environment specific secret SECRET_FRUIT
variable to your application, you could add SANDBOX_SECRET_FRUIT
with value strawberry
and DEV_SECRET_FRUIT
with value dewberry
.
Add both secret and public variables to manifest.yml
In the application manifest, add your SECRET_FRUIT
variable to the env:
object. If you need another non-secret but environment specific variable, like PUBLIC_VEGGIE
, in your application, add that here.
manifest.yml
---
applications:
- name: tta-smarthub-((env))
env:
SECRET_FRUIT: ((SECRET_FRUIT))
PUBLIC_VEGGIE: ((public_veggie))
If public, add the variable values to your deployment_config files
deployment_config/sandbox_vars.yml
public_veggie: spinach
deployment_config/dev_vars.yml
public_veggie: dill
You're all done with public env variables! In sandbox, process.env.PUBLIC_VEGGIE
will be "spinach"
. In dev, process.env.PUBLIC_VEGGIE
will be "dill"
. 🎉
If secret, pass your variables to the cf_deploy
command in the circleci config.
Make two additions here:
parameters
. Give your variable a description and a type of env_var_name
.steps:
, pass your new parameter to cf push
with the --var
flag. You can think of cf_push
as a function, which uses the parameters
as inputs. Make sure to retain the ${}
syntax. This forces CircleCI to interpret your secret_fruit
parameter as a project-based environment variable, and make the correct substitution. This will become clearer in the next step.config/config.yml
commands:
...
cf_deploy:
...
parameters:
...
secret_fruit:
description: "Name of CircleCI project environment variable that
holds the secret fruit"
type: env_var_name
steps:
...
- run:
name: Push application with deployment vars
command: |
cf push --vars-file << parameters.deploy_config_file >> \
--var SECRET_FRUIT=${<< parameters.secret_fruit >>}
If secret, in the deploy
job, add the circle ci project environment variable name that you created in step 1 to the cf_deploy
command as a parameter.
config/config.yml
jobs:
...
deploy:
...
when: # sandbox
...
steps:
- cf_deploy:
secret_fruit: SANDBOX_SECRET_FRUIT
when: # dev
...
steps:
- cf_deploy:
secret_fruit: DEV_SECRET_FRUIT
You're all done! In sandbox, process.env.SECRET_FRUIT
will be "strawberry"
. In dev, process.env.SECRET_FRUIT
will be "dewberry"
. 🎉
Read TTAHUB-System-Operations for information on how production may be accessed.
Our project includes four deployed Postgres databases, one to interact with each application environment (sandbox, dev, staging, prod). For instructions on how to create and modify databases instances within the cloud.gov ecosystem see the terraform/README.md.
Install Version 7 of the Cloud Foundry CLI tool
brew install cloudfoundry/tap/cf-cli@7
Login to cloud.gov account
cf login -a api.fr.cloud.gov --sso
# follow temporary authorization code prompts
Follow prompts to target the desired space
If you haven't used the the cloud foundry plugin cf-service-connect before, install it now
# Example install for macOS
cf install-plugin https://github.com/cloud-gov/cf-service-connect/releases/download/1.1.0/cf-service-connect-darwin-386
Connect to your desired database
cf connect-to-service <app_name> <service_instance_name>
# Example for sandbox
cf connect-to-service tta-smarthub-sandbox ttahub-sandbox
On success, your terminal prompt will change to match the db_name
from the database instance credentials.
This indicates you are in an open psql session, the command-line interface to PostgreSQL.
Use cf run-task command
cf run-task <app_name> --command "<yarn command>"
# Example 1: running data validation script against sandbox
cf run-task tta-smarthub-sandbox --command "yarn db:validation"
# Example 2: undo most recent database migration
cf run-task tta-smarthub-sandbox --command "yarn db:migrate:undo:prod:last"
Check log output, including those from task
cf logs <app_name> --recent
# Example 1: checking sandbox logs
cf logs tta-smarthub-sandbox --recent
# Example 2: checking sandbox logs, grep just for task logs
cf logs tta-smarthub-sandbox --recent | grep APP/TASK/
If on prod, enable shh in space first
cf allow-space-ssh ttahub-prod
Ssh into your desired application (to see application names run cf apps
)
cf ssh <app_name>
# ssh example for sandbox application
cf ssh tta-smarthub-sandbox
Open shell
/tmp/lifecycle/shell
Run your desired command
# example
node ./build/server/src/tools/dataValidationCLI.js
If on prod, disable ssh in space
cf disallow-space-ssh ttahub-prod
We can quickly take a production backup via the CircleCI web interface. To do so, go to the production
branch there and trigger a pipeline with the variable manual-trigger
set to true. You can then retrieve this backup with the script bin/latest_backup.sh
.
In order to keep the non-production environments as close to production as possible we developed a way to transform a restored version of the production database locally if using local database. The script can be run using the following:
yarn processData:local
The transformed database can then be restored in the non-production environments. For details on how to perform a backup and restore, there is information on the cloud.gov site:
https://cloud.gov/docs/management/database-backup-restore/
Refreshing data in non-production environments
In order to keep the non-production environments as close to production as possible we developed a way to transform a restored version of the production database locally if using local database. The script can be run using the following:
yarn processData:local
The transformed database can then be restored in the non-production environments. For details on how to perform a backup and restore, there is information on the cloud.gov site:
https://cloud.gov/docs/management/database-backup-restore/
Using Maintenance Mode
if you need to put the application into maintenance mode, you can run the maintenance script located at bin/maintenance
.
This script require that you have Cloud Foundry's CLI v7 installed to run.
The script takes two flags
Ex.
# Puts the dev environment into maintenance mode
./bin/maintenance -e dev -m on
If you are not logged into the cf cli, it will ask you for an sso temporary password. You can get a temporary password at https://login.fr.cloud.gov/passcode. The application will stay in maintenance mode even through deploys of the application. You need to explicitly run ./bin/maintenance -e ${env} -m off
to turn off maintenance mode.
To update the version of node the project uses, the version number needs to be specified in a few places. Cloud.gov only supports certain versions of node; you can find supported versions on the repo for their buildpack.
Once you have that version number, you need to update it in the following files
You should also update it where it is specified this README file.
You would then need to rebuild the relevant browser images (docker will likely need to pull new ones) and run yarn docker:deps
to rebuild your dependencies.
If you are using NVM, you can set intall a new node version with nvm install VERSION
and set it to be the default version of node via nvm alias default VERSION
.
In the past, we've needed to destroy and recreate particular services (for example, redis). This can be done through the Cloud.gov UI, through the Terraform architecture, and through the cloud foundry command line interface. The following are instructions for using the cloud foundry CLI (cf
) for this.
cf login --sso
).cf services
to list your servicescf help COMMAND
to get the documentation for a particular commandTo delete and recreate a service (this should not be done lightly, as it is a destructive action)
1 Unbind a service:
cf us APP_NAME SERVICE
ex:
cf us tta-smarthub-staging ttahub-redis-staging
2 Delete a service:
cf ds SERVICE
ex:
cf ds ttahub-redis-staging
3 Create a service:
cf cs SERVICE_TYPE SERVICE_LABEL SERVICE
ex:
cf cs aws-elasticache-redis redis-dev ttahub-redis-staging
4 Bind a service:
cf bs APP_NAME SERVICE
ex:
cf bs ttahub-smarthub-staging ttahub-redis-staging
Trigger a redeploy through the Circle CI UI (rather than restaging)
Finally, you may need to reconfigure the network policies to allow the app to connect to the virus scanning api. Check your network policies with:
cf network-policies
If you see nothing there, you'll need to add an appropriate policy.
cf add-network-policy tta-smarthub-APP_NAME clamav-api-ttahub-APP_NAME --protocol tcp --port 9443
ex:
cf add-network-policy tta-smarthub-dev clamav-api-ttahub-dev --protocol tcp --port 9443
You may need to connect across spaces (for example, our clamav-api-ttahub-dev app is shared by all of our ephemeral environments). If so, use the -s flag.
ex:
cf add-network-policy tta-smarthub-staging -s ttahub-dev clamav-api-ttahub-dev --protocol tcp --port 9443