DFE-Digital / get-help-with-tech

Provide guidance and forms for the "Get Help With Tech" COVID-19 response initiative
MIT License
10 stars 5 forks source link

Get Help With Tech

An app to host content and forms for the "Get Help With Tech" COVID-19 response initiative.

Prerequisites

For running the app:

For performing releases:

There are two ways to perform a release -

  1. Using the 'Promote container' workflow in Github, for which you only need write access to this repository
  2. Manually, from your terminal command prompt.

To perform manual releases, you will need:

For generating Entity Relationship Diagrams:

Setting up the app in development

  1. Run bundle install to install the gem dependencies
  2. Run yarn to install node dependencies
  3. Run bin/rails db:setup to set up the database development and test schemas, and seed with test data
  4. Run bundle exec rails server to launch the app on http://localhost:3000
  5. Run ./bin/webpack-dev-server in a separate shell for faster compilation of assets

Setting up local development data

These are the tasks that you will need to run to set up your local db:

Setting up GOV.UK Notify

Creating Support and Supplier (ComputaCenter) users

Open up a rails console with...

bundle exec rails c`

Single user

# creates a single support user
CreateAdminUsersService.new('first.last@digital.education.gov.uk').create!
# or (:support is default)
CreateAdminUsersService.new('first.last@digital.education.gov.uk', :support).create!

# creates a single computacenter user
CreateAdminUsersService.new('first.last@computacenter.com', :supplier).create!

Multiple users

You can only create multiple users of the same type in one batch.

# creates multiple support users
CreateAdminUsersService.new(['first1.last1@digital.education.gov.uk', 'first2.last2@digital.education.gov.uk']).create!
# or (:support is default)
CreateAdminUsersService.new(['first1.last1@digital.education.gov.uk', 'first2.last2@digital.education.gov.uk'], :support).create!

# creates muliple computacenter users
CreateAdminUsersService.new(['first1.last1@computacenter.com', 'first2.last2@computacenter.com'], :supplier).create!

Notes

Running specs

bundle exec rake parallel:setup
bundle exec parallel_rspec spec/models # to run spec directories (or individual spec files) in parallel
bundle exec rake # runs rspec (in parallel) and other code checks

Linting

It's best to lint just your app directories and not those belonging to the framework, e.g.

bundle exec rubocop app config db lib spec Gemfile --format clang -a

or

bundle exec scss-lint app/webpacker/styles

Static analysis for security issues

bundle exec brakeman

All the above are run automatically on GitHub Actions when pushing a PR.

Generating an Entity Relationship Diagram

The rails-erd gem uses Graphviz to draw diagrams of the ActiveRecord models and their relationships. You can run it at any time using:

bundle exec rails erd

Integrations

GOV.UK Notify for sending emails

Computacenter TechSource - this app will post cap update requests to TechSource when we change the number of devices allocated to a school.

APIs

Computacenter API

Deploying

In the normal flow of things, the simplest way to perform a release is by using the Github Actions we have set up on this repository.

Merging a Pull Request to main will automatically run the tests and perform a release to the 'dev' environment. Once this has completed (you can check the currently deployed commit SHA from http://get-help-with-tech-dev.london.cloudapps.digital/healthcheck.json, and make sure it matches 'main'), you can then promote that image from dev to staging, and then from staging to prod as follows:

  1. Click the 'Actions' tab
  2. Under 'Workflows' on the left, click 'Promote container between environments'
  3. In the blue header, click 'Run workflow', fill in the from and to environments and then click the green 'Run workflow' button (see screenshot below)

    Screenshot of the user interface for promoting a container between environments

Deploying manually on GOV.UK PaaS

Prerequisites

The manual deployment process

  1. Sign in to Cloud Foundry (using either your GOV.UK PaaS account or single sign-on, once you've enabled it for your account)
  2. Run docker login to log in to Docker Hub
  3. Run make dev release to build, push and deploy the Docker image to GOV.UK PaaS development instance
  4. Test on https://get-help-with-tech-dev.london.cloudapps.digital
  5. Run make staging promote FROM=dev to deploy the -dev image to staging
  6. Test on https://staging-get-help-with-tech.education.gov.uk/
  7. Run make prod promote FROM=staging to deploy to production
  8. Test on https://get-help-with-tech.education.gov.uk/

The app should be available at https://get-help-with-tech.education.gov.uk/

If you need to rollback a release

Sometimes releases fail on production - sometimes this is due to a migration which fails due to unexpected data in production, for instance. When this happens, you'll probably want to minimize user impact by rolling back the release to a previous version ASAP, while you investigate offline.

There are some make tasks that can help:

make prod remote-docker-tags - list all available tags for production Docker image, on Docker Hub (you can replace prod with another environment) make prod rollback-to TAG=... - re-deploy the Docker image tagged with the given TAG

How does this work?

The normal release process works as follows:

  1. Build a new Docker image called get-help-with-tech-(env name)
  2. Pull any existing image from Docker Hub called dfedigital/get-help-with-tech-(env name):latest. Re-tag it with replaced-at-(timestamp) and push it back up to Docker Hub.
  3. Tag the newly-built image as dfedigital/get-help-with-tech-(env name):latest, and push it to Docker Hub, overwriting any existing image with the same name and tag
  4. Tell GOV.UK PaaS to pull dfedigital/get-help-with-tech-(env name):latest from Docker Hub, and deploy it to the app called get-help-with-tech-(env name)

If you're not building a new image but promoting an image from another environment, the process is a little simpler, but largely the same:

  1. Pull any existing image from Docker Hub called dfedigital/get-help-with-tech-(env name):latest. Re-tag it with replaced-at-(timestamp) and push it back up to Docker Hub.
  2. Pull the image dfedigital/get-help-with-tech-(FROM env name):latest, re-tag it as dfedigital/get-help-with-tech-(TO env name):latest' and push it back up to Docker Hub
  3. Tell GOV.UK PaaS to pull dfedigital/get-help-with-tech-(env name):latest from Docker Hub, and deploy it to the app called get-help-with-tech-(env name)

In either case, before replacing the dfedigital/get-help-with-tech-(env name):latest image on Docker Hub, it will automatically re-tag the existing image as replaced-at-(timestamp) - making it straightforward to rollback to any available previous tag.

Over time, the timestamped tags will accumulate in Docker Hub, and should be pruned occasionally. You can do this by visiting https://hub.docker.com/repository/registry-1.docker.io/dfedigital/get-help-with-tech-dev/tags as a logged-in member of the dfedigital organisation, clicking the checkboxes next to any no-longer-required tags, and choosing 'Delete' from the select box at the top of the list.

Environment variables

Some values are configurable with environment variables:

Name Description Default
GHWT__SIGN_IN_TOKEN_TTL_SECONDS Sign-in tokens will expire after this many seconds 600
GHWT__GOVUK_NOTIFY__API_KEY API key for the GOV.UK Notify service, used for sending emails REQUIRED
GHWT__HOSTNAME_FOR_URLS Hostname used for generating URLs in emails http://localhost:3000/
GHWTHUAWEIDEVICES__PASSWORD Password for Huawei routers secret
GHWT__GOVUK_NOTIFYTEMPLATESSIGN_IN_TOKEN_MAIL ID of the template in GOV.UK Notify used for mailing sign-in tokens '89b4abbb-0f01-4546-bf30-f88db5e0ae3c'
GHWT__STATIC_FILE_CACHE_TTL how long CDNs and browsers should cache static assets for in production, in seconds. (nil)
GHWTTHROTTLE* Request throttling limits, see settings.yaml for more info (see settings)
GHWTLOGSTASHHOST Hostname for where logstash should send logs (nil)
GHWTLOGSTASHPORT Port for where logstash should send logs (nil)
GHWTSENTRYDSN DSN (Client key) for Sentry.io error reporting (nil)
GHWTCOMPUTACENTEROUTGOING_API__ENDPOINT URL of the CapUpdateRequest API at TechSource (nil)
GHWTCOMPUTACENTEROUTGOING_API__USERNAME Basic auth username to use for the TechSource CapUpdateRequest API (nil)
GHWTCOMPUTACENTEROUTGOING_API__PASSWORD Basic auth password to use for the TechSource CapUpdateRequest API (nil)
GHWTZENDESKUSERNAME Username for Zendesk account to be able to use the Zendesk API. Both Zendesk options need to set before Zendesk API can be used. (nil)
GHWTZENDESKTOKEN Token for Zendesk account to be able to use the Zendesk API.Both Zendesk options need to set before Zendesk API can be used. (nil)
GHWTDATABASE_FIELD_ENCRYPTIONKEY Secret key for the encrytion fields in the assets table REQUIRED
GHWTDATABASE_FIELD_ENCRYPTIONSALT Salt for the encryption field in the assets table REQUIRED
GHWT__SLUG_CHECKSUM_SECRET A secret for hashing a checksum for IDs in the URL REQUIRED
GHWT__API_TOKEN_TTL TTL in days after which API Tokens CC use will expire from creation date 90
GHWT__SITE_BANNER_MESSAGE Banner message text to appear at the top of all pages of the site (nil)
GHWT__LONG_FORM_SITE_BANNER_MESSAGE_FLAG Boolean value, when true, displays the long form banner message text to appear at the top of all pages of the site from somefile.md false
GHWT__LONG_FORM_SITE_BANNER_MESSAGE_PARTIAL Path to the markdown partial file containing the long form site banner message. Default resolves to site_banner/_long_form.md site_banner/long_form

See the settings.yaml file for full details on configurable options.

Feature Flags

Certain aspects of app behaviour are governed by a minimal implementation of Feature Flags. These are activated by having an environment variable FEATURES_(flag name) set to 'active', for example:

# start the rails server with rate limiting active
FEATURES_rate_limiting=active bundle exec rails s

The available flags are listed in app/services/feature_flag.rb, and available in the constant FeatureFlag::FEATURES.

Name Description
FEATURES_gias_data_stage_pause Pauses any GIAS data staging of trusts and schools, should be used e.g. in Septemeber when GIAS data is changed a lot and unsettled

To display, set and unset feature flags on GOV.UK PaaS:

# display the feature flags
cf env (app name) | grep FEATURES

# For example:
cf env get-help-with-tech-prod | grep FEATURES

# set an env var
cf set-env (app name) (environment variable name) (value)

# For example:
cf set-env get-help-with-tech-prod FEATURES_rate_limiting active

# To unset the var:
cf unset-env (app name) (environment variable name)

# For example:
cf unset-env get-help-with-tech-prod FEATURES_rate_limiting

Operations

Connecting to the deployed containers

Log on to the first instance of the sidekiq container with:

make (env) ssh

If you want to ssh to a particular process and/or particular instance, you can be specific as follows:

make (env) ssh PROCESS=(web|sidekiq) INSTANCE=(0|1|...)

Both parameters are optional. PROCESS defaults to sidekiq, to prevent any possibility of using too much memory in the console and accidentally getting the web container restarted by the hosts' monitoring. INSTANCE defaults to 0.

Running the Rails console

Some service steps can only be carried out using the Rails console. To get to the console on GOV.UK PaaS, first ssh as above. Once you have a prompt on the container, you'll be in a subshell with the Rails app's environment variables, and in the app's root directory: In this subshell, you can then launch the console in the normal way:

bundle exec rails c

Note that the log messages you see will be in structured JSON format, for aggregation to Kibana via Logstash. For clarity, you might want to unset RAILS_LOG_TO_STDOUT before launching the console.

Transferring files to/from the deployed containers

Files can be uploaded/downloaded as follows:

make (env) upload LOCAL_PATH=/some/local/file/path REMOTE_PATH=/some/remote/path PROCESS=(web|sidekiq) INSTANCE=(0|1|...)
make (env) download LOCAL_PATH=/some/local/file/path REMOTE_PATH=/some/remote/path PROCESS=(web|sidekiq) INSTANCE=(0|1|...)

The PROCESS and INSTANCE parameters follow the same conventions as for make (env) ssh - defaulting to instance 0 of the sidekiq process.

You will be prompted for a password for the connection - this is a one-time-only password, generated on-demand via the GOV.UK PaaS platform. The output just above the prompt will tell you what the password is for this session - you should see a message like this:

Connecting to 'sidekiq' instance '0'

*** Enter (some hex string) at the password prompt (this is a one-time-only password) ***

scp -P 2222 -o StrictHostKeyChecking=no -o User=cf:(...) ssh.london.cloud.service.gov.uk:/tmp/test.txt /tmp/
cf:(....)@ssh.london.cloud.service.gov.uk's password:

Just copy-and-paste the given password into the prompt, and the transfer will begin.

Viewing Logs

Tail the logs for a given env:

make (env) logs

View recent logs for a given env:

make (env) logs-recent

Log Aggregration

Semantic Logger is used to generate single-line logs. In production environments, when RAILS_LOG_TO_STDOUT is enabled, this is configured to output JSON logs. These logs are then sent to the log aggregator.

Development

You can configure logstash to send logs to your log aggregator by setting the logstash host and port environment variables.

Logstash Configuration

A copy of the logstash filter we use exists in the repo. This has to be installed manually in the log aggregator to be used.

Amending the devices guidance

If you need to amend the Get help with technology: devices guidance:

  1. Create a new branch on this repository

  2. Make your amends:

  3. Create a pull request from the branch

  4. Get a colleague to review your pull request – the change cannot be merged without this step

  5. Merge the pull request with a merge commit

Once the pull request is merged, it will be released the next time the service is released.