how-many-buzzwords

Some buzzwords are incredibly overused, a simple tool tool to find the biggest culprits

The currently released version of this project can be used here:

• https://howmanybuzzwords.com
• https://howmanybuzzwords.co.uk

High Level Architecture

This project comprises of a React (TypeScript) SPA and several microservices deployed on AWS that communicate between each other using events.

The microservices use a Serverless architecture, making use of Lambda, API Gateway (HTTP/WebSocket), Step Functions, DynamoDB etc. The lambda functions are largely written using TypeScript, however, a transition from JavaScript is still in progress.

The following C4 Container diagram shows how the various components of the How many buzzwords system provide the functionality provided on the site:

A larger set of C4 model diagrams can be found in the docs folder.

Requirements

The following CLI tools must be installed to validate, build, and test the how many buzzword components:

• AWS CLI - More information can be found here: https://aws.amazon.com/cli/
• SAM CLI - More information can be found here: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html
• CloudFormation Linter - More information can be found here: https://github.com/aws-cloudformation/cfn-lint

Many of the scripts used within the project require jq to function, this can be installed by executing:

sudo apt-get install jq

General script usage

Each of the scripts used within this project have usage instructions, these can be found by providing the -h flag to each script.

You may need to give the scripts permissions to run on your system, this can be done by running the following command:

chmod u+x $(find ./scripts/ -type f)

Usage

Install Dependencies

Run the following command to install all of the dependencies for the project:

npm run ci

Deploying components to AWS

Run the following commands to deploy all components:

./scripts/deploy.sh

Tearing down deployed components

Run the following commands to teardown all deployed components, along with their related resources:

./scripts/teardown.sh

Running the SPA component locally

Generate a .env file in the ./ui folder by running the following the command:

./scripts/helpers/generate-ui-env.sh \
    -c $INSERT_NAME_OF_CRAWL_STACK \
    -k $INSERT_NAME_OF_KEYPHRASE_STACK \
    -o ./ui/.env

Run the following script to run the UI locally:

npm --prefix ./ui/ start

Running tests

Unit tests can be ran simply by running the following command:

npm run test:unit

Integration tests require a local instance of DynamoDB local running. This can be started by running:

./scripts/setup-local-db.sh -i

This will start a local instance and setup a GUI on http://localhost:8001 to view the DB's contents.

Once started, the integration tests can be ran using the following command:

npm run test:integration

Cloudformation Template Validation

Run the following command to validate all template definitions within the project:

xargs -r0a <(find ! -path "*/.aws-sam/*" -name *-template.yml -print0) -I {} sh -c 'cfn-lint {} -i W3002 -i W3045'

Programatically starting a crawl

The test-crawl.sh script can be used to trigger a crawl without running the UI:

Example usage:

./scripts/testing/test-crawl.sh -s dev -u https://www.example.com

Programatically listening to results

The test-connect.sh script can be used to listen to the results of a crawl in real time without running the UI:

Example usage:

./scripts/testing/test-connect.sh -s dev -u https://www.example.com

Local Testing of Lambda functions

Run the following commands to invoke lambda functions locally:

sam local start-lambda --template ./templates/buzzword-template.yml
aws lambda invoke --function-name "INSERT_NAME_OF_LAMBDA_RESOURCE" --endpoint-url "http://127.0.0.1:3001" --no-verify-ssl local-lambda-output.txt

To debug the function locally, you can start the lambda function container in debug mode using the following command:

sam local start-lambda --debug-port 9229 --template ./templates/buzzword-template.yml

Once the lambda has been invoked, then you can attach the debugger to the process using the VS Code debug configuration called "Attach to Local SAM resources"

CI/CD Setup

The CI/CD pipeline requires specific permissions in order to perform validation and deployment. Run the following command to set up OIDC access for the CI pipeline (Replace the organisation and repository name values appropriately):

aws cloudformation deploy \
    --template-file ./templates/buzzword-ci-users-template.yml \
    --stack-name buzzword-ci-users \
    --capabilities CAPABILITY_IAM \
    --parameter-overrides GithubOrganisation=$INSERT_VALUE RepositoryName=$INSERT_VALUE GithubOIDCProvider=$INSERT_VALUE

Once the above command has executed, run the following commands to get the role ARNs:

./scripts/helpers/fetch-stack-outputs.sh -s buzzword-ci-users

The following GitHub secrets should be created with the appropriate key/value output from the previous command:

The template validation performed by the CI pipeline searches for .yml files that are suffixed with -template, therefore, if you wish for a template file to be validated then simply suffix the file with -template.