This is a mirror of a government repo hosted on Repo1 by DoD Platform One. Please direct all code changes, issues and comments to https://repo1.dso.mil/platform-one/big-bang/customers/template**
[[TOC]]
This folder contains a template that you can replicate in your own Git repo to get started with Big Bang configuration. If you are new to Big Bang it is recommended you start with the Big Bang Quickstart before attempting customization.
The main benefits of this template include:
helm upgrade using values.yaml that are not controlledNOTE: SOPS supports multiple keys for encrypting the same secret so that each environment can use a different SOPS key but share a secret.
To deploy Big Bang, the following items are required:
User Profile, copy the CLI secret.read_repository permissions.In addition, the following items are recommended to assist with troubleshooting:
This template supports two distinct workflows for staying up-to-date with Big Bang:
package-strategy represents an environment that prioritizes the fastest path to receiving package updates as they are released from Big Bang development.
umbrella-strategy represents an environment that receives updates to all Big Bang packages exclusively from new releases of Big Bang.
Each strategy consists of a Kubernetes manifest containing Flux resources (bigbang.yaml), a Kustomization file (kustomization.yaml), values to pass to Big Bang (configmap.yaml), secrets (secrets.enc.yaml), and additional files used to deploy resources.
We recommend that you adopt a multi environment workflow based off of one of these strategies. If following this path each environment can share a base folder to allow reusability of values between environments.
We need to work off our own Git repo for storing configuration. So, you should fork this repo into a private Git repo owned by yourself or your project. Then, clone your repo locally.
git clone https://<your domain>/<your repo>.git
cd <your repo>
# Create branch for your changes
git checkout -b template-demo
# Copy one of the strategy directories into named-environment directories
cp -r package-strategy/ dev
# update path - switch to gsed on MacOS
sed -i 's/package-strategy/dev/g' dev/bigbang.yaml
cp -r umbrella-strategy/ prod
# update path - switch to gsed on MacOS
sed -i 's/umbrella-strategy/prod/g' prod/bigbang.yamlIt is recommended that you create your own branch so that you can pull the original repository's
mainbranch as a mirror to keep it in sync.
To make sure your pull secrets are not compromised when uploaded to Git, you must generate your own encryption key:
# Figure out what version of gpg your running
gpg --version | head -n 1
# Centos 7 and older AmazonLinux tend to run gpg 2.0.x
# Centos 8, Ubuntu 20, and Debian 11 tend run gpg 2.2.x
# Centos 9 and Mac tend to run gpg 2.3.x
# Generate a GPG master key
# Note:
# By default, gpg 2.3.x generated keys, will generate errors when
# consumed by clients running gpg 2.0.x - 2.2.x
# gpg 2.3.x users must use the following multiline keygen command for
# compatibility with sops and older clients using gpg 2.0.x - 2.2.x
# gpg 2.2.x users should also user the following key gen multiline command
# gpg 2.0.x users substitute --full-generate-key for --gen-key
# Use the following command to generate a compatible key.
# Compatibility issues may arise with GPG versions older than 2.4.3 unless explicitly specified above.
gpg --batch --full-generate-key --rfc4880 --digest-algo sha512 --cert-digest-algo sha512 <<EOF
%no-protection
# %no-protection: means the private key won't be password protected
# (no password is a fluxcd requirement, it might also be true for argo & sops)
Key-Type: RSA
Key-Length: 4096
Subkey-Type: RSA
Subkey-Length: 4096
Expire-Date: 0
Name-Real: bigbang-dev-environment
Name-Comment: bigbang-dev-environment
EOF
# The following command will store the GPG Key's Fingerprint in the $fp variable
# (The following command has been verified to work consistently between multiple versions of gpg: 2.0.x, 2.2.x, 2.3.x)
export fp=$(gpg --list-keys --fingerprint | grep "bigbang-dev-environment" -B 1 | grep -v "bigbang-dev-environment" | tr -d ' ' | tr -d 'Keyfingerprint=')
echo $fp
# The above command will make a key that doesn't expire
# You can optionally run the following if you need the key to expire after 1 year.
gpg --quick-set-expire ${fp} 1y
# cd to the location of the .sops.yaml, then run the following to set the encryption key
# sed: stream editor is like a cli version of find and replace
# This ensures your secrets are only decryptable by your key
## On linux
sed -i "s/pgp: FALSE_KEY_HERE/pgp: ${fp}/" .sops.yaml
## On MacOS
sed -i "" "s/pgp: FALSE_KEY_HERE/pgp: ${fp}/" .sops.yaml
# Save encrypted secrets into Git
# Configuration changes must be stored in Git to take affect
git add .sops.yaml
git commit -m "chore: update default encryption key"
git push --set-upstream origin template-demoFile Editing Tip:
vi file_to_edit, subl file_to_edit, or code file_to_editsudo ln -s "/Applications/Sublime Text.app/Contents/SharedSupport/bin/subl" /usr/local/bin/subl
echo 'export PATH="$PATH:/Applications/Visual Studio Code.app/Contents/Resources/app/bin"' >> ~/.bashrc
tail ~/.bashrc
source ~/.bashrcSops Tip: You can edit a sops encrypted file with your preferred text editor.
# Note the following commands require .sops.yaml to have a valid gpg fingerprint configured
sops ~/file_to_edit #Will edit the file using the default editor, usually vi
EDITOR="subl --wait" sops ~/file_to_edit # Will edit the file using Sublime Text
EDITOR="code --wait" sops ~/file_to_edit # Will edit the file using Visual Studio Code
# You can also permanently add the EDITOR variable to your profile if desired:
echo 'export EDITOR="subl --wait"' >> ~/.bashrcSops Note: Sops isn't guaranteed to preserve the yaml formatting of a file it encrypts
Sops Note: sops is an abstraction layer that allows several secret backends to be used.
.sops.yaml based on the following links:The base/configmap.yaml is setup to use the domain dev.bigbang.mil by default. (Which results in sites that look like this: https://*.dev.bigbang.mil) A demo TLS wildcard certificate is provided in base/bigbang-dev-cert.yaml to use.
Since the private key of the demo cert is in a public repo all traffic sent to a demo site should be treated as compromised and can be decrypted by MITM packet sniffers, so it's necessary to do one of the following:
cd base
# Encrypt the existing certificate
sops -e bigbang-dev-cert.yaml > secrets.enc.yaml
# Save encrypted TLS certificate into Git
git add secrets.enc.yaml
git commit -m "chore: add dev.bigbang.mil tls certificates"
git pushYou will need pull credentials for Iron Bank to retrieve images for Big Bang.
Secrets can be specific to an environment if they are located in that environment's folder (e.g.
prod,dev). Or, they can be shared between environments if located in thebasedirectory.
# Edit the same secret holding your TLS certificates to add the pull credentials
sops secrets.enc.yamlAdd the following contents to the newly created sops secret. Put your Iron Bank user/PAT where it states replace-with-your-iron-bank-user and replace-with-your-iron-bank-personal-access-token.
The name of the secret must be
common-bbif the secret is in thebasefolder orenvironment-bbif the secret is in thedevorprodfolder. Theenvironment-bbvalues take precedence over thecommon-bbvalues.
apiVersion: v1
kind: Secret
metadata:
name: common-bb
stringData:
values.yaml: |-
registryCredentials:
- registry: registry1.dso.mil
username: replace-with-your-iron-bank-user
password: replace-with-your-iron-bank-personal-access-token
istio:
gateways:
# ...
# Clarifying Note: The 'Add TLS Certificates', section above had you
# create this file from a pre-existing file with an embedded demo HTTPS
# cert. So when you go to edit this file you should see a pre-existing
# section with istio.gateways.public.tls.key, etc.
# This note is to clarify that you should manually edit the pre-existing
# file to append the registryCredentials section as shown.When you save the file, it will automatically re-encrypt your secret using SOPS.
# Save pull credentials into Git
git add secrets.enc.yaml
git commit -m "chore: added iron bank pull credentials"
git pushYour private key to decrypt these secrets is stored in your GPG key ring. You must NEVER export this key and commit it to your Git repository since this would compromise your secrets.
We need to reference your git repository so that Big Bang will use the configuration. Add your repository into the GitRepository resource in dev/bigbang.yaml:
cd ../dev/Replace your forked Git repo where it states
replace-with-your-git-repo. Replacereplace-with-your-branchwith your branch name (e.g.template-demoas created above).
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: environment-repo
namespace: bigbang
spec:
interval: 1m
url: https://replace-with-your-git-repo.git
ref:
branch: replace-with-your-branch
secretRef:
name: private-gitThe
Kustomizationresource contains the path in your repo to thekustomization.yamlto start with. If your folder changes, makes sure to updatespec.pathwith the new path.
Now, save and commit your change:
git add bigbang.yaml
git commit -m "chore: updated git repo"
git pushIf you are in an airgap or high side environment and will need to be re-hosting BigBang packages and are not able to use SSH to have Fluxv2 pull the repos, you can still use HTTPS with a "self-signed" DoD certificate via configuring the following:
Create a sops encrypted secret template within prod/repo-ca-credentials.yaml.enc :
apiVersion: v1
kind: Secret
metadata:
name: repo-ca-credentials
namespace: bigbang
stringData:
username: USERNAME
password: PASS
caCert: |
-------PEM ENCODED CERT--------Encrypt the secret with sops: sops -e -i prod/repo-ca-credentials.yaml.enc .
Ensure the above file is added to prod/kustomization.yaml under the resources section so it gets created in the bigbang namespace:
...
resources:
- repo-ca-credentials.yaml.encAdd the following patches to base/kustomization.yaml to add the following secretRef configuration for every GitRepository you will have enabled, eg Jaeger:
patchesStrategicMerge:
...
- |-
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: bigbang
namespace: bigbang
spec:
postRenderers:
- kustomize:
patchesStrategicMerge:
- apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: jaeger
namespace: bigbang
spec:
secretRef:
name: https-ca-credentialsThis will deploy the BigBang HelmRelease and patch all above GitRepositories with the https-ca-credentials secret.
# Save pull credentials into Git
git add prod/repo-ca-credentials.yaml.enc prod/kustomization.yaml base/kustomization.yaml
git commit -m "chore: added gitrepo creds secret and overlays"
git pushThe BigBang Chart is available to be deployed from either Git (GitRepository) or OCI Registry (HelmRepository). The BigbBang Helm Chart is currently signed using Cosign. If using the HelmRepository then we have the ability to validated the chart using the Public Key. To do this, we have to perform the following steps:
Enable additional templates to be installed in the /base/ using base/kustomization.yaml.
Update base/kustomization.yaml to enable/add helmrepo.yaml and cosignsecret.yaml under the bases: section:
# When updating the version of BigBang, make sure to update
# both the bases reference and the GitRepository reference
bases:
- https://repo1.dso.mil/platform-one/big-bang/bigbang.git//base?ref=2.22.0
- helmrepo.yaml
- cosignsecret.yamlEnable override to the HelmRelease to use the HelmRepository by updating the base/kustomization.yaml:
patchesStrategicMerge:
#Enable this section if you wish to use the OCI HelmRepo with Cosign Verfiy instead of the GitRepository.
- |-
apiVersion: helm.toolkit.fluxcd.io/v2beta2
kind: HelmRelease
metadata:
name: bigbang
spec:
interval: 1m
chart:
spec:
chart: bigbang
version: 2.22.0
sourceRef:
kind: HelmRepository
name: registry1
verify:
provider: cosign
secretRef:
name: bigbang-cosign-pubDuring the deploy step, create a Docker secret in the bigbang namespace allowing Flux to pull the OCI chart. This command is defined below in Step 1 of the Deploy section.
Once pointed to the HelmRepository and it successfully pulling, each package within BigBang can then be pointed to their helmRepo sourceType value in BigBang eg:
istio:
sourceType: "helmRepo"Big Bang follows a GitOps approach to deployment. All configuration changes will be pulled and reconciled with what is stored in the Git repository. The only exception to this is the initial manifests (e.g. bigbang.yaml) which points to the Git repository and path to start from.
Deploy SOPS private key for Big Bang to decrypt secrets
# The private key is not stored in Git (and should NEVER be stored there). We deploy it manually by exporting the key into a secret.
kubectl create namespace bigbang
gpg --export-secret-key --armor ${fp} | kubectl create secret generic sops-gpg -n bigbang --from-file=bigbangkey.asc=/dev/stdinCreate imagePullSecrets for Flux
# Image pull secrets for Iron Bank are required to install flux. After that, it uses the pull credentials we installed above
kubectl create namespace flux-system
# Adding a space before this command keeps our PAT out of our history
kubectl create secret docker-registry private-registry --docker-server=registry1.dso.mil --docker-username=<Your IronBank Username> --docker-password=<Your IronBank Personal Access Token> -n flux-systemCreate Git credentials for Flux
# Flux needs the Git credentials to access your Git repository holding your environment
# Adding a space before this command keeps our PAT out of our history
kubectl create secret generic private-git --from-literal=username=<Your Repo1 Username> --from-literal=password=<Your Repo1 Personal Access Token> -n bigbangOptionally add Secret for OCI HelmRepository Deployment method
If you are using HelmRepository to validate the OCI Signed BigBang Chart instead of the GitRepository, create the private-registry secret to pull from the Helm Chart from the OCI repository defined in /base/helmrepo.yaml.
# The private key is not stored in Git (and should NEVER be stored there). We deploy it manually by exporting the key into a secret.
kubectl create secret docker-registry private-registry -n bigbang --docker-server=registry1.dso.mil --docker-username=<Your IronBank Username>--docker-password=<Your IronBank Personal Access Token>Deploy Flux to handle syncing
# Flux is used to sync Git with the the cluster configuration
# If you are using a different version of Big Bang, make sure to update the `?ref=2.22.0` to the correct tag or branch.
kubectl apply -k https://repo1.dso.mil/platform-one/big-bang/bigbang.git//base/flux?ref=2.22.0
# Wait for flux to complete
kubectl get deploy -o name -n flux-system | xargs -n1 -t kubectl rollout status -n flux-systemDeploy Big Bang
kubectl apply -f bigbang.yaml
# Verify 'bigbang' namespace is created
kubectl get namespaces
# Verify Pull from Git was successful
kubectl get gitrepositories -A
# Verify Kustomization was successful
# NOTE: The Kustomization resource may fail at first with an error about the istio-system namespace. This is normal since the Helm Release for istio will create that namespace and it has not run yet. This should resolve itself within a few minutes
kubectl get -n bigbang kustomizations
# Verify secrets and configmaps are deployed
# At a minimum, you will have the following:
# secrets: sops-gpg, private-git, common-bb, and environment-bb
# configmaps: common, environment
kubectl get -n bigbang secrets,configmaps
# Watch deployment
watch kubectl get hr,po -A
# Test deployment by opening a browser to "kiali.dev.bigbang.mil" to get to the Kiali application deployed by Istio.
# Note that the owner of "dev.bigbang.mil" has setup the domain to point to 127.0.0.1 for this type of testing.
# If you are deployed on a remote host you will need to point "kiali.dev.bigbang.mil" to your cluster master node via your /etc/hosts fileIf you cannot get to the main page of Kiali, it may be due to an expired certificate. Check the expiration of the certificate in
base/configmap.yaml.For troubleshooting deployment problems, refer to the Big Bang documentation.
You now have successfully deployed Big Bang. Your next step is to customize the configuration.
In dev/configmap.yaml, enable Twistlock
twistlock:
enabled: truePush changes to Git
git add configmap.yaml
git commit -m "feat: enable twistlock"
git pushBig Bang will automatically pick up your change and make the necessary changes.
# Watch deployment for twistlock to be deployed
watch kubectl get hr,po -A
# Test deployment by opening a browser to "twistlock.dev.bigbang.mil" to get to the Twistlock applicationTo minimize the risk of an unexpected deployment of a BigBang release, the BigBang release version is explicitly stored in the kustomization.yaml files and can be updated for a planned upgrades. The default release is stored in base/kustomization.yaml, but can be overridden in a specific environment like dev/kustomization.yaml.
Reference for the Big Bang kustomize base:
bases:
- https://repo1.dso.mil/platform-one/big-bang/bigbang.git/base/?ref=2.22.0To update dev/kustomization.yaml, you would create a mergePatch like the following:
patchesStrategicMerge:
- |-
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: bigbang
spec:
interval: 1m
ref:
$patch: replace
tag: "2.22.0"This does not update the kustomize base, but it is unusual for that to change.
Then, commit your change:
git add kustomization.yaml
git commit -m "feat(dev): update bigbang to 2.22.0"
git pushIt may take Big Bang up to 10 minutes to recognize your changes and start to deploy them. This is based on the interval set for polling. You can force Big Bang to recheck by running the sync.sh script.
It is recommended that you track Big Bang releases using the version. However, you can use branch in place of tag if needed. The kustomize base uses Go-Getter's syntax for the reference. The helm release (GitRepository) resource uses the GitRepository CRD's syntax.
When you are done testing, you can update the reference in base (and delete this setting in dev) to update Big Bang in all environments.
Do not forget to also update the
base/kustomization.yaml'sbase:reference to point to the new release.
Big Bang deploys applications to *.dev.bigbang.mil by default. You can override the dev.bigbang.mil domain to your domain by updating base/configmap.yaml and adding the following:
domain: insert-your-domain-hereIn addition, you will need to update the TLS certificates by updating base/secrets.enc.yaml.
# Open and edit the encrypted file
sops base/secrets.enc.yamlAfter saving the secrets.enc.yaml file, it will be automatically re-encrypted.
# Push changes to Git
git add base/configmap.yaml base/secrets.enc.yaml
git commit -m "chore: updated domain and tls certificates"
git pushIf you have different certificates for
devandprod, you can also put the values indev/secrets.enc.yamlorprod/secrets.enc.yamlrespectively. The name of the secret must becommon-bbif the secret is in thebasefolder orenvironment-bbif the secret is in thedevorprodfolder. Theenvironment-bbvalues take precedence over thecommon-bbvalues. Make sure to add the file tokustomization.yamlas a resource if it is not already.
Istio Ingress Gateway is deployed with the LoadBalancer service type by default. You can configure Istio to use nodePorts instead by updating dev/configmap.yaml:
istio:
ingressGateways:
public-ingressgateway:
type: "NodePort"
nodePortBase: 30000Node ports are assigned starting from nodePortBase. The nodePortBase specifies the start of a range of 4 unused node ports. Node port will be assigned as follows: Port 15021 (Status) = nodePortBase, Port 80 = nodePortBase+1, Port 443 = nodePortBase+2, Port 15443 (SNI) = nodePortBase+3. Node port base should be in the range from 30000 to 32764. Alternatively, the kubernetesResourceSpec can be used to configure all port parameters. If nodePortBase isn't specified ports will be assigned randomly.
For additional configuration options, refer to the Big Bang and Big Bang Package documentation. Big Bang values can be passed down in the configmap.yaml or secrets.enc.yaml. See the Kubernetes documentation on configmaps and secrets for differences between the two. Secrets should always be SOPS encrypted before committing to Git.
NOTE: The
devtemplate includes several overrides in theconfigmap.yamlto minimize resource usage and increase polling time in a development environment. They are provided for convenience and are NOT required.
Using Kustomize, you can add additional resources to the deployment if needed. Read the Kustomization documentation for further details.
If you need to rotate your GPG encryption keys for any reason, you will also need to re-encrypt any encrypted secrets.
Update .sops.yaml configuration file
.sops.yaml holds all of the key fingerprints used for SOPS. Update pgp's value to the new key's fingerprint. You can list your locally stored fingerprints using gpg -k.
creation_rules:
- encrypted_regex: '^(data|stringData)$'
pgp: INSERT_NEW_KEY_FINGERPRINT_HERERe-encrypt secrets.enc.yaml with your new SOPS keys. This will decrypt the file with the old key and re-encrypt with the new key.
sops updatekeys base/secrets.enc.yaml -y
# Repeat this for all other encrypted files (e.g. dev/secrets.enc.yaml)Deploy new SOPS private key for Big Bang to decrypt secrets
# The private key is not stored in Git (and should NEVER be stored there). We deploy it manually by exporting the key into a secret.
kubectl delete secret sops-gpg -n bigbang
gpg --export-secret-key --armor INSERT_NEW_KEY_FINGERPRINT_HERE | kubectl create secret generic sops-gpg -n bigbang --from-file=bigbangkey.asc=/dev/stdinCommit changes
git add .sops.yaml **/secrets.enc.yaml
git commit -m "chore: rekey secrets"
git pushYou can encrypt files with SOPS using more than one key to allow different keys to decrypt the same file. The encrypted file contains copies of the data encrypted with each key and all of the public keys needed to re-encrypt the file if changes are made.
Only one of the private keys is required to decrypt the file
Add the second key's fingerprint to .sops.yaml:
creation_rules:
- encrypted_regex: '^(data|stringData)$'
pgp: ORIGINAL_KEY
,INSERT_SECOND_KEY_HERERe-encrypt all encrypted files with your new SOPS keys. This will decrypt the file with the original key and re-encrypt with both of the keys.
sops updatekeys base/secrets.enc.yaml -y
# Repeat this for all other encrypted files (e.g. dev/secrets.enc.yaml)Commit changes
git add .sops.yaml **/secrets.enc.yaml
git commit -m "chore: added second key to secrets"
git pushIn our template, we have a dev and a prod environment with a shared base. Let's say we wanted the following:
dev and proddev and prod.sops.yaml for multiple folders:creation_rules:
# Base is shared, so add fingerprints of both keys
- path_regex: base/.*
encrypted_regex: '^(data|stringData)$'
pgp: INSERT_DEV_KEY_FINGERPRINT_HERE
,INSERT_PROD_KEY_FINGERPRINT_HERE
- path_regex: dev/.*
encrypted_regex: '^(data|stringData)$'
pgp: INSERT_DEV_KEY_FINGERPRINT_HERE
- path_regex: prod/.*
encrypted_regex: '^(data|stringData)$'
pgp: INSERT_PROD_KEY_FINGERPRINT_HEREpath_regex.If you do not have
secrets.enc.yamlindevorprod, you can can copy the one inbaseto test out these commands.
sops updatekeys base/secrets.enc.yaml -y
sops updatekeys dev/secrets.enc.yaml -y
sops updatekeys prod/secrets.enc.yaml -yThere is an excellent tutorial on multiple key SOPS here.
Commit changes
git add .sops.yaml **/secrets.enc.yaml
git commit -m "chore: split dev and prod keys"
git pushUpdating values in an encrypted file can be achieved by simply opening the file with sops:
sops base/secrets.enc.yamlWhen you save the file, sops automatically re-encrypts it for all of the keys specified in .sops.yaml.
In this template, we provide the package-strategy and umbrella-strategy deployment strategies. These can be used in any combination you see fit to manage multiple environments. Optimally updates are introduced to your environments through quality gates - where you would test updates in a dev environment before promoting to a prod environment.
In this instance - you can copy the strategy directory you select for each environment and name them accordingly. We will use dev and prod in this example.
This accomplishes:
dev environment before deploying into proddev as close as possible to prod by sharing valuesdev and prod specific settings for resources, external connections, and secretsTo start, we may have the following in each folder:
base
dev
prod
Big Bang dev value changes can be made by simply modifying dev/configmap.yaml. base and dev create two separate configmaps, named common and environment respectively, with the environment values taking precedence over common values in Big Bang.
The same concept applies to dev secret changes, with two separate secrets named common-bb and environment-bb used for values to Big Bang, with the environment-bb values taking precedence over the common-bb values in Big Bang.
If a new resource must be deployed, for example a TLS cert, you must add a resources: section to the kustomization.yaml to refer to the new file. See the base directory for an example.
As documented in Big Bang, optimally the repository that maintains your GitOps state (this template) is being monitored by a tool such as Renovate to provide alerting and automation around updates for Big Bang and packages.
The Renovate configuration in this repository provides an example that will target Repo1 git repositories for both individual packages as well the Big Bang umbrella chart and provide automated merge requests for updates. If following the package-strategy for consuming updates, Renovate will open pull/merge-requests on a per-package basis.
The Renovate Deployment documentation supports deploying Renovate on a Big Bang cluster that can monitor both this template as well as dependencies in other repositories as a generic capability.
Once an environment directory has been created from a given strategy directory, the strategy directories can be omitted from renovate update by adding an ignorePaths option to the renovate.json:
"ignorePaths": ["package-strategy/**", "umbrella-strategy"],