NOTE: Watch the end2end demo clicking the DevSecOps demo image above!
DevSecOps CICD pipeline demo using several technologies such as:
Vulnerability and configuration management methods included in this demo are the following:
This pipeline also improve security adding the following Open Source components:
NOTE: Tested and fully working for 4.7+ OpenShift Clusters, including >=4.9!
On every push to the spring-petclinic git repository on Gogs git server, the following steps are executed within the Tekton pipeline:
Advanced Cluster Security for Kubernetes controls clusters and applications from a single console, with built-in security policies.
Using roxctl and ACS API, we integrated in our pipeline several additional security steps into our DevSecOps pipeline:
NOTE: these 3 steps are executed in parallel for saving time in our DevSecOps pipeline.
Argo CD continuously monitor the configurations stored in the Git repository and uses Kustomize to overlay environment specific configurations when deploying the application to DEV and STAGE environments.
and deploys every manifest that is defined in the branch/repo of our application:
Once our application is deployed, we need to ensure of our application is stable and performant and also that nobody can hack our application easily.
ACS can be integrated with several Notifier for notify if certain events happened in the clusters managed. In our case, we integrated with Slack in order to receive notifications when some Policies are violated in order to have more useful information:
These policies notification can be enabled by each system policy enabled in our system, so you can create your own notification baseline in order to have only the proper information received in your systems.
NOTE: By now the integration is manual. WIP to automate it.
The original demo can be extended to use Cosign to Sign Image artifacts and also to sign the Tekton Build Pipeline via Tekton Chaining.
To extend the pipeline run the extend.sh script
./extend.sh
This will install Noobaa(Object Storage), Quay, and create a pod for cosign secret generation and verification.It will also install the tekton chains operator and integrate with ACS policies to generate violations for non signed images.
After installation the pipeline will build images to quay and have a task that signs the image.
We also create a policy in ACS that will generate a violation for every unsigned image
Pipeline can be run normally via the Run the demo Instructions below.
After Pipeline is run Quay will show the image signed by Cosign
Since we have Tekton Chaining enabled, successfully completed Taskruns will also be annotated with cosign signatures and payload information.
And we can verify the signature and payload information of our last successful pipelinerun using the below command.
./demo.sh sign-verify
In this demo, we can control the security policies applied into our pipelines, scanning the images and analysing the different deployments templates used for deploy our applications.
We can enforce the different Security Policies in ACS, failing our CI pipelines if a violation of this policy appears in each step of our DevSecOps pipelines (steps 6,7,8).
This Security Policies can be defined at BUILD level (during the build/push of the image), or at DEPLOYMENT level (preventing to deploy the application).
For example this Security Policy, checks if a RH Package Manager (dnf,yum) is installed in your Image, and will FAIL the pipeline if detects that the image built contains any RH Package Manager:
This ensures that we have the total control of our pipelines, and no image is pushed into your registry or deployed in your system that surpases the Security Policies defined.
To show a complete demo and show the transition from a "bad image" to an image that passes the build enforcement, we can update the Tekton task of the image build and fix the image. In this example, we will be enabling the enforcement of the "Red Hat Package Manager in Image" policy in ACS, which will fail our pipeline at the image-check as both yum
and rpm
package managers are present in our base image.
Update the tekton task:
s2i-java-11
task
s2i-java-11
task.tkn task delete s2i-java-11
kubectl apply -f fix-image/s2ijava-mgr.yaml
You can check the s2ijava-mgr.yaml
file for more details. We have added a step to this Task which leverages buildah to remove the package managers from the image (search for "rpm" or "yum" in the file).
A RHEL or Fedora box
Openshift Cluster 4.7+
oc
binary
Ansible 2.7+
Git
ansible-galaxy collection install community.kubernetes
pip3 install kubernetes
pip3 install openshift
Install some extra Python dependency:
pip3 install jmespath
Fully automated deployment and integration of every resource and tool needed for this demo.
oc login --token=yourtoken --server=https://yourocp
Run the installer:
./install.sh
Check the resources deployed for this demo with:
./status.sh
cd ..
./demo.sh start
NOTE: This pipeline will fail if you don't disable the "Fixable at least Important" policy enforcement behaviour of ACS. This is expected to demonstrate the failure when a violation of the system policy occurs.
Big thanks for the contributors and reviews that helped so much in this demo! We grow as we share!