Deploy security scanner pipeline to SCS Zuul

[bitkeks]

After #526 is nearly completed, the produced pipelines shall be pushed to https://github.com/SovereignCloudStack/security-infra-scan-pipeline/ and be executed by SCS Zuul.

Story to epic #525.

Definition of Ready:

• [ ] User Story is small enough to be finished within one sprint
• [ ] User Story is clear and understood by the whole team
• [ ] Acceptance criteria are defined
• [ ] Acceptance criteria are clear and understood by the whole team

Definition of Done:

• [ ] SCS Zuul can run security pipeline jobs with configurable targets
• [ ] The deployment is documented completely in either README.md, repo docs or docs.scs
• [ ] All acceptance criteria are met
• [ ] Changes have been reviewed
• [ ] CI tests have run successfully
• [ ] Documentation has been updated
• [ ] Release Notes have been updated

[bitkeks]

As discussed, the container pipeline is ready to be tested and ran in SCS Zuul.

As a first target, the security-test testbed is used. Reports are uploaded to the DefectDojo installed in this environment as well.

[90n20]

Initial draft of the pipeline has been commited to dev_pipeline branch at security-infra-scan-pipeline repository.

The repository has been included into SCS Zuul configuration via zuul PR#20

As for now it executes nightly the following tools aginst a testbed instance: naabu + httpx + nuclei. The ouptut can be seen on SCS Zuul logs while the DefectDojo instance is being configured properly.

[90n20]

Pipeline development has been performed through private repos and a test Zuul deployment in order to not interfere with SCS Zuul instance and working pipelines.

After some blockers with ZAP integration, as by default official containers only support scanning one target at a time, daily scans work as expected, uploading resutls to defect dojo instance.

This daily scan performs a baseline passive scanning, leaving the heavy workloads of active scans to the weekly trigger (still under development). This trigger has been already added to Zuul configuration in zuul-config PR#41

Mentioned updates are being merged via security-infra-scan-pipeline PR#2.

[bitkeks]

Please give an update about the current state of blockers and progress in the next days, maybe in IAM/Sec call?

[90n20]

Please give an update about the current state of blockers and progress in the next days, maybe in IAM/Sec call?

Will add everything we have worked until now in the call agenda.

Regarding current state of the pipeline, we are dealing with uploading weekly Greenbone Scans to DefectDojo. The scans themselves are working and providing valid xml results, but the import process is failing and debug information is not very "explicit" on what is happening.

[90n20]

A new branch has been created with the files and configuration that allow to perform weekly scans. This includes all previous tools plus Greenbone CE => feat/weekly-pipeline

During the development and deployment of this weekly pipeline, we encountered some blockers, that were investigated, mitigated and tested in our private environment before getting it into this stable state:

• Docker-compose file modifications. In order to expose the gvmd unix socket to the worker node, and allow the communication between the python scanning script and the container architecture, some changes on the configuration of the volumes mappings has been done.
• XML report was not being imported to DefectDojo. "import-scan" api POST request and specifically the "scan_type" variable was configured to use "OpenVAS Parser".
• Pipeline was being stopped unexpectedly. The workaround was to specify a default timeout of some hours (1 for daily scans and 6 for weekly ones) in job configuration, as the scan of a single target (including tools deployment and setup) takes around 1,5 hrs to complete.

Minor fixes have been carried on the daily pipeline:

• Feat/zap baseline scan + minor fixes PR
• fix/variables-naming

[90n20]

Weekly pipeline has been merged through https://github.com/SovereignCloudStack/security-infra-scan-pipeline/pull/6. During the review of this PR some concerns were introduced:

• SCS Zuul default timeout prevented long run jobs to finish as was set to 3 hours. This value was increased.
• @gtema proposed to split jobs into individual playbooks and reuse them, in order to improve maintainability and reusability of the code. This is being worked in https://github.com/SovereignCloudStack/security-infra-scan-pipeline/pull/7 and https://github.com/SovereignCloudStack/security-infra-scan-pipeline/pull/7.

Installation and configuration scripts have been provided in order to allow the deployment of a fully functional DefectDojo instance. The infrastructure is created via terraform scripts and the installation via ansible playbooks and roles. Instructions for deployment have been provided in README file: https://github.com/SovereignCloudStack/defectdojo/pull/1

Documentation for the security scanner pipeline is WIP. It will expand https://docs.scs.community/docs/category/automated-pentesting with details on the following:

• Comment latest code and playbooks modifications to give an overview on what each task do at them.
• How tools build up into each other
• Specific tools configurations, in case there is any need to fine tune them
• Quick-start guide:
  • DefectDojo installation and initial configuration (creation of minimum needed engagements, API token for auth, etc)
  • Playbooks custom variables
  • Scans target/s definition