search

mercedes-benz / sechub

SecHub provides a central API to test software with different security tools.
https://mercedes-benz.github.io/sechub/
MIT License
260 stars 63 forks source link

System testing test suite #453

Open Jeeppler opened 3 years ago

Jeeppler commented 3 years ago

SecHub already has a set of integration tests, allowing one to test different parts of the system (subsystems). However, SecHub does not have system tests. System tests in the context of SecHub are tests, which test the entire system and real products used by SecHub. System tests are emulating a user interacting with SecHub.

We currently run system tests manually. For example, by triggering a code scan against a known repository or application with vulnerabilities. This naive approach works well with a very limited amount of products. However, as we integrate more products this approach does not scale. In addition, this approach relies on humans repeating the same tasks over and over again with high precision. Humans tend to make errors. Computers are much better at executing the same steps over and over again.

The purpose of this issue is to create a test suite which makes it easy to initialize all system tests. There should be a single Gradle task executing and running all the system tests. Additionally, the result of the system tests should be in the JUnit test format, to make it easy to identify issues in tests which are not working.

The main objective is to be able to test SecHub with real products in an int (preview) environment. It is important to be able to test the same test scenarios in an automated and repeatable way.

Jeremias Eppler jeremias.eppler@daimler.com, Daimler TSS GmbH, imprint

Jeeppler commented 3 years ago

Libraries we could use for system tests and REST Assured for REST API testing and Selenium for the WebUI.

A behavior driven testing approach for system tests is probably favorable, as behavior driven testing focuses more on use cases.