Resilience for terminated JVM but ongoing scans

[de-jcup]

Defintions

• Zombie job
  means a job where execution process has been interrupted by a JVM crash, so JOB will never be updated with results from security product. So caller would wait "forever" - like "Waiting for Godot"

Usecases

Restart zombie job

As an administrator I want to be able to restart a job supposed to be a "Zombie". Existing product results shall be re-fetched and job result updated as fast as possible.

Restart (hard) zombie job

As an administrator I want to be able to restart a job supposed to be a "Zombie" in a hard way, means complete job process is restarted. Existing former product results will be dropped.

Situation details

• user starts job -e.g. trigger sechub client scan (sync)
• job is executed by scheduler, so a spring batch operation runs leading to adapter execution where product adapter has started a scan and waits for product result
• now ... sechub instance is terminated (e.g. Kubernetes decides to terminate POD) the JVM has been terminated/hard crashed , so no event handling any longer running actions just not finished
• the security product does finish its job
• but the sechub job will never been updated, because the running spring batch job jas been killed and no new one has been started
• we have a "zombie" and user /build server waits infinite...

Solution

Adapter resilience via meta data

• We provide meta data for every product result DB entry
• Product execution gives automatically a callback handler to adapter start. With this callback it'S possible for adapers to read and write adapter relevant meta data.
• Adapters do handle resilience at adapter level: E.g. when scanId is already given the adapter will not trigger a new scan, but will just fech/check for results of this scan Id .

Access

Restart and Restart (hard) will be done as usual by REST calls. Can only be done by SUPER_ADMINrole

Audit

Every restart will show admin ID in autdit log. Also a notification to admins about the restart will be send on success of restart

Integration testing

Formerly mocked adapters did not really do things. But wit introduction of meta data every mocked adaper should at least read/write some dummy data - to simulate the persistence as well. Maybe even a mocked adapter should write same meta data identifier/data as origin one.

What happens when having started a restart accidently to a finished job ?

When a job has been finished, we do not provide any restart! It will give back an HTTP error and (maybe) a notification mail.

What happens when having race condition ?

E.g. A job is not really a zombie, but restart was triggered.

Solution: We try to stop the other running process - but if this would not work we don't bother. Just let the second scan job write again to DB - reason: Result is same, so it doesn't matter, only end time will be changed.

[de-jcup]

We must add a new column to store the product scan id.

[de-jcup]

Started new feature branch - because there were too much changes inside currently not necessary (and producing to much noise...)

[de-jcup]

Still todo:

• [x] fix current failing unit test cases / fix bugs
• [ ] add unit test cases for checkmarx adapter about fast response behaviour and meta data usage
• [ ] integrate special retry behaviour for netsparker adapter (meta data...)
• [ ] integrate special retry behaviour for nessus adapter (meta data...)
• [x] provide REST API for doing a restart of an existing job (same as normal doing for Spring Boot Batchjob, but without new Job UUID creation)
• [ ] add retry action inside developer admin UI
• [x] provide Integration test for checkmarx scan
  • trigger a code scan
  • check meta data created in way as expected (file upload, scan id, report id)
  • reset scheduler status back to running to simulate JVM crash
  • retry by API (must trigger adapter, scan again + upgrade result in scheduler again)
    • [ ] Former integration test will NOT test the real adapter resilience behaviour, so we should test this in a spring boot test with database etc.

[de-jcup]

Merged by https://github.com/Daimler/sechub/pull/221 so can close