search

mercedes-benz / sechub

SecHub provides a central API to test software with different security tools.
https://mercedes-benz.github.io/sechub/
MIT License
259 stars 61 forks source link

Simple resilience for problems with product execution #100

Closed de-jcup closed 4 years ago

de-jcup commented 4 years ago

Common

A product executor should handle resilience for its own. Currently no usage of other libraries (E.g. Hystrix ) is planned and there should be a simple approach used. (Maybe we use Hystrix in future, but not this time).

Checkmarx product executor

E.g. checkmarx seems to have a problem sometimes when multiple scan agents are scanning same project same time (upload source zip - e.g. when having 2 differnt branches like master and develop) what can be a valid case.

Sometimes we got a 400 Bad Request as result, doing same operation again a little bit time shifted does work. So instead of just breaking here the Checkmarx executor implementation should just wait some time, and retry in this case (with maximum loop count to avoid intinfite retries)

Details

At server side we got following output

com.daimler.sechub.sharedkernel.execution.SecHubExecutionException: CHECKMARX execution failed.__[SECHUB-UID:87f6188b-00d1-4985-bccb-a70f3b11aaa0]__
    at com.daimler.sechub.domain.scan.product.AbstractInstallSetupProductExecutor.execute(AbstractInstallSetupProductExecutor.java:74) ~[sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.AbstractProductExecutionService.execute(AbstractProductExecutionService.java:77) [sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.AbstractProductExecutionService.executeAndPersistResults(AbstractProductExecutionService.java:116) [sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.AbstractProductExecutionService.executeProductsAndStoreResults(AbstractProductExecutionService.java:64) [sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.AbstractProductExecutionService$$FastClassBySpringCGLIB$$2ec9ff9d.invoke(<generated>) [sechub-scan-0.0.0.jar!/:na]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) [spring-core-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:685) [spring-aop-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at com.daimler.sechub.domain.scan.product.CodeScanProductExecutionServiceImpl$$EnhancerBySpringCGLIB$$354d01e7.executeProductsAndStoreResults(<generated>) [sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.ScanService.executeScan(ScanService.java:101) [sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.ScanService.startScan(ScanService.java:72) [sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.ScanService.receiveSynchronMessage(ScanService.java:154) [sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.sharedkernel.messaging.DomainMessageService.sendSynchron(DomainMessageService.java:138) [sechub-shared-kernel-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.schedule.batch.ScanExecutionTasklet.executeSafe(ScanExecutionTasklet.java:67) [sechub-schedule-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.schedule.batch.ScanExecutionTasklet.execute(ScanExecutionTasklet.java:43) [sechub-schedule-0.0.0.jar!/:na]
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) [spring-tx-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) [spring-batch-infrastructure-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) [spring-batch-infrastructure-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) [spring-batch-infrastructure-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:68) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:68) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:137) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:319) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:147) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]
Caused by: com.daimler.sechub.adapter.AdapterException: __[SECHUB-UID:87f6188b-00d1-4985-bccb-a70f3b11aaa0]__ CheckmarxAdapterV1:Was not able to perform scan!
    at com.daimler.sechub.adapter.AdapterException.asAdapterException(AdapterException.java:47) ~[sechub-adapter-0.0.0.jar!/:na]
    at com.daimler.sechub.adapter.AbstractAdapter.asAdapterException(AbstractAdapter.java:30) ~[sechub-adapter-0.0.0.jar!/:na]
    at com.daimler.sechub.adapter.checkmarx.CheckmarxAdapterV1.start(CheckmarxAdapterV1.java:53) ~[sechub-adapter-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.adapter.checkmarx.CheckmarxAdapterV1.start(CheckmarxAdapterV1.java:27) ~[sechub-adapter-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.checkmarx.CheckmarxProductExecutor.executeWithAdapter(CheckmarxProductExecutor.java:78) ~[sechub-scan-product-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.checkmarx.CheckmarxProductExecutor.executeWithAdapter(CheckmarxProductExecutor.java:29) ~[sechub-scan-product-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.AbstractInstallSetupProductExecutor.executeAdapterWhenTargetTypeSupported(AbstractInstallSetupProductExecutor.java:180) ~[sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.AbstractInstallSetupProductExecutor.execute(AbstractInstallSetupProductExecutor.java:154) ~[sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.AbstractInstallSetupProductExecutor.execute(AbstractInstallSetupProductExecutor.java:66) ~[sechub-scan-0.0.0.jar!/:na]
    ... 32 common frames omitted
Caused by: org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request
    at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:79) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:123) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:102) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:785) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:743) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:677) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:586) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at com.daimler.sechub.adapter.checkmarx.support.CheckmarxScanSupport.triggerNewEntryInQueue(CheckmarxScanSupport.java:95) ~[sechub-adapter-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.adapter.checkmarx.support.CheckmarxScanSupport.startNewScan(CheckmarxScanSupport.java:42) ~[sechub-adapter-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.adapter.checkmarx.CheckmarxAdapterV1.uploadSourceCodeAndStartScan(CheckmarxAdapterV1.java:77) ~[sechub-adapter-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.adapter.checkmarx.CheckmarxAdapterV1.handleUploadSourceCodeAndStartScan(CheckmarxAdapterV1.java:61) ~[sechub-adapter-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.adapter.checkmarx.CheckmarxAdapterV1.start(CheckmarxAdapterV1.java:46) ~[sechub-adapter-checkmarx-0.0.0.jar!/:na]
de-jcup commented 4 years ago

Hmm... The problem seems still to occurre:

2019-12-05 09:03:40.420 ERROR 6 --- [c-job-launcher3] .s.d.s.p.AbstractProductExecutionService : Product executor failed:CHECKMARX __[SECHUB-UID:7bdb6d05-0953-4797-ac1d-52ccceaab405]__
com.daimler.sechub.sharedkernel.execution.SecHubExecutionException: CHECKMARX execution failed.__[SECHUB-UID:7bdb6d05-0953-4797-ac1d-52ccceaab405]__
    at com.daimler.sechub.domain.scan.product.AbstractInstallSetupProductExecutor.execute(AbstractInstallSetupProductExecutor.java:74) ~[sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.AbstractProductExecutionService.execute(AbstractProductExecutionService.java:77) [sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.AbstractProductExecutionService.executeAndPersistResults(AbstractProductExecutionService.java:116) [sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.AbstractProductExecutionService.executeProductsAndStoreResults(AbstractProductExecutionService.java:64) [sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.AbstractProductExecutionService$$FastClassBySpringCGLIB$$2ec9ff9d.invoke(<generated>) [sechub-scan-0.0.0.jar!/:na]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) [spring-core-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:685) [spring-aop-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at com.daimler.sechub.domain.scan.product.CodeScanProductExecutionServiceImpl$$EnhancerBySpringCGLIB$$a9bf867b.executeProductsAndStoreResults(<generated>) [sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.ScanService.executeScan(ScanService.java:101) [sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.ScanService.startScan(ScanService.java:72) [sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.ScanService.receiveSynchronMessage(ScanService.java:154) [sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.sharedkernel.messaging.DomainMessageService.sendSynchron(DomainMessageService.java:138) [sechub-shared-kernel-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.schedule.batch.ScanExecutionTasklet.executeSafe(ScanExecutionTasklet.java:67) [sechub-schedule-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.schedule.batch.ScanExecutionTasklet.execute(ScanExecutionTasklet.java:43) [sechub-schedule-0.0.0.jar!/:na]
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:407) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:331) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) [spring-tx-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:273) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:375) [spring-batch-infrastructure-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) [spring-batch-infrastructure-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:145) [spring-batch-infrastructure-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:258) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:208) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:68) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:68) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:137) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:319) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:147) [spring-batch-core-4.2.0.RELEASE.jar!/:4.2.0.RELEASE]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]
Caused by: com.daimler.sechub.adapter.AdapterException: __[SECHUB-UID:7bdb6d05-0953-4797-ac1d-52ccceaab405]__ CheckmarxAdapterV1:Was not able to perform scan!
    at com.daimler.sechub.adapter.AdapterException.asAdapterException(AdapterException.java:47) ~[sechub-adapter-0.0.0.jar!/:na]
    at com.daimler.sechub.adapter.AbstractAdapter.asAdapterException(AbstractAdapter.java:30) ~[sechub-adapter-0.0.0.jar!/:na]
    at com.daimler.sechub.adapter.checkmarx.CheckmarxAdapterV1.start(CheckmarxAdapterV1.java:53) ~[sechub-adapter-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.adapter.checkmarx.CheckmarxAdapterV1.start(CheckmarxAdapterV1.java:27) ~[sechub-adapter-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.checkmarx.CheckmarxProductExecutor.lambda$executeWithAdapter$0(CheckmarxProductExecutor.java:89) ~[sechub-scan-product-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.sharedkernel.resilience.ResilientActionExecutor.executeResilient(ResilientActionExecutor.java:41) ~[sechub-shared-kernel-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.checkmarx.CheckmarxProductExecutor.executeWithAdapter(CheckmarxProductExecutor.java:69) ~[sechub-scan-product-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.checkmarx.CheckmarxProductExecutor.executeWithAdapter(CheckmarxProductExecutor.java:30) ~[sechub-scan-product-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.AbstractInstallSetupProductExecutor.executeAdapterWhenTargetTypeSupported(AbstractInstallSetupProductExecutor.java:180) ~[sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.AbstractInstallSetupProductExecutor.execute(AbstractInstallSetupProductExecutor.java:154) ~[sechub-scan-0.0.0.jar!/:na]
    at com.daimler.sechub.domain.scan.product.AbstractInstallSetupProductExecutor.execute(AbstractInstallSetupProductExecutor.java:66) ~[sechub-scan-0.0.0.jar!/:na]
    ... 32 common frames omitted
Caused by: org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request
    at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:79) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:123) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:102) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:785) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:743) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:677) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:586) ~[spring-web-5.2.0.RELEASE.jar!/:5.2.0.RELEASE]
    at com.daimler.sechub.adapter.checkmarx.support.CheckmarxScanSupport.triggerNewEntryInQueue(CheckmarxScanSupport.java:95) ~[sechub-adapter-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.adapter.checkmarx.support.CheckmarxScanSupport.startNewScan(CheckmarxScanSupport.java:42) ~[sechub-adapter-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.adapter.checkmarx.CheckmarxAdapterV1.uploadSourceCodeAndStartScan(CheckmarxAdapterV1.java:77) ~[sechub-adapter-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.adapter.checkmarx.CheckmarxAdapterV1.handleUploadSourceCodeAndStartScan(CheckmarxAdapterV1.java:61) ~[sechub-adapter-checkmarx-0.0.0.jar!/:na]
    at com.daimler.sechub.adapter.checkmarx.CheckmarxAdapterV1.start(CheckmarxAdapterV1.java:46) ~[sechub-adapter-checkmarx-0.0.0.jar!/:na]
    ... 40 common frames omitted