search

secureCodeBox / engine

Part of the deprecated secureCodeBox v1, see secureCodeBox/secureCodeBox Repo for v2
Apache License 2.0
8 stars 7 forks source link

Feature/elasticsearch authentication #164

Open rseedorff opened 4 years ago

rseedorff commented 4 years ago

First implementation of https://github.com/secureCodeBox/secureCodeBox/issues/121

J12934 commented 4 years ago

Have you tested out the compatability of the changes with both elasticsearch 6.x and 7.x?

rseedorff commented 4 years ago

I testet this PR with elasticsearch v.7.x but not with v.6.x

J12934 commented 4 years ago

Tried it out with kibana 6.x seems to be crashing while trying to look for the dashboards/ kibana index, see logs below.

Error Logs ```log 2020-04-20 10:56:50.008 INFO 6 --- [aTaskExecutor-2] i.s.p.e.ElasticSearchPersistenceProvider : Initializing ElasticSearchPersistenceProvider 2020-04-20 10:56:50.018 INFO 6 --- [aTaskExecutor-2] i.s.p.e.ElasticSearchPersistenceProvider : No elasticsearch authentication configured. Trying to connect without authentication 2020-04-20 10:56:51.361 WARN 6 --- [aTaskExecutor-2] org.elasticsearch.client.RestClient : request [PUT http://persistence-elasticsearch:9200/securecodebox_localhost_2020-04-20?master_timeout=30s&include_type_name=true&timeout=30s] returned 1 warnings: [299 Elasticsearch-6.8.6-3d9f765 "the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template"] 2020-04-20 10:56:52.303 ERROR 6 --- [aTaskExecutor-2] i.s.s.delegate.SummaryGeneratorDelegate : Unexpected Error while trying to init a persistence provider! org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=illegal_argument_exception, reason=request [/.kibana/_search] contains unrecognized parameter: [ccs_minimize_roundtrips]] at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[elasticsearch-persistenceprovider-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1793) ~[elasticsearch-persistenceprovider-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1770) ~[elasticsearch-persistenceprovider-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1527) ~[elasticsearch-persistenceprovider-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484) ~[elasticsearch-persistenceprovider-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454) ~[elasticsearch-persistenceprovider-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:970) ~[elasticsearch-persistenceprovider-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at io.securecodebox.persistence.elasticsearch.ElasticSearchPersistenceProvider.initializeKibana(ElasticSearchPersistenceProvider.java:467) ~[elasticsearch-persistenceprovider-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at io.securecodebox.persistence.elasticsearch.ElasticSearchPersistenceProvider.init(ElasticSearchPersistenceProvider.java:155) ~[elasticsearch-persistenceprovider-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at io.securecodebox.persistence.elasticsearch.ElasticSearchPersistenceProvider.persist(ElasticSearchPersistenceProvider.java:224) ~[elasticsearch-persistenceprovider-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at io.securecodebox.scanprocess.delegate.SummaryGeneratorDelegate.persist(SummaryGeneratorDelegate.java:90) [sdk-0.0.1-SNAPSHOT.jar!/:na] at io.securecodebox.scanprocess.delegate.SummaryGeneratorDelegate.execute(SummaryGeneratorDelegate.java:72) [sdk-0.0.1-SNAPSHOT.jar!/:na] at org.camunda.bpm.engine.impl.bpmn.delegate.JavaDelegateInvocation.invoke(JavaDelegateInvocation.java:39) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.delegate.DelegateInvocation.proceed(DelegateInvocation.java:57) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.delegate.DefaultDelegateInterceptor.handleInvocationInContext(DefaultDelegateInterceptor.java:90) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.delegate.DefaultDelegateInterceptor.handleInvocation(DefaultDelegateInterceptor.java:62) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.bpmn.behavior.ServiceTaskDelegateExpressionActivityBehavior$3.call(ServiceTaskDelegateExpressionActivityBehavior.java:117) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.bpmn.behavior.ServiceTaskDelegateExpressionActivityBehavior$3.call(ServiceTaskDelegateExpressionActivityBehavior.java:101) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.bpmn.behavior.AbstractBpmnActivityBehavior.executeWithErrorPropagation(AbstractBpmnActivityBehavior.java:110) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.bpmn.behavior.ServiceTaskDelegateExpressionActivityBehavior.performExecution(ServiceTaskDelegateExpressionActivityBehavior.java:126) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.bpmn.behavior.TaskActivityBehavior.execute(TaskActivityBehavior.java:68) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityExecute$2.callback(PvmAtomicOperationActivityExecute.java:60) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityExecute$2.callback(PvmAtomicOperationActivityExecute.java:49) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.continueIfExecutionDoesNotAffectNextOperation(PvmExecutionImpl.java:1988) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityExecute.execute(PvmAtomicOperationActivityExecute.java:41) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityExecute.execute(PvmAtomicOperationActivityExecute.java:30) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:95) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:127) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:107) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:82) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:640) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:614) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl$6.callback(PvmExecutionImpl.java:1927) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl$6.callback(PvmExecutionImpl.java:1924) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.continueExecutionIfNotCanceled(PvmExecutionImpl.java:1994) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.dispatchDelayedEventsAndPerformOperation(PvmExecutionImpl.java:1943) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.dispatchDelayedEventsAndPerformOperation(PvmExecutionImpl.java:1924) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationTransitionNotifyListenerStart.eventNotificationsCompleted(PvmAtomicOperationTransitionNotifyListenerStart.java:60) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationTransitionNotifyListenerStart.eventNotificationsCompleted(PvmAtomicOperationTransitionNotifyListenerStart.java:30) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.core.operation.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:67) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:95) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:127) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:107) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:82) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:72) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:649) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:624) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.core.operation.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:60) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:95) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:127) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:107) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:82) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:72) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:649) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:624) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.core.operation.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:60) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:95) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:127) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:107) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:82) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:640) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:614) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationTransitionCreateScope.scopeCreated(PvmAtomicOperationTransitionCreateScope.java:37) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationCreateScope.execute(PvmAtomicOperationCreateScope.java:53) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationCreateScope.execute(PvmAtomicOperationCreateScope.java:27) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:95) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:127) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:114) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext$1.call(CommandInvocationContext.java:98) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext$1.call(CommandInvocationContext.java:96) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.context.ProcessApplicationClassloaderInterceptor.call(ProcessApplicationClassloaderInterceptor.java:47) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.application.AbstractProcessApplication.execute(AbstractProcessApplication.java:117) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.application.AbstractProcessApplication.execute(AbstractProcessApplication.java:128) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.context.Context.executeWithinProcessApplication(Context.java:194) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:96) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:82) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:72) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.jobexecutor.AsyncContinuationJobHandler.execute(AsyncContinuationJobHandler.java:80) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.jobexecutor.AsyncContinuationJobHandler.execute(AsyncContinuationJobHandler.java:39) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.persistence.entity.JobEntity.execute(JobEntity.java:138) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:102) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:39) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:27) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:106) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:45) [camunda-engine-spring-7.10.0.jar!/:7.10.0] at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) [spring-tx-5.2.2.RELEASE.jar!/:5.2.2.RELEASE] at org.camunda.bpm.engine.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:43) [camunda-engine-spring-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:69) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:50) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:43) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.executeJob(ExecuteJobsRunnable.java:91) [camunda-engine-7.10.0.jar!/:7.10.0] at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:60) [camunda-engine-7.10.0.jar!/:7.10.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212] Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://persistence-elasticsearch:9200], URI [/.kibana/_search?pre_filter_shard_size=128&typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 400 Bad Request] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"request [/.kibana/_search] contains unrecognized parameter: [ccs_minimize_roundtrips]"}],"type":"illegal_argument_exception","reason":"request [/.kibana/_search] contains unrecognized parameter: [ccs_minimize_roundtrips]"},"status":400} at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283) ~[elasticsearch-persistenceprovider-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261) ~[elasticsearch-persistenceprovider-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-persistenceprovider-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1514) ~[elasticsearch-persistenceprovider-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] ... 92 common frames omitted ```

Setting SECURECODEBOX_PERSISTENCE_ELASTICSEARCH_KIBANA_INITIALIZE to false doesn't seem to help, as it is already crashing at the indexExists check:

https://github.com/secureCodeBox/engine/blob/058de94418d575f4208efab8d79d80e6f3b11896/scb-persistenceproviders/elasticsearch-persistenceprovider/src/main/java/io/securecodebox/persistence/elasticsearch/ElasticSearchPersistenceProvider.java#L160-L166

rseedorff commented 4 years ago

That makes sense... The kibana initialize step is seperated from the elasticsearch index initialize step. A second config option could help us here to seperate this: SECURECODEBOX_PERSISTENCE_ELASTICSEARCH_ELASTICSEARCH_INITIALIZE=true|false.

An we could think about supporting different versions: SECURECODEBOX_PERSISTENCE_ELASTICSEARCH_VERSION = 6|7|8