problem in connecting to elasticsearch

Amir-Ameri commented 5 years ago

Hi I'm new in wso2 EI and elasticsearch. I tired to connect EI 6.4.0 to elasticserach cluster V. 6.6.1 without X-Pack, but i didn't succeed. I know that elasticserach cluster is OK! Thanks for helping.

here is my carbon config. carbon.txt And I get this error in wso2 EI log: [2019-03-31 05:50:09,880] [-1234] [] [localhost-startStop-1] ERROR {org.wso2.custom.elastic.publisher.observer.ElasticMediationFlowObserver} - Can not connect to any Elasticsearch nodes. Please give correct configurations, run Elasticsearch and restart WSO2-EI.

ThejanRupasinghe commented 5 years ago

Carbon config looks ok. This error is because there are no nodes connected to the client. This can happen due to,

Incorrect ClusterName provided. (cluster.name as in the elasticsearch.yml configuration file)
Given elasticsearch cluster is not running or not accessible from the EI server.
Wrong or no access credentials are provided to a protected cluster.

Please verify that the above problems are not there.

arabsoheyl commented 5 years ago

I have the same problem. I check cluster-name and connection between EI and Elasticsearch and I don't see any problem, also, FYI, I don't use access credentials in Elasticsearch. 1- is it possible that my problem is related to Elasticsearch version? My Elasticsearch version is 6.6.1

2- is it necessary to use port 9300 to connect to Elasticseach?? port 9300 is for communication between nodes and 9200 is for REST. your program connecting to Elastic in which type?? 3- is it necessary to use SSL in communication between EI and Elasticsearch? my config in wso2ei/6.4.0/conf/synapse.properties is:

# Configuration to enable mediation flow analytics
mediation.flow.statistics.enable=true
mediation.flow.statistics.tracer.collect.payloads=true
mediation.flow.statistics.tracer.collect.properties=true
mediation.flow.statistics.event.consume.interval=1000
mediation.flow.statistics.event.clean.interval=15000

# Configuration to enable statistics globally irrespective of the individual artifact level setting
mediation.flow.statistics.collect.all=true

# Script Mediator Pool (impatcts only external scripts)
#synapse.script.mediator.pool.size=15

My config in wso2ei/6.4.0/conf/carbon.xml is:

<Server>
.....
...
..
   <MediationFlowStatisticConfig>
       <AnalyticPublishingDisable>true</AnalyticPublishingDisable>
       <Observers>
           org.wso2.custom.elastic.publisher.observer.ElasticMediationFlowObserver
       </Observers>
       <ElasticObserver>
           <Host>192.168.95.42</Host>
           <Port>9200</Port>
           <ClusterName>mycluster</ClusterName>
           <BufferSize>5000</BufferSize>
           <BulkSize>500</BulkSize>
           <BulkCollectingTimeOut>5000</BulkCollectingTimeOut>
           <BufferEmptySleepTime>1000</BufferEmptySleepTime>
           <NoNodesSleepTime>5000</NoNodesSleepTime>
       </ElasticObserver>
   </MediationFlowStatisticConfig>
</Server>

My config in elasticsearch.yml is:


# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: mycluster

#
# Node Type
#
node.master: true
node.data: true
node.ingest: true
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: elk01

#
network.host: 192.168.95.42
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
action.auto_create_index: .filebeat*,filebeat*,.logstash*,logstash*,kibana*,.kibana*,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*, esb*, *

and this is your program log: wso2-ei-analytics-elk.txt

ThejanRupasinghe commented 5 years ago

@arabsoheyl 1 - Yes. It can be the problem. This was tested through Elastic Stack 5.4.3 2 - This uses Transport Client to connect to Elasticsearch, which uses node-to-node communication. So the port should be 9300 in carbon.xml. 3 - No. It is not necessary to use SSL between EI and Elasticsearch.

@amerireza can you please create a PR for this fix?

ThejanRupasinghe commented 5 years ago

@Amir-Ameri @arabsoheyl Latest code in master branch works for EI 6.4.0 and Elasticsearch 6.6.1, with the fix from PR https://github.com/ThejanRupasinghe/ei-elastic-custom-publisher/pull/5 . Please verify.

arabsoheyl commented 5 years ago

@Amir-Ameri @arabsoheyl Latest code in master branch works for EI 6.4.0 and Elasticsearch 6.6.1, with the fix from PR #5 . Please verify.

it works for me(on Elasticsearch 6.6.1). thanks

ThejanRupasinghe / ei-elastic-custom-publisher

problem in connecting to elasticsearch #3