Client request error: connect ECONNREFUSED 127.0.0.1:3030

[yangcaixing]

Hi all, I encountered an error after install the elastalert-kibana-plugin as below, I have pre-installed the ElastAlert Server with this command "pip install "elastalert>=0.2.0b", but I don't know if need some configuration for the elastalert server, I don't use docker to setup the ELK and elastalert, so if anyone have the same issue as me, I'm stuck in it, looking forward your help, thanks in advance.

[yangcaixing]

and after I installed the elastalert server with docker image, it also display the bellow error, I also add configuration"elastalert-kibana-plugin.serverHost: 127.0.0.1, elastalert-kibana-plugin.serverPort: 9200" to the /etc/kibana/kibana.yml, if anyone can help me ,thanks a lot.

[robincher]

Do you have any logs?

I got a similar error but that was because my container image wasn't able to start properly due to trendmicro dsagent.exe having some issue.

[yangcaixing]

Do you have any logs?

I got a similar error but that was because my container image wasn't able to start properly due to trendmicro dsagent.exe having some issue.

I don't find the docker logs, but it seems the docker container run up and immediately closed, so kibana cannot connect port 3033, it will display the error"502-Bad Gateway", and now I still not find the solution.

[robincher]

Hi,

Are you able to see some system logs for docker? Probably it will show the reason why the container wasn't able to start properly.

[yangcaixing]

Hi,

Are you able to see some system logs for docker? Probably it will show the reason why the container wasn't able to start properly.

Hi Rob, I got bellow error, could you please help check ,thanks.

@bitsensor/elastalert@2.0.1 start /opt/elastalert-server sh ./scripts/start.sh

03:00:02.566Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json. 03:00:02.568Z INFO elastalert-server: Config: Proceeding to look for normal config file. 03:00:02.570Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config. 03:00:02.601Z INFO elastalert-server: Router: Listening for GET request on /. 03:00:02.601Z INFO elastalert-server: Router: Listening for GET request on /status. 03:00:02.602Z INFO elastalert-server: Router: Listening for GET request on /status/control/:action. 03:00:02.602Z INFO elastalert-server: Router: Listening for GET request on /status/errors. 03:00:02.602Z INFO elastalert-server: Router: Listening for GET request on /rules. 03:00:02.603Z INFO elastalert-server: Router: Listening for GET request on /rules/:id. 03:00:02.603Z INFO elastalert-server: Router: Listening for POST request on /rules/:id. 03:00:02.603Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id. 03:00:02.603Z INFO elastalert-server: Router: Listening for GET request on /templates. 03:00:02.603Z INFO elastalert-server: Router: Listening for GET request on /templates/:id. 03:00:02.604Z INFO elastalert-server: Router: Listening for POST request on /templates/:id. 03:00:02.604Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id. 03:00:02.604Z INFO elastalert-server: Router: Listening for POST request on /test. 03:00:02.604Z INFO elastalert-server: Router: Listening for GET request on /config. 03:00:02.604Z INFO elastalert-server: Router: Listening for POST request on /config. 03:00:02.604Z INFO elastalert-server: Router: Listening for POST request on /download. 03:00:02.604Z INFO elastalert-server: Router: Listening for GET request on /metadata/:type. 03:00:02.604Z INFO elastalert-server: Router: Listening for GET request on /mapping/:index. 03:00:02.604Z INFO elastalert-server: Router: Listening for POST request on /search/:index. 03:00:02.607Z INFO elastalert-server: ProcessController: Starting ElastAlert 03:00:02.607Z INFO elastalert-server: ProcessController: Creating index 03:00:14.583Z ERROR elastalert-server: ProcessController: Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/opt/elastalert/elastalert/create_index.py", line 275, in main() File "/opt/elastalert/elastalert/create_index.py", line 127, in main esversion = es.info()["version"]["number"] File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped return func(*args, params=params, **kwargs) File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/init.py", line 259, in info return self.transport.perform_request("GET", "/", params=params) File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/transport.py", line 353, in perform_request timeout=timeout, File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 127, in perform_request raise ConnectionError("N/A", str(e), e) elasticsearch.exceptions.ConnectionError: ConnectionError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb5f5ba110>: Failed to establish a new connection: [Errno 111] Connection refused',))) caused by: ConnectionError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fdb5f5ba110>: Failed to establish a new connection: [Errno 111] Connection refused',)))

03:00:14.584Z ERROR elastalert-server: ProcessController: Index create exited with code 1 03:00:14.584Z WARN elastalert-server: ProcessController: ElastAlert will start but might not be able to save its data! 03:00:14.585Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none] 03:00:14.597Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 49) 03:00:14.598Z INFO elastalert-server: Server: Server listening on port 3030 03:00:14.599Z INFO elastalert-server: Server: Websocket listening on port 3333 03:00:14.599Z INFO elastalert-server: Server: Server started 03:00:15.431Z ERROR elastalert-server: ProcessController: WARNING:elasticsearch:GET http://localhost:9200/ [status:N/A request:0.002s] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 111, in perform_request response = self.session.send(prepared_request, send_kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/sessions.py", line 646, in send r = adapter.send(request, kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe8653fde90>: Failed to establish a new connection: [Errno 111] Connection refused',))

03:00:16.431Z ERROR elastalert-server: ProcessController: WARNING:elasticsearch:GET http://localhost:9200/ [status:N/A request:0.001s] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 111, in perform_request response = self.session.send(prepared_request, send_kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/sessions.py", line 646, in send r = adapter.send(request, kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe86524c150>: Failed to establish a new connection: [Errno 111] Connection refused',))

03:00:19.438Z ERROR elastalert-server: ProcessController: WARNING:elasticsearch:GET http://localhost:9200/ [status:N/A request:0.003s] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 111, in perform_request response = self.session.send(prepared_request, send_kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/sessions.py", line 646, in send r = adapter.send(request, kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe86524c1d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

03:00:26.447Z ERROR elastalert-server: ProcessController: WARNING:elasticsearch:GET http://localhost:9200/ [status:N/A request:0.002s] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 111, in perform_request response = self.session.send(prepared_request, send_kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/sessions.py", line 646, in send r = adapter.send(request, kwargs) File "/usr/lib/python2.7/site-packages/requests-2.21.0-py2.7.egg/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) ConnectionError: HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe86524c110>: Failed to establish a new connection: [Errno 111] Connection refused',))

03:00:26.448Z ERROR elastalert-server: ProcessController: Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/opt/elastalert/elastalert/elastalert.py", line 1929, in

03:00:26.448Z ERROR elastalert-server: ProcessController: sys.exit(main(sys.argv[1:])) File "/opt/elastalert/elastalert/elastalert.py", line 1925, in main

03:00:26.449Z ERROR elastalert-server: ProcessController: client.start() File "/opt/elastalert/elastalert/elastalert.py", line 1106, in start

03:00:26.449Z ERROR elastalert-server: ProcessController: self.run_all_rules() File "/opt/elastalert/elastalert/elastalert.py", line 1158, in run_all_rules

03:00:26.450Z ERROR elastalert-server: ProcessController: self.send_pending_alerts() File "/opt/elastalert/elastalert/elastalert.py", line 1534, in send_pending_alerts

03:00:26.450Z ERROR elastalert-server: ProcessController: pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit) File "/opt/elastalert/elastalert/elastalert.py", line 1517, in find_recent_pending_alerts

03:00:26.451Z ERROR elastalert-server: ProcessController: if self.is_atleastfive(): File "/opt/elastalert/elastalert/elastalert.py", line 159, in is_atleastfive

03:00:26.451Z ERROR elastalert-server: ProcessController: return int(self.es_version.split(".")[0]) >= 5 File "/opt/elastalert/elastalert/elastalert.py", line 155, in es_version

03:00:26.451Z ERROR elastalert-server: ProcessController: self._es_version = self.get_version() File "/opt/elastalert/elastalert/elastalert.py", line 149, in get_version info = self.writeback_es.info() File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped

03:00:26.451Z ERROR elastalert-server: ProcessController: return func(*args, params=params, **kwargs) File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/init.py", line 259, in info

03:00:26.451Z ERROR elastalert-server: ProcessController: return self.transport.perform_request("GET", "/", params=params) File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/transport.py", line 353, in perform_request

03:00:26.451Z ERROR elastalert-server: ProcessController: timeout=timeout, File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/connection/http_requests.py", line 127, in perform_request

03:00:26.451Z ERROR elastalert-server: ProcessController: raise ConnectionError("N/A", str(e), e) elasticsearch.exceptions.ConnectionError 03:00:26.451Z ERROR elastalert-server: ProcessController: : ConnectionError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe86524c110>: Failed to establish a new connection: [Errno 111] Connection refused',))) caused by: ConnectionError(HTTPConnectionPool(host='localhost', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe86524c110>: Failed to establish a new connection: [Errno 111] Connection refused',)))

03:00:26.490Z ERROR elastalert-server: ProcessController: ElastAlert exited with code 1 03:00:26.490Z INFO elastalert-server: Server: Stopping server 03:00:26.490Z INFO elastalert-server: ProcessController: ElastAlert is not running 03:00:26.490Z INFO elastalert-server: Server: Server stopped. Bye!

[robincher]

It is trying to connect to elasticsearch at localhost:9200. Where is your elsaticsearch hosted? You might need to update the config file to the exact elasticsearch hostname.

[yangcaixing]

It is trying to connect to elasticsearch at localhost:9200. Where is your elsaticsearch hosted? You might need to update the config file to the exact elasticsearch hostname.

Hi rob,
My elasticsearch host is localhost, same host with docker, the config file default also localhost, so I don't know why elastialert can not connect elasticsearch port 9200.

[robincher]

Is your elasticsearch up and running?

curl -vvv http://localhost:9200

Do you have any anti-virus or other programs using port 9200?

[yangcaixing]

of course, it's up . My ELK is v.7.0.1, and in elasticseach.yml ,network.host must be set to 127.0.0.1 or localhost, otherwise elasticseach.service will also failed. I don't know if it have any impact to the issue.

Set the bind address to a specific IP (IPv4 or IPv6):

# network.host: 127.0.0.1 #

Set a custom port for HTTP:

# http.port: 9200 the docker will try to connect elastiseach 9200 about 35 seconds, then failed. root@elk:~/elastalert# docker container ls -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4480117ab0a5 bitsensor/elastalert:latest "npm start" 38 seconds ago Up 34 seconds 0.0.0.0:3030->3030/tcp, 0.0.0.0:3333->3333/tcp elastalert root@elk:~/elastalert# docker container ls -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4480117ab0a5 bitsensor/elastalert:latest "npm start" 39 seconds ago Exited (0) 2 seconds ago elastalert

root@elk:~/elastalert# curl -vvv http://localhost:9200

• Rebuilt URL to: http://localhost:9200/
• Trying ::1...
• TCP_NODELAY set
• connect to ::1 port 9200 failed: Connection refused
• Trying 127.0.0.1...
• TCP_NODELAY set
• Connected to localhost (127.0.0.1) port 9200 (#0)

  GET / HTTP/1.1 Host: localhost:9200 User-Agent: curl/7.58.0 Accept: /

  < HTTP/1.1 200 OK < content-type: application/json; charset=UTF-8 < content-length: 495 < { "name" : "elk", "cluster_name" : "elasticsearch", "cluster_uuid" : "f0ewVq1kR8CEC57OB07yAQ", "version" : { "number" : "7.0.1", "build_flavor" : "default", "build_type" : "deb", "build_hash" : "e4efcb5", "build_date" : "2019-04-29T12:56:03.145736Z", "build_snapshot" : false, "lucene_version" : "8.0.0", "minimum_wire_compatibility_version" : "6.7.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" }

• Connection #0 to host localhost left intact

[robincher]

Can pase your elasticsearch.yml thanks.

[yangcaixing]

here is my elasticsearch.yml:

elasticsearch.txt

[robincher]

Have you tried setting this in elasticsearch.yml?

network.host: 0.0.0.0

[yangcaixing]

yes, I tried, but for elk v7.0.1, if setup network.host:0.0.0.0, elasticsearch.service will always failed, must setup 127.0.0.1 or localhost , you can have a try.

[akamzin]

+1 docker container does not see the elastic running. [Errno 111] Connection refused

[AcidRobot]

This is happening for me as well.

[vvvprabhakar]

+1 same thing

[vvvprabhakar]

pass the docker ports -p 3030:3030 -p 3000:3000

[sprksh]

@vvvprabhakar has that solved your problem? And shall i mount these 2 ports in kibana docker?

[santosh-kore]

same issue here

[santosh-kore]

any updates here?

[kk-stateless]

It's giving me the same error. My setup is outlined below ELK containers on 7.5.2 Elastalert container on 0.2.1 Elastalert-kibana-plugin 7.5.0 (tweaked package.json to point to 7.5.2) Elastalert.yaml and config.json pointed to my server private ip (same host) and connection seems successful as it returns elastic version 7.5.2 in logs

23:05:33.328Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json. 23:05:33.329Z INFO elastalert-server: Config: Proceeding to look for normal config file. 23:05:33.330Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config. 23:05:33.337Z INFO elastalert-server: Router: Listening for GET request on /. 23:05:33.337Z INFO elastalert-server: Router: Listening for GET request on /status. 23:05:33.337Z INFO elastalert-server: Router: Listening for GET request on /status/control/:action. 23:05:33.338Z INFO elastalert-server: Router: Listening for GET request on /status/errors. 23:05:33.338Z INFO elastalert-server: Router: Listening for GET request on /rules. 23:05:33.339Z INFO elastalert-server: Router: Listening for GET request on /rules/:id. 23:05:33.339Z INFO elastalert-server: Router: Listening for POST request on /rules/:id. 23:05:33.339Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id. 23:05:33.340Z INFO elastalert-server: Router: Listening for GET request on /templates. 23:05:33.340Z INFO elastalert-server: Router: Listening for GET request on /templates/:id. 23:05:33.340Z INFO elastalert-server: Router: Listening for POST request on /templates/:id. 23:05:33.340Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id. 23:05:33.340Z INFO elastalert-server: Router: Listening for POST request on /test. 23:05:33.340Z INFO elastalert-server: Router: Listening for GET request on /config. 23:05:33.340Z INFO elastalert-server: Router: Listening for POST request on /config. 23:05:33.340Z INFO elastalert-server: Router: Listening for POST request on /download. 23:05:33.341Z INFO elastalert-server: Router: Listening for GET request on /metadata/:type. 23:05:33.341Z INFO elastalert-server: Router: Listening for GET request on /mapping/:index. 23:05:33.341Z INFO elastalert-server: Router: Listening for POST request on /search/:index. 23:05:33.344Z INFO elastalert-server: ProcessController: Starting ElastAlert 23:05:33.344Z INFO elastalert-server: ProcessController: Creating index 23:05:34.129Z INFO elastalert-server: ProcessController: Elastic Version: 7.5.2 Reading Elastic 6 index mappings: Reading index mapping 'es_mappings/6/silence.json' Reading index mapping 'es_mappings/6/elastalert_status.json' Reading index mapping 'es_mappings/6/elastalert.json' Reading index mapping 'es_mappings/6/past_elastalert.json' Reading index mapping 'es_mappings/6/elastalert_error.json' Index elastalert_status already exists. Skipping index creation.

23:05:34.130Z INFO elastalert-server: ProcessController: Index create exited with code 0 23:05:34.131Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none] 23:05:34.151Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 37) 23:05:34.154Z INFO elastalert-server: Server: Server listening on port 3030 23:05:34.157Z INFO elastalert-server: Server: Websocket listening on port 3333 23:05:34.157Z INFO elastalert-server: Server: Server started

Able to curl on localhost:3030 {"name":"elastalert-server","port":3030,"version":"3.0.0-beta.0"}

Any insights here?

[kk-stateless]

Update - I edited Kibana.yaml inside the container with the following lines and restarted it, it worked! elastalert-kibana-plugin.serverHost: 10.x.x.x elastalert-kibana-plugin.serverPort: 3030

10.x.x.x is my host ip.

[santosh-kore]

worked with setup suggested by @kk-stateless ...Thanks....

[bcisse]

@kk-stateless @santosh-kore I'm having the same issue. Can you please let me know how exactly you modified the kibana.yaml inside the container? Can't seem to find the file in my elastalert docker container. Unless I'm missing something... most likely because my docker-foo is very lame :-( Much appreciated.

[kk-stateless]

Kibana.yml inside the container is present under /usr/share/kibana/config

I'd recommend mapping Kibana.yml from the host to /usr/share/kibana/config/Kibana.yml inside the container so you can make changes from the host and don't have to exec into the container

[bcisse]

@kk-stateless thx a bunch for the tip. Did as suggested and it is working now!