search

ElasticHQ / elasticsearch-HQ

Monitoring and Management Web Application for ElasticSearch instances and clusters.
http://www.elastichq.org
Other
4.96k stars 530 forks source link

Docker option to pass CA file (for self-signed CA) #479

Closed jlim0930 closed 4 years ago

jlim0930 commented 5 years ago

General information

Issue Description

running ElasticHQ in docker container and passed both HQ_ENABLE_SSL & HQ_CA_CERTS as docker run -p 5000:5000 -e HQ_ENABLE_SSL=true -e HQ_CA_CERTS=/certs/ca.crt -v /config/certs/ca/ca.crt:/certs/ca.crt elastichq/elasticsearch-hq

Source Code / Logs

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='elastic.local', port=9200): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/src/elastichq/service/ConnectionService.py", line 71, in create_connection response = requests.get(scheme + "://" + ip + ":" + port, timeout=REQUEST_TIMEOUT) File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 75, in get return request('get', url, params=params, kwargs) File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, kwargs) File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, send_kwargs) File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, kwargs) File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 514, in send raise SSLError(e, request=request)

valentin-fischer commented 5 years ago

Having the same issue, haven't found a solution yet...

royrusso commented 4 years ago

Yes. This seems related to #471

Hope to get this patched this week.

qakart commented 4 years ago

@royrusso thanks for closing this issue and i am running Docker image I did build my own image using the latest code to get this above fix but still can't able to see the issue fixed.

(Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))",),),) --- Logging error --- Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn conn.connect() File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect ssl_context=context) File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket return context.wrap_socket(sock, server_hostname=server_hostname) File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket return GreenSSLSocket(sock, *a, _context=self, **kw) File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__ self.do_handshake() File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake super(GreenSSLSocket, self).do_handshake) File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining return func(*a, **kw) File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake self._sslobj.do_handshake() File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

Me to facing the same issue and looking for the fix or workaround

gittygoo commented 4 years ago

This also does not work for me either with the latest version. I am passing the following env vars:

gplechuck commented 4 years ago

Also getting a [SSL: CERTIFICATE_VERIFY_FAILED] error. Specified a HQ_CA_CERTS var when running the container and this can be seen in the output from 'env' within the container, but it doesn't appear that the custom CA cert location passed by the var is being read when a connection is attempted.

Ran opensnoop on the container host and observed which files were being opened in the container when attempting to access a https enabled cluster - 

~# opensnoop-bpfcc             
PID    COMM               FD ERR PATH            
...
10589  gunicorn           13   0 /etc/hosts
10589  gunicorn           14   0 /usr/local/lib/python3.6/site-packages/certifi/cacert.pem

The default cacert.pem bundle is opened, but the custom ca cert is not accessed.

Haven't looked into it further, but as a dirty workaround have replaced the container CMD with a chain of commands that will cat the custom ca cert to the default ca bundle before ElasticHQ is started. Docker compose file looks something like this - 

version: '3.4'
services:
    elastichq:
        image: elastichq/elasticsearch-hq
        restart: always
        deploy:
            replicas: 1
            restart_policy:
                condition: on-failure
        container_name: elastichq
        environment:
          - HQ_ENABLE_SSL=True
          - HQ_VERIFY_CERTS=False
#          - HQ_CA_CERTS=/usr/local/share/ca-certificates/MYCA.crt
        volumes:
          - "/usr/local/share/ca-certificates:/usr/local/share/ca-certificates:ro"
        ports:
            - "5000:5000/tcp"
        command: bash -c 'cat /usr/local/share/ca-certificates/MYCA.crt >> /usr/local/lib/python3.6/site-packages/certifi/cacert.pem && supervisord -c /etc/supervisor/supervisord.conf'
gittygoo commented 4 years ago

@royrusso can this issue please be reopened? not being able to test elastichq atm due to this

royrusso commented 4 years ago

I see the bug now: https://github.com/ElasticHQ/elasticsearch-HQ/blob/e61695a6e6fad911613a1980298644f56f7add3a/application.py#L27

The internal name used in code is simply VERIFY_CERTS.

gittygoo commented 4 years ago

@royrusso i tried using VERIFY_CERTS also that didnt work, should it?

royrusso commented 4 years ago

Working on a new release. I'll update this when it's out.

royrusso commented 4 years ago

v3.5.6 is released and merged to master. Cross your fingers, because I can't exactly test this feature. ;-)

gittygoo commented 4 years ago

I can test it tomorrow and feedback you here if there is a docker container update for it also (on the phone at the moment so cant check) but will feedback tomorrow if so :)

gittygoo commented 4 years ago

@royrusso I'm still getting the same: ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

options im using HQ_DEBUG = True HQ_DEFAULT_URL = https://XXXX:YYYYY@my.host:31425 HQ_ENABLE_SSL = True HQ_VERIFY_CERTS = False VERIFY_CERTS = False

royrusso commented 4 years ago

Are you setting the HQ_CA_CERTS value? If you have SSL enabled, you need to pass a cert.

gittygoo commented 4 years ago

Yes, here is some more debugging info that might help


LOG.error(message, ex)
Message: 'Unable to create connection to: https://XXXX:YYYYY@my.host:31425'
Arguments: (SSLError(MaxRetryError("HTTPSConnectionPool(host='my.host', port=31425): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))",),),)
--- Logging error ---
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
return GreenSSLSocket(sock, *a, _context=self, **kw)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
self.do_handshake()
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
super(GreenSSLSocket, self).do_handshake)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
return func(*a, **kw)
File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)```
royrusso commented 4 years ago

Make sure you are running 3.5.6. The stack trace I think should include info messages like "SSL is True". Can you post that or the whole log file?

gittygoo commented 4 years ago

The version on the gui says it is correct: "You are on the current version: 3.5.6" By the way, i am of course redacting the host , ports and credentials on the logs so i'm not trying with "my.host" :)

here you go the full log


2019-11-14 14:25:40,392 INFO supervisord started with pid 1
2019-11-14 14:25:41,397 INFO spawned: 'gunicorn' with pid 8
2019-11-14 14:25:42,401 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
loading config /src/elastichq/config/logger.json
2019-11-14 14:25:44,138 INFO engineio server.__init__:140 Server initialized for eventlet.
loading config /src/elastichq/config/logger_debug.json
2019-11-14 14:26:15,129 DEBUG sqlalchemy.orm.path_registry path_registry.set:63 set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x7fbbed7d7cc0; ClusterModel>,))' to '{}'
2019-11-14 14:26:15,131 DEBUG sqlalchemy.pool.NullPool pool.__connect:668 Created new connection <sqlite3.Connection object at 0x7fbbed15e490>
2019-11-14 14:26:15,132 DEBUG sqlalchemy.pool.NullPool pool.checkout:550 Connection <sqlite3.Connection object at 0x7fbbed15e490> checked out from pool
2019-11-14 14:26:15,132 INFO sqlalchemy.engine.base.Engine base._begin_impl:682 BEGIN (implicit)
2019-11-14 14:26:15,134 INFO sqlalchemy.engine.base.Engine base._execute_context:1151 SELECT cluster.cluster_name AS cluster_cluster_name, cluster.cluster_ip AS cluster_cluster_ip, cluster.cluster_port AS cluster_cluster_port, cluster.cluster_scheme AS cluster_cluster_scheme, cluster.cluster_version AS cluster_cluster_version, cluster.cluster_username AS cluster_cluster_username, cluster.cluster_password AS cluster_cluster_password
FROM cluster
2019-11-14 14:26:15,134 INFO sqlalchemy.engine.base.Engine base._execute_context:1154 ()
2019-11-14 14:26:15,136 DEBUG sqlalchemy.engine.base.Engine result._init_metadata:681 Col ('cluster_cluster_name', 'cluster_cluster_ip', 'cluster_cluster_port', 'cluster_cluster_scheme', 'cluster_cluster_version', 'cluster_cluster_username', 'cluster_cluster_password')
2019-11-14 14:26:15,142 DEBUG sqlalchemy.pool.NullPool pool._finalize_fairy:697 Connection <sqlite3.Connection object at 0x7fbbed15e490> being returned to pool
2019-11-14 14:26:15,142 DEBUG sqlalchemy.pool.NullPool pool._reset:869 Connection <sqlite3.Connection object at 0x7fbbed15e490> rollback-on-return, via agent
2019-11-14 14:26:15,143 INFO sqlalchemy.engine.base.Engine base._rollback_impl:702 ROLLBACK
2019-11-14 14:26:15,143 DEBUG sqlalchemy.pool.NullPool pool._close_connection:314 Closing connection <sqlite3.Connection object at 0x7fbbed15e490>
2019-11-14 14:26:15,165 DEBUG sqlalchemy.orm.path_registry path_registry.set:63 set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x7fbbed7d7cc0; ClusterModel>,))' to '{}'
2019-11-14 14:26:15,167 DEBUG sqlalchemy.pool.NullPool pool.__connect:668 Created new connection <sqlite3.Connection object at 0x7fbbed0ad2d0>
2019-11-14 14:26:15,167 DEBUG sqlalchemy.pool.NullPool pool.checkout:550 Connection <sqlite3.Connection object at 0x7fbbed0ad2d0> checked out from pool
2019-11-14 14:26:15,168 INFO sqlalchemy.engine.base.Engine base._begin_impl:682 BEGIN (implicit)
2019-11-14 14:26:15,169 INFO sqlalchemy.engine.base.Engine base._execute_context:1151 SELECT cluster.cluster_name AS cluster_cluster_name, cluster.cluster_ip AS cluster_cluster_ip, cluster.cluster_port AS cluster_cluster_port, cluster.cluster_scheme AS cluster_cluster_scheme, cluster.cluster_version AS cluster_cluster_version, cluster.cluster_username AS cluster_cluster_username, cluster.cluster_password AS cluster_cluster_password
FROM cluster
2019-11-14 14:26:15,170 INFO sqlalchemy.engine.base.Engine base._execute_context:1154 ()
2019-11-14 14:26:15,171 DEBUG sqlalchemy.engine.base.Engine result._init_metadata:681 Col ('cluster_cluster_name', 'cluster_cluster_ip', 'cluster_cluster_port', 'cluster_cluster_scheme', 'cluster_cluster_version', 'cluster_cluster_username', 'cluster_cluster_password')
2019-11-14 14:26:15,174 DEBUG sqlalchemy.pool.NullPool pool._finalize_fairy:697 Connection <sqlite3.Connection object at 0x7fbbed0ad2d0> being returned to pool
2019-11-14 14:26:15,175 DEBUG sqlalchemy.pool.NullPool pool._reset:869 Connection <sqlite3.Connection object at 0x7fbbed0ad2d0> rollback-on-return, via agent
2019-11-14 14:26:15,175 INFO sqlalchemy.engine.base.Engine base._rollback_impl:702 ROLLBACK
2019-11-14 14:26:15,176 DEBUG sqlalchemy.pool.NullPool pool._close_connection:314 Closing connection <sqlite3.Connection object at 0x7fbbed0ad2d0>
2019-11-14 14:26:15,218 DEBUG sqlalchemy.orm.path_registry path_registry.set:63 set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x7fbbed7d7cc0; ClusterModel>,))' to '{}'
2019-11-14 14:26:15,219 DEBUG sqlalchemy.pool.NullPool pool.__connect:668 Created new connection <sqlite3.Connection object at 0x7fbbed0ad9d0>
2019-11-14 14:26:15,220 DEBUG sqlalchemy.pool.NullPool pool.checkout:550 Connection <sqlite3.Connection object at 0x7fbbed0ad9d0> checked out from pool
2019-11-14 14:26:15,221 INFO sqlalchemy.engine.base.Engine base._begin_impl:682 BEGIN (implicit)
2019-11-14 14:26:15,222 INFO sqlalchemy.engine.base.Engine base._execute_context:1151 SELECT cluster.cluster_name AS cluster_cluster_name, cluster.cluster_ip AS cluster_cluster_ip, cluster.cluster_port AS cluster_cluster_port, cluster.cluster_scheme AS cluster_cluster_scheme, cluster.cluster_version AS cluster_cluster_version, cluster.cluster_username AS cluster_cluster_username, cluster.cluster_password AS cluster_cluster_password
FROM cluster
2019-11-14 14:26:15,222 INFO sqlalchemy.engine.base.Engine base._execute_context:1154 ()
2019-11-14 14:26:15,224 DEBUG sqlalchemy.engine.base.Engine result._init_metadata:681 Col ('cluster_cluster_name', 'cluster_cluster_ip', 'cluster_cluster_port', 'cluster_cluster_scheme', 'cluster_cluster_version', 'cluster_cluster_username', 'cluster_cluster_password')
2019-11-14 14:26:15,225 DEBUG elastichq status.get:63 {"name": "ElasticHQ", "installed_version": "3.5.6", "current_stable_version": "3.5.6", "tagline": "You know, for Elasticsearch", "clusters": [], "default_url": "https://XXXX:YYYYY@my.host:31425"}
2019-11-14 14:26:15,227 DEBUG sqlalchemy.pool.NullPool pool._finalize_fairy:697 Connection <sqlite3.Connection object at 0x7fbbed0ad9d0> being returned to pool
2019-11-14 14:26:15,227 DEBUG sqlalchemy.pool.NullPool pool._reset:869 Connection <sqlite3.Connection object at 0x7fbbed0ad9d0> rollback-on-return, via agent
2019-11-14 14:26:15,228 INFO sqlalchemy.engine.base.Engine base._rollback_impl:702 ROLLBACK
2019-11-14 14:26:15,228 DEBUG sqlalchemy.pool.NullPool pool._close_connection:314 Closing connection <sqlite3.Connection object at 0x7fbbed0ad9d0>
None
None
2019-11-14 14:26:40,524 INFO elastichq ConnectionService.create_connection:69 Basic Auth is True
2019-11-14 14:26:40,526 INFO elastichq ConnectionService.create_connection:71 SSL enabled
config settings.json not found, searched /etc/elastic-hq/settings.json,~/settings.json,/src/settings.json,/src/elastichq/settings.json,/src/elastichq/config/settings.json
--- Logging error ---
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
return GreenSSLSocket(sock, *a, _context=self, **kw)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
self.do_handshake()
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
super(GreenSSLSocket, self).do_handshake)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
return func(*a, **kw)
File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='my.host', port=31425): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/src/elastichq/service/ConnectionService.py", line 73, in create_connection
timeout=REQUEST_TIMEOUT, verify=ca_certs, cert=client_cert_credentials)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='my.host', port=31425): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/logging/__init__.py", line 994, in emit
msg = self.format(record)
File "/usr/local/lib/python3.6/logging/__init__.py", line 840, in format
return fmt.format(record)
File "/usr/local/lib/python3.6/logging/__init__.py", line 577, in format
record.message = record.getMessage()
File "/usr/local/lib/python3.6/logging/__init__.py", line 338, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File "/usr/local/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
result = function(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 116, in handle
super(EventletWorker, self).handle(listener, client, addr)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/async.py", line 56, in handle
self.handle_request(listener_name, req, client, addr)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/async.py", line 107, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1997, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.6/site-packages/flask_socketio/__init__.py", line 43, in __call__
start_response)
File "/usr/local/lib/python3.6/site-packages/engineio/middleware.py", line 67, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.6/site-packages/flask_restful/__init__.py", line 480, in wrapper
resp = resource(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_restful/__init__.py", line 595, in dispatch_request
resp = meth(*args, **kwargs)
File "/src/elastichq/common/exceptions.py", line 29, in _request_wrapper
return functor(*args, **kwargs)
File "/src/elastichq/api/clusters.py", line 129, in post
client_cert=client_cert)
File "/src/elastichq/service/ConnectionService.py", line 133, in create_connection
LOG.error(message, ex)
Message: 'Unable to create connection to: https://my.host:31425'
Arguments: (SSLError(MaxRetryError("HTTPSConnectionPool(host='my.host', port=31425): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))",),),)
--- Logging error ---
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
return GreenSSLSocket(sock, *a, _context=self, **kw)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
self.do_handshake()
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
super(GreenSSLSocket, self).do_handshake)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
return func(*a, **kw)
File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)
During handling of the above exception, another exception occurred:```
royrusso commented 4 years ago

I added more logging and a new docker is being built. I may have found the problem, but again... difficult for me to test locally.

gittygoo commented 4 years ago

Im guessing this is the part you are looking for? 


2019-11-14 16:31:06,759 INFO elastichq ConnectionService.create_connection:58 Cert File: /certs/tls.crt
2019-11-14 16:31:06,759 INFO elastichq ConnectionService.create_connection:73 Basic Auth is True
2019-11-14 16:31:06,760 INFO elastichq ConnectionService.create_connection:75 SSL enabled
2019-11-14 16:31:06,760 INFO elastichq ConnectionService.create_connection:82 Verify Certs is True
config settings.json not found, searched /etc/elastic-hq/settings.json,~/settings.json,/src/settings.json,/src/elastichq/settings.json,/src/elastichq/config/settings.json```
royrusso commented 4 years ago

Yeah. I just added that. So that's the most recent code changes. Is it working? If not, stack trace?

gittygoo commented 4 years ago

I had to run with :development on the docker image to be able to pull this one On the bottom says "Installed Version: 3.5.7. Latest version: 3.5.6 Consider upgrading!"

But no luck yet :(


2019-11-14 16:36:32,339 INFO supervisord started with pid 1
2019-11-14 16:36:33,344 INFO spawned: 'gunicorn' with pid 8
2019-11-14 16:36:34,348 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
loading config /src/elastichq/config/logger.json
2019-11-14 16:36:36,082 INFO engineio server.__init__:140 Server initialized for eventlet.
loading config /src/elastichq/config/logger_debug.json
2019-11-14 16:36:39,972 DEBUG sqlalchemy.orm.path_registry path_registry.set:63 set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x7f5551fd2cc0; ClusterModel>,))' to '{}'
2019-11-14 16:36:39,974 DEBUG sqlalchemy.pool.NullPool pool.__connect:668 Created new connection <sqlite3.Connection object at 0x7f55519599d0>
2019-11-14 16:36:39,975 DEBUG sqlalchemy.pool.NullPool pool.checkout:550 Connection <sqlite3.Connection object at 0x7f55519599d0> checked out from pool
2019-11-14 16:36:39,976 INFO sqlalchemy.engine.base.Engine base._begin_impl:682 BEGIN (implicit)
2019-11-14 16:36:39,977 INFO sqlalchemy.engine.base.Engine base._execute_context:1151 SELECT cluster.cluster_name AS cluster_cluster_name, cluster.cluster_ip AS cluster_cluster_ip, cluster.cluster_port AS cluster_cluster_port, cluster.cluster_scheme AS cluster_cluster_scheme, cluster.cluster_version AS cluster_cluster_version, cluster.cluster_username AS cluster_cluster_username, cluster.cluster_password AS cluster_cluster_password
FROM cluster
2019-11-14 16:36:39,977 INFO sqlalchemy.engine.base.Engine base._execute_context:1154 ()
2019-11-14 16:36:39,979 DEBUG sqlalchemy.engine.base.Engine result._init_metadata:681 Col ('cluster_cluster_name', 'cluster_cluster_ip', 'cluster_cluster_port', 'cluster_cluster_scheme', 'cluster_cluster_version', 'cluster_cluster_username', 'cluster_cluster_password')
2019-11-14 16:36:39,984 DEBUG sqlalchemy.pool.NullPool pool._finalize_fairy:697 Connection <sqlite3.Connection object at 0x7f55519599d0> being returned to pool
2019-11-14 16:36:39,984 DEBUG sqlalchemy.pool.NullPool pool._reset:869 Connection <sqlite3.Connection object at 0x7f55519599d0> rollback-on-return, via agent
2019-11-14 16:36:39,985 INFO sqlalchemy.engine.base.Engine base._rollback_impl:702 ROLLBACK
2019-11-14 16:36:39,985 DEBUG sqlalchemy.pool.NullPool pool._close_connection:314 Closing connection <sqlite3.Connection object at 0x7f55519599d0>
2019-11-14 16:36:40,016 DEBUG sqlalchemy.orm.path_registry path_registry.set:63 set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x7f5551fd2cc0; ClusterModel>,))' to '{}'
2019-11-14 16:36:40,018 DEBUG sqlalchemy.pool.NullPool pool.__connect:668 Created new connection <sqlite3.Connection object at 0x7f55518a11f0>
2019-11-14 16:36:40,018 DEBUG sqlalchemy.pool.NullPool pool.checkout:550 Connection <sqlite3.Connection object at 0x7f55518a11f0> checked out from pool
2019-11-14 16:36:40,019 INFO sqlalchemy.engine.base.Engine base._begin_impl:682 BEGIN (implicit)
2019-11-14 16:36:40,020 INFO sqlalchemy.engine.base.Engine base._execute_context:1151 SELECT cluster.cluster_name AS cluster_cluster_name, cluster.cluster_ip AS cluster_cluster_ip, cluster.cluster_port AS cluster_cluster_port, cluster.cluster_scheme AS cluster_cluster_scheme, cluster.cluster_version AS cluster_cluster_version, cluster.cluster_username AS cluster_cluster_username, cluster.cluster_password AS cluster_cluster_password
FROM cluster
2019-11-14 16:36:40,020 INFO sqlalchemy.engine.base.Engine base._execute_context:1154 ()
2019-11-14 16:36:40,022 DEBUG sqlalchemy.engine.base.Engine result._init_metadata:681 Col ('cluster_cluster_name', 'cluster_cluster_ip', 'cluster_cluster_port', 'cluster_cluster_scheme', 'cluster_cluster_version', 'cluster_cluster_username', 'cluster_cluster_password')
2019-11-14 16:36:40,024 DEBUG sqlalchemy.pool.NullPool pool._finalize_fairy:697 Connection <sqlite3.Connection object at 0x7f55518a11f0> being returned to pool
2019-11-14 16:36:40,024 DEBUG sqlalchemy.pool.NullPool pool._reset:869 Connection <sqlite3.Connection object at 0x7f55518a11f0> rollback-on-return, via agent
2019-11-14 16:36:40,025 INFO sqlalchemy.engine.base.Engine base._rollback_impl:702 ROLLBACK
2019-11-14 16:36:40,025 DEBUG sqlalchemy.pool.NullPool pool._close_connection:314 Closing connection <sqlite3.Connection object at 0x7f55518a11f0>
2019-11-14 16:36:40,072 DEBUG sqlalchemy.orm.path_registry path_registry.set:63 set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x7f5551fd2cc0; ClusterModel>,))' to '{}'
2019-11-14 16:36:40,074 DEBUG sqlalchemy.pool.NullPool pool.__connect:668 Created new connection <sqlite3.Connection object at 0x7f55518b31f0>
2019-11-14 16:36:40,074 DEBUG sqlalchemy.pool.NullPool pool.checkout:550 Connection <sqlite3.Connection object at 0x7f55518b31f0> checked out from pool
2019-11-14 16:36:40,075 INFO sqlalchemy.engine.base.Engine base._begin_impl:682 BEGIN (implicit)
2019-11-14 16:36:40,076 INFO sqlalchemy.engine.base.Engine base._execute_context:1151 SELECT cluster.cluster_name AS cluster_cluster_name, cluster.cluster_ip AS cluster_cluster_ip, cluster.cluster_port AS cluster_cluster_port, cluster.cluster_scheme AS cluster_cluster_scheme, cluster.cluster_version AS cluster_cluster_version, cluster.cluster_username AS cluster_cluster_username, cluster.cluster_password AS cluster_cluster_password
FROM cluster
2019-11-14 16:36:40,076 INFO sqlalchemy.engine.base.Engine base._execute_context:1154 ()
2019-11-14 16:36:40,078 DEBUG sqlalchemy.engine.base.Engine result._init_metadata:681 Col ('cluster_cluster_name', 'cluster_cluster_ip', 'cluster_cluster_port', 'cluster_cluster_scheme', 'cluster_cluster_version', 'cluster_cluster_username', 'cluster_cluster_password')
2019-11-14 16:36:40,079 DEBUG elastichq status.get:63 {"name": "ElasticHQ", "installed_version": "3.5.7", "current_stable_version": "3.5.7", "tagline": "You know, for Elasticsearch", "clusters": [], "default_url": "https://XXX:YYY@my.host:31425"}
2019-11-14 16:36:40,081 DEBUG sqlalchemy.pool.NullPool pool._finalize_fairy:697 Connection <sqlite3.Connection object at 0x7f55518b31f0> being returned to pool
2019-11-14 16:36:40,081 DEBUG sqlalchemy.pool.NullPool pool._reset:869 Connection <sqlite3.Connection object at 0x7f55518b31f0> rollback-on-return, via agent
2019-11-14 16:36:40,082 INFO sqlalchemy.engine.base.Engine base._rollback_impl:702 ROLLBACK
2019-11-14 16:36:40,082 DEBUG sqlalchemy.pool.NullPool pool._close_connection:314 Closing connection <sqlite3.Connection object at 0x7f55518b31f0>
None
None
2019-11-14 16:36:43,111 INFO elastichq ConnectionService.create_connection:57 Verify: False
2019-11-14 16:36:43,112 INFO elastichq ConnectionService.create_connection:58 Cert File: /certs/tls.crt
2019-11-14 16:36:43,112 INFO elastichq ConnectionService.create_connection:73 Basic Auth is True
2019-11-14 16:36:43,113 INFO elastichq ConnectionService.create_connection:75 SSL enabled
2019-11-14 16:36:43,113 INFO elastichq ConnectionService.create_connection:82 Verify Certs is True
config settings.json not found, searched /etc/elastic-hq/settings.json,~/settings.json,/src/settings.json,/src/elastichq/settings.json,/src/elastichq/config/settings.json
--- Logging error ---
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
return GreenSSLSocket(sock, *a, _context=self, **kw)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
self.do_handshake()
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
super(GreenSSLSocket, self).do_handshake)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
return func(*a, **kw)
File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='my.host', port=31425): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/src/elastichq/service/ConnectionService.py", line 84, in create_connection
timeout=REQUEST_TIMEOUT, verify=ca_certs, cert=client_cert_credentials)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='my.host', port=31425): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/logging/__init__.py", line 994, in emit
msg = self.format(record)
File "/usr/local/lib/python3.6/logging/__init__.py", line 840, in format
return fmt.format(record)
File "/usr/local/lib/python3.6/logging/__init__.py", line 577, in format
record.message = record.getMessage()
File "/usr/local/lib/python3.6/logging/__init__.py", line 338, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
File "/usr/local/lib/python3.6/site-packages/eventlet/greenthread.py", line 221, in main
result = function(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 116, in handle
super(EventletWorker, self).handle(listener, client, addr)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/async.py", line 56, in handle
self.handle_request(listener_name, req, client, addr)
File "/usr/local/lib/python3.6/site-packages/gunicorn/workers/async.py", line 107, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1997, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.6/site-packages/flask_socketio/__init__.py", line 43, in __call__
start_response)
File "/usr/local/lib/python3.6/site-packages/engineio/middleware.py", line 67, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.6/site-packages/flask_restful/__init__.py", line 480, in wrapper
resp = resource(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask/views.py", line 84, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_restful/__init__.py", line 595, in dispatch_request
resp = meth(*args, **kwargs)
File "/src/elastichq/common/exceptions.py", line 29, in _request_wrapper
return functor(*args, **kwargs)
File "/src/elastichq/api/clusters.py", line 129, in post
client_cert=client_cert)
File "/src/elastichq/service/ConnectionService.py", line 152, in create_connection
LOG.error(message, ex)
Message: 'Unable to create connection to: https://my.host:31425'
Arguments: (SSLError(MaxRetryError("HTTPSConnectionPool(host='my.host', port=31425): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))",),),)
--- Logging error ---
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/usr/local/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 438, in wrap_socket
return GreenSSLSocket(sock, *a, _context=self, **kw)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 128, in __init__
self.do_handshake()
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 301, in do_handshake
super(GreenSSLSocket, self).do_handshake)
File "/usr/local/lib/python3.6/site-packages/eventlet/green/ssl.py", line 150, in _call_trampolining
return func(*a, **kw)
File "/usr/local/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)
During handling of the above exception, another exception occurred:```
royrusso commented 4 years ago

Can you post your ENV vars? Looks like you're running with HQ_VERIFY_CERTS=False

gittygoo commented 4 years ago

Yes i am, posted them here previously, here they are again

options im using
HQ_DEBUG = True
HQ_DEFAULT_URL = https://XXXX:YYYYY@my.host:31425
HQ_ENABLE_SSL = True
HQ_VERIFY_CERTS = False
VERIFY_CERTS = False
HQ_CA_CERTS = /certs/tls.crt
acarsercan commented 4 years ago

Hi,

I'm having the problem above with 3.5.12

Passing ENV:

1   │         env:
  22   │         - name: HQ_VERIFY_CERTS
  23   │           value: "False"

elasticsearch/es-hq-69b59cdb48-rprvs[es-hq]: Arguments: (SSLError(MaxRetryError("HTTPSConnectionPool(host='XXXX', port=9200): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))",),),)

mgiammarco commented 3 years ago

Please reopen I have the same problem with latest version

sangdrax8 commented 3 years ago

I am trying to test this now, and can't get the docker to read my cert or turn off verification.