Open KACAH opened 2 years ago
This appears to still be an issue on elasticsearch-py version 8.5.0 and elasticsearch version 8.5.3. The misleading error message cost us quite some time re-evaluating our elasticsearch configuration. Would be nice if someone could take a look at it.
I have experienced the same issue. And indeed it
cost us quite some time re-evaluating our elasticsearch configuration
due to the observed behavior being:
Client was configured using RFC-1738 formatted URL, which is one of the officially proposed and supported options.
connections.configure(
default={
"hosts": [settings.ES_URL], # URL formatted as "http://user:password@host:port"
"retry_on_timeout": True,
"sniffer_timeout": 3600,
}
)
Nevertheless, after sniffing was done, connection pool lost all auth info and attempted to make unauthenticated requests.
The solution was either a. disable sniffing b. pass auth params explicitly:
from yarl import URL
es_url = URL(settings.ES_URL) # here ES_URL is parsed into components
connections.configure(
default={
"hosts": [str(es_url.origin())], # "host:port"
"http_auth": f"{es_url.user}:{es_url.password}",
"retry_on_timeout": True,
"sniffer_timeout": 3600,
}
)
I find this behavior misleading and not intuitive. It is also not described anywhere seemingly.
Just ran into this today. Our Elasticsearch API is exposed via an Nginx reverse proxy protected by basic auth. The reverse proxy itself uses HTTPS with self signed certificate
Trying to instantiate Elasticsearch with
Elasticsearch("https://internal.foo.com:443/elknew", sniff_on_connection_fail=True, sniff_on_start=True, max_retries=10, retry_on_timeout=True, basic_auth=["username", "password"], verify_certs=False)
fails with
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "lib/python3.6/site-packages/elasticsearch/_sync/client/__init__.py", line 398, in __init__
**transport_kwargs,
File "lib/python3.6/site-packages/elastic_transport/_transport.py", line 246, in __init__
self.sniff(True)
File "lib/python3.6/site-packages/elastic_transport/_transport.py", line 450, in sniff
"No viable nodes were discovered on the initial sniff attempt"
elastic_transport.SniffingError: No viable nodes were discovered on the initial sniff attempt
And nginx logs show that 401 was raised.
However, if we instantiate Elasticsearch with sniff_on_start=False
, it works correctly.
Is there any way to have sniff_on_start=True
and send authorization to sniff request? Any update or fix?
Howdy, I ran into this problem upgrading our clients from 7.x to 8.8 and wrote a workaround by overwriting the internal callback that does these sniff requests. Hope this issue can be addressed soon, but in the meantime you can still use the sniff options by initializing the connection like so (note this is for the sync client, and may need changes depending on your client version and needs):
https://gist.github.com/jhopkins219/2b5b38061f1b87515c7a5c29ca91a0a3
@jhopkins219 Thanks for your fix. I have tried it and it works with some adaptation (I need api key in my case).
But next step is a new problem, direct queries on nodes seems to lost my ca_certs
configuration and throw this error:
elastic_transport.TlsError: TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'https://NODE_HOSTNAME:9200'. (_ssl.c:1007))
I'm currently investigating in URLlilb, HttpRequest from elastic_transport
but without success.
Bumping cause this 2022 issue is still persistent.....classic enterprise devs lol
Howdy, I ran into this problem upgrading our clients from 7.x to 8.8 and wrote a workaround by overwriting the internal callback that does these sniff requests. Hope this issue can be addressed soon, but in the meantime you can still use the sniff options by initializing the connection like so (note this is for the sync client, and may need changes depending on your client version and needs):
https://gist.github.com/jhopkins219/2b5b38061f1b87515c7a5c29ca91a0a3
I can confirm @jhopkins219 patch fixes this issue for me! Props
is this fix released? facing the same on 8.12 elasticsearch-py
is this fix released? facing the same on 8.12 elasticsearch-py
nope. lazy enterprise devs still havent paid attention to this 2 year old issue.....
https://github.com/elastic/elasticsearch-py/blob/8.12/elasticsearch/_sync/client/_base.py#L166 ^ missing the auth header entirely still. Use the patch @jhopkins219 shared, it works perfectly.
I have also sent in a PR for a potential fix to this issue...but with this issue being from 2022, who knows when the developers will push this to the upstream.
Elasticsearch version (
bin/elasticsearch --version
): Version: 8.2.2, Build: default/docker/9876968ef3c745186b94fdabd4483e01499224ef/2022-05-25T15:47:06.259735307Z, JVM: 18.0.1.1elasticsearch-py
version (elasticsearch.__versionstr__
): 8.2.3Sniffing callback uses Transport instance directly to perform
/_nodes/_all/http
requests. https://github.com/elastic/elasticsearch-py/blob/0da2ba901514447089f8a290f0ddba4f18cf53f8/elasticsearch/_async/client/_base.py#L172However, both Authorization and opaque-id headers are NOT passed to Transport class during initialization. https://github.com/elastic/elasticsearch-py/blob/0da2ba901514447089f8a290f0ddba4f18cf53f8/elasticsearch/_async/client/__init__.py#L324
Instead they are saved to Elasticsearch client instance after Transport is already created https://github.com/elastic/elasticsearch-py/blob/0da2ba901514447089f8a290f0ddba4f18cf53f8/elasticsearch/_async/client/__init__.py#L407
and used for API requests only. https://github.com/elastic/elasticsearch-py/blob/0da2ba901514447089f8a290f0ddba4f18cf53f8/elasticsearch/_async/client/__init__.py#L407
As the result sniffing fails with Exception
elastic_transport.SniffingError: No viable nodes were discovered on the initial sniff attempt
while the real problem isThe problem is the same for both Sync and Async clients.
The fix might be to build opaque-id and authorization headers before transport intialization to and pass them to
client_node_configs
method (headers
argument). However, I am not sure if it will work "as intended" with.options()
feature and internal_transport
constructor argument ofElasticsearch
andAsyncElasticsearch
classes.