elasticsearch "URL must include a 'scheme', 'host', and 'port' component (ie 'https://localhost:9200')" not accepting paths in the URL

[francoisv100]

pip list | grep elastic
elastic-transport                                 8.4.1
elasticsearch                                     8.10.0
elasticsearch-dsl                                 8.9.0
elasticsearch-kibana-cli                          0.6.1
elasticsearch-query                               2.4.0

When adding a path in the url a Value Error is raised and connection fails to the remote elk host , url without a path initiates a connection , but in our setup we require the path to be included when initiating the connection,

Python Code used with elasticsearch version 8.10

from elasticsearch import Elasticsearch
es = Elasticsearch('https://api.elkserver.co.za/elastic/v1:443', headers={"v_token": v_token}, timeout=120)

 % ./elk_test.py
Traceback (most recent call last):
  File "./elk_test.py", line 108, in <module>
    kibana_get_info(elasticip, vtoken, timenow, prevtime)
  File "./elk_test.py", line 31, in kibana_get_info
    es = Elasticsearch('https://api.elkserver.co.za/elastic/v1:443', headers={"v_token": v_token}, timeout=120)
  File "/Users/username/PycharmProjects/GNS3-Python/venv1/lib/python3.7/site-packages/elasticsearch/_sync/client/__init__.py", line 347, in __init__
    ssl_show_warn=ssl_show_warn,
  File "/Users/username/PycharmProjects/GNS3-Python/venv1/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 105, in client_node_configs
    node_configs = hosts_to_node_configs(hosts)
  File "/Users/username/PycharmProjects/GNS3-Python/venv1/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 143, in hosts_to_node_configs
    return hosts_to_node_configs([hosts])
  File "/Users/username/PycharmProjects/GNS3-Python/venv1/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 151, in hosts_to_node_configs
    node_configs.append(url_to_node_config(host))
  File "/Users/username/PycharmProjects/GNS3-Python/venv1/lib/python3.7/site-packages/elastic_transport/client_utils.py", line 217, in url_to_node_config
    "URL must include a 'scheme', 'host', and 'port' component (ie 'https://localhost:9200')"
ValueError: URL must include a 'scheme', 'host', and 'port' component (ie 'https://localhost:9200')

from elasticsearch import Elasticsearch
es = Elasticsearch('https://api.elkserver.co.za/elastic/v1', headers={"v_token": v_token}, timeout=120)

 % ./elk_test.py
Traceback (most recent call last):
  File "./elk_test.py", line 108, in <module>
    kibana_get_info(elasticip, vtoken, timenow, prevtime)
  File "./elk_test.py", line 31, in kibana_get_info
    es = Elasticsearch('https://api.elkserver.co.za/elastic/v1', headers={"v_token": v_token}, timeout=120)
  File "/Users/username/PycharmProjects/GNS3-Python/venv1/lib/python3.7/site-packages/elasticsearch/_sync/client/__init__.py", line 347, in __init__
    ssl_show_warn=ssl_show_warn,
  File "/Users/username/PycharmProjects/GNS3-Python/venv1/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 105, in client_node_configs
    node_configs = hosts_to_node_configs(hosts)
  File "/Users/username/PycharmProjects/GNS3-Python/venv1/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 143, in hosts_to_node_configs
    return hosts_to_node_configs([hosts])
  File "/Users/username/PycharmProjects/GNS3-Python/venv1/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 151, in hosts_to_node_configs
    node_configs.append(url_to_node_config(host))
  File "/Users/username/PycharmProjects/GNS3-Python/venv1/lib/python3.7/site-packages/elastic_transport/client_utils.py", line 217, in url_to_node_config
    "URL must include a 'scheme', 'host', and 'port' component (ie 'https://localhost:9200')"
ValueError: URL must include a 'scheme', 'host', and 'port' component (ie 'https://localhost:9200')

from elasticsearch import Elasticsearch
es = Elasticsearch('https://api.elkserver.co.za:443', headers={"v_token": v_token}, timeout=120)

Without a path in the url , connection gets initiated , can ignore the index error

 ./elk_test.py
Traceback (most recent call last):
  File "./elk_test.py", line 108, in <module>
    kibana_get_info(elasticip, vtoken, timenow, prevtime)
  File "./elk_test.py", line 64, in kibana_get_info
    result = es.search(index="voice.*", body=query_body, size=100)
  File "/Users/username/PycharmProjects/GNS3-Python/venv1/lib/python3.7/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)
  File "/Users/username/PycharmProjects/GNS3-Python/venv1/lib/python3.7/site-packages/elasticsearch/_sync/client/__init__.py", line 3925, in search
    "POST", __path, params=__query, headers=__headers, body=__body
  File "/Users/username/PycharmProjects/GNS3-Python/venv1/lib/python3.7/site-packages/elasticsearch/_sync/client/_base.py", line 321, in perform_request
    message=message, meta=meta, body=resp_body
elasticsearch.NotFoundError: NotFoundError(404, "{'message': 'no Route matched with those values'}") 

[pquentin]

Hello, thanks for the feature request! This is something I would like to add in the future, but for now the client only works without a path.

[francoisv100]

Thanks I have downgraded to version 7.17 for now to get connectivity working again.

[pquentin]

I didn't realize this used to work, thanks for the information.

[cachedout]

This issue may also affect our published getting started instructions. In those instructions, it says to copy the URL from the cloud console and that URL is returned without a path and without a port. (ex: https://https://my-cool-cluster.es.europe-west9.gcp.elastic-cloud.com)

As a result, following the instructions we provide for getting started with the Python client results in the following stacktrace:

ValueError: URL must include a 'scheme', 'host', and 'port' component (ie 'https://localhost:9200')

[pquentin]

@francoisv100 While trying to work on this issue, I realized we already implement this, and found your mistake:

es = Elasticsearch('https://api.elkserver.co.za/elastic/v1:443', ...)

Parsing this url gives:

• scheme: https
• host: api.elkserver.co.za
• path: /elastic/v1:443 (!)

The URL should be https://api.elkserver.co.za:443/elastic/v1 instead. Now this goes back to @cachedout's comment where it is annoying to have to specify the port for the HTTPS endpoint. It was on my TODO list and I just opened https://github.com/elastic/elastic-transport-python/pull/127 to make 443 the default HTTPS port. This will be relased as elastic-transport 8.11.0.

[francoisv100]

Thanks for the feedback , updated my side and its working now,

es = Elasticsearch('https://api.elkserver.co.za:443/elastic/v1', headers={"v_token": v_token}, timeout=120, verify_certs=False)

[pquentin]

@francoisv100 Good to know!

For the record, elastic-transport 8.11.0 is out with the fix to default to port 443 for HTTPS endpoints. So if you upgrade, you won't need to specify the port anymore.