esrally.exceptions.SystemSetupError: Received a protocol error. Are you sure you're using the correct scheme (HTTP or HTTPS)?

[ziyunxiao]

Rally version (get with esrally --version): esrally 2.10.0

Invoked command: esrally race \ --track=geonames \ --target-hosts=es1.mydomain.com:443 \ --client-options="use_ssl:true,verify_certs:false,basic_auth_user:'benchmark',basic_auth_password:'${ELASTICSEARCH_PASSWORD}'" \ --pipeline=benchmark-only \ --test-mode

Configuration file (located in ~/.rally/rally.ini)):

[meta]
config.version = 17

[system]
env.name = local

[node]
root.dir = /home2/rsong/.rally/benchmarks
src.root.dir = /home2/rsong/.rally/benchmarks/src

[source]
remote.repo.url = https://github.com/elastic/elasticsearch.git
elasticsearch.src.subdir = elasticsearch

[benchmarks]
local.dataset.cache = /home2/rsong/.rally/benchmarks/data

[reporting]
datastore.type = in-memory
datastore.host =
datastore.port =
datastore.secure = False
datastore.user =
datastore.password =

[tracks]
default.url = https://github.com/elastic/rally-tracks

[teams]
default.url = https://github.com/elastic/rally-teams

[defaults]
preserve_benchmark_candidate = false

[distributions]
release.cache = true

JVM version: openjdk 11.0.21 2023-10-17 OpenJDK Runtime Environment (build 11.0.21+9-post-Ubuntu-0ubuntu120.04) OpenJDK 64-Bit Server VM (build 11.0.21+9-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)

OS version: No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.4 LTS Release: 20.04 Codename: focal

Description of the problem including expected versus actual behavior: When try to run esrally race against existing Elasticsearch cluster I got error "esrally.exceptions.SystemSetupError: Received a protocol error. Are you sure you're using the correct scheme (HTTP or HTTPS)?"

Steps to reproduce:

1. curl -u "benchmark:$ELASTICSEARCH_PASSWORD" -k https://es1.mydomain.com/ I use this command to verify that cluster is running and the username and password works
2. Issue command

   
   esrally race \
   --track=geonames \
   --target-hosts=es1.mydomain.com:443 \
   --client-options="use_ssl:true,verify_certs:false,basic_auth_user:'benchmark',basic_auth_password:'${ELASTICSEARCH_PASSWORD}'" \
   --pipeline=benchmark-only \
   --test-mode

**Provide logs (if relevant)**:

____        ____

/ __ / / / / /_/ / `/ / / / / / / , / // / / / // / // ||_,///__, / /____/

[INFO] Race id is [f3e0b144-6d8b-453a-b2b3-e3139485e26f] [ERROR] Cannot race. Traceback (most recent call last): File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/esrally/client/factory.py", line 289, in wait_for_rest_layer es.cluster.health(wait_for_nodes=f">={expected_node_count}") File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped return api(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/elasticsearch/_sync/client/cluster.py", line 464, in health return self.perform_request( # type: ignore[return-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/elasticsearch/_sync/client/_base.py", line 390, in perform_request return self._client.perform_request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/esrally/client/synchronous.py", line 195, in perform_request meta, resp_body = self.transport.perform_request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/elastic_transport/_transport.py", line 328, in perform_request meta, raw_data = node.perform_request( ^^^^^^^^^^^^^^^^^^^^^ File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request raise err from None elastic_transport.ConnectionError: Connection error caused by: ConnectionError(Connection error caused by: ProtocolError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/esrally/actor.py", line 92, in guard return f(self, msg, sender) ^^^^^^^^^^^^^^^^^^^^ File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup self.coordinator.setup(sources=msg.sources) File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/esrally/racecontrol.py", line 194, in setup ) = client.factory.cluster_distribution_version(hosts, client_options) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/esrally/client/factory.py", line 364, in cluster_distribution_version wait_for_rest_layer(es) File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/esrally/client/factory.py", line 307, in wait_for_rest_layer raise exceptions.SystemSetupError( esrally.exceptions.SystemSetupError: Received a protocol error. Are you sure you're using the correct scheme (HTTP or HTTPS)?

Getting further help:

---

• Check the log files in /home2/rsong/.rally/logs for errors.
• Read the documentation at https://esrally.readthedocs.io/en/2.10.0/.
• Ask a question on the forum at https://discuss.elastic.co/tags/c/elastic-stack/elasticsearch/rally.
• Raise an issue at https://github.com/elastic/rally/issues and include the log files in /home2/rsong/.rally/logs.

---

[INFO] FAILURE (took 4 seconds)

<!--
If you are filing a feature request, please remove the above bug
report block and provide responses for all of the below items.
-->

**Describe the feature**:
Expecting to be able to run race when ElasticSearch is configured behind a load balancer with TLS enabled.