Rally cannot create rally-annotations index in Elasticsearch metric store

Rally version (get with esrally --version): esrally 2.8.1.dev0 (git revision: 01f3c25361d1128def77e3ea5d29c899580993b1)

Invoked command: esrally add annotation --track=elastic/security --message="test" --race-timestamp=20230712T000000Z

Configuration file (~/.rally/rally.ini):

[..]
[reporting]
datastore.type = elasticsearch
datastore.host = <HOST>
datastore.port = <PORT>
datastore.secure = true
datastore.user = <USER>
datastore.password = <PASSWORD>
datastore.number_of_shards = <X>
datastore.number_of_replicas = 1
[..]

JVM version: n/a

OS version: Mac / Python 3.11.0

Description of the problem including expected versus actual behavior:

When rally-annotations index is missing the command fails as follows:

(.venv) grzegorz:elastic/rally% esrally add annotation --track=elastic/security --message="test" --race-timestamp=20230712T000000Z

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[ERROR] Cannot add. 'index'.

The expected result is rally-annotations index gets created when missing.

Steps to reproduce:

Remove rally-annotations from Elasticsearch metric store.
Run the above command.

Provide logs (if relevant):

2023-07-12 06:25:44,25 -not-actor-/PID:51384 esrally.rally ERROR A fatal error occurred while running subcommand [add].
Traceback (most recent call last):
  File "/Users/grzegorz/Documents/src/elastic/rally/esrally/rally.py", line 1100, in dispatch_sub_command
    dispatch_add(cfg)
  File "/Users/grzegorz/Documents/src/elastic/rally/esrally/rally.py", line 865, in dispatch_add
    metrics.add_annotation(cfg)
  File "/Users/grzegorz/Documents/src/elastic/rally/esrally/metrics.py", line 1305, in add_annotation
    race_store(cfg).add_annotation()
  File "/Users/grzegorz/Documents/src/elastic/rally/esrally/metrics.py", line 1655, in add_annotation
    return self.es_store.add_annotation()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grzegorz/Documents/src/elastic/rally/esrally/metrics.py", line 1776, in add_annotation
    body = self.index_template_provider.annotations_template()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grzegorz/Documents/src/elastic/rally/esrally/metrics.py", line 301, in annotations_template
    return self._read("annotation-template")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/grzegorz/Documents/src/elastic/rally/esrally/metrics.py", line 312, in _read
    template["settings"]["index"]["number_of_shards"] = int(self._number_of_shards)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'index'

elastic / rally

Rally cannot create rally-annotations index in Elasticsearch metric store #1746