Replica indices have no replicas shards

[hannes-ucsc]

To reproduce, request GET _cat/indices/azul_*_replica, pick one index, and request GET <index>, e.g., GET azul_v2_anvilprod_anvil6_replica. It returns

{
    "azul_v2_anvilprod_anvil6_replica": {
        "aliases": {},
        "mappings": {…},
        "settings": {
            "index": {
                "refresh_interval": "1s",
                "number_of_shards": "37",
                "provided_name": "azul_v2_anvilprod_anvil6_replica",
                "creation_date": "1715194280959",
                "number_of_replicas": "0",
                "uuid": "xJAQK2_9T8WYknkmIHE4TQ",
                "version": {
                    "created": "7100299"
                }
            }
        }
    }
}

Note "number_of_replicas": "0",

Given that replica indices are read frequently, just like aggregate indices, we should consider replicating them.

Note that the term replica is used in two distinct contexts: the Azul team uses the term to refer to a verbatim copy of a metadata entity. In the context of Elasticsearch the term is used to refer to a copy of an index or an index shard.

[achave11-ucsc]

Assignee to consider next steps.

[hannes-ucsc]

Replicating replica indices should increase the number of manifest the system can generate concurrently. It may also speed up the generation of a single manifest by reducing or, if the number of replicas is the cluster size minus one, eliminating the need to coordinate with other nodes.

We should set up an experiment in which we replicate the replica index so that each cluster node has a complete copy of the index, and then compare the time it takes to generate one large manifest, and multiple large manifests (as many as there are cluster nodes), before and after that change.

For now, this can go to the backlog.