[CI] SearchWithMinCompatibleSearchNodeIT testMinVersionAsOldVersion failing

[mark-vieira]

This and other tests in SearchWithMinCompatibleSearchNodeIT are failing consistently on Java 23. The underlying error looks to be a date parsing issue related to changes in Java 23.

Build scan: https://gradle-enterprise.elastic.co/s/zpuwiiomt72u4/tests/:qa:mixed-cluster:v7.17.22%23mixedClusterTest/org.elasticsearch.backwards.SearchWithMinCompatibleSearchNodeIT/testMinVersionAsOldVersion

Reproduction line:

./gradlew ':qa:mixed-cluster:v7.17.22#mixedClusterTest' -Dtests.class="org.elasticsearch.backwards.SearchWithMinCompatibleSearchNodeIT" -Dtests.method="testMinVersionAsOldVersion" -Dtests.seed=5DBDE67B8A1D2A3D -Dtests.bwc=true -Dtests.locale=ar-JO -Dtests.timezone=Etc/GMT -Druntime.java=23

Applicable branches: main

Reproduces locally?: Yes

Failure history: Failure dashboard for org.elasticsearch.backwards.SearchWithMinCompatibleSearchNodeIT#testMinVersionAsOldVersion&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('testMinVersionAsOldVersion'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.backwards.SearchWithMinCompatibleSearchNodeIT'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure excerpt:

java.lang.AssertionError: timed out waiting for green state for index [test_min_version] cluster state [{
  "cluster_name" : "v7.17.22",
  "cluster_uuid" : "upGvfBNWRviN_O8sn9-pJg",
  "version" : 20502,
  "state_uuid" : "O9zoiGd0ROG-vW7sQjjfVw",
  "master_node" : "snTotwKFQZ6zXVHyCSOMIw",
  "blocks" : { },
  "nodes" : {
    "L-A50_QORHaisFrnjXNHwg" : {
      "name" : "v7.17.22-0",
      "ephemeral_id" : "1lsPVfVuRZSK_UPXSOJbUQ",
      "transport_address" : "127.0.0.1:36557",
      "attributes" : {
        "ml.allocated_processors_double" : "32.0",
        "upgraded" : "true",
        "ml.machine_memory" : "126604296192",
        "xpack.installed" : "true",
        "transform.config_version" : "10.0.0",
        "testattr" : "test",
        "ml.config_version" : "12.0.0",
        "ml.max_jvm_size" : "536870912",
        "ml.allocated_processors" : "32"
      },
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ]
    },
    "snTotwKFQZ6zXVHyCSOMIw" : {
      "name" : "v7.17.22-2",
      "ephemeral_id" : "MEXPuMTbSqqQyMRjTSzXRA",
      "transport_address" : "127.0.0.1:33727",
      "attributes" : {
        "testattr" : "test",
        "ml.machine_memory" : "126604296192",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "536870912",
        "transform.node" : "true"
      },
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ]
    },
    "aJOu_q1aTuy_bkCdMteq6g" : {
      "name" : "v7.17.22-3",
      "ephemeral_id" : "ItxWXuamR--qLUh94s410A",
      "transport_address" : "127.0.0.1:37087",
      "attributes" : {
        "testattr" : "test",
        "ml.machine_memory" : "126604296192",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "536870912",
        "transform.node" : "true"
      },
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ]
    }
  },
  "metadata" : {
    "cluster_uuid" : "upGvfBNWRviN_O8sn9-pJg",
    "cluster_uuid_committed" : true,
    "cluster_coordination" : {
      "term" : 3,
      "last_committed_config" : [
        "snTotwKFQZ6zXVHyCSOMIw",
        "L-A50_QORHaisFrnjXNHwg",
        "aJOu_q1aTuy_bkCdMteq6g"
      ],
      "last_accepted_config" : [
        "snTotwKFQZ6zXVHyCSOMIw",
        "L-A50_QORHaisFrnjXNHwg",
        "aJOu_q1aTuy_bkCdMteq6g"
      ],
      "voting_config_exclusions" : [ ]
    },
    "templates" : {
      ".monitoring-beats" : {
        "order" : 0,
        "version" : 7140099,
        "index_patterns" : [
          ".monitoring-beats-7-*"
        ],
        "settings" : {
          "index" : {
            "format" : "7",
            "codec" : "best_compression",
            "number_of_shards" : "1",
            "auto_expand_replicas" : "0-1",
            "number_of_replicas" : "0"
          }
        },
        "mappings" : {
          "_doc" : {
            "dynamic" : false,
            "properties" : {
              "interval_ms" : {
                "type" : "long"
              },
              "cluster_uuid" : {
                "type" : "keyword"
              },
              "source_node" : {
                "properties" : {
                  "ip" : {
                    "type" : "keyword"
                  },
                  "transport_address" : {
                    "type" : "keyword"
                  },
                  "host" : {
                    "type" : "keyword"
                  },
                  "name" : {
                    "type" : "keyword"
                  },
                  "uuid" : {
                    "type" : "keyword"
                  }
                }
              },
              "type" : {
                "type" : "keyword"
              },
              "beats_stats" : {
                "properties" : {
                  "beat" : {
                    "properties" : {
                      "host" : {
                        "type" : "keyword"
                      },
                      "name" : {
                        "type" : "keyword"
                      },
                      "type" : {
                        "type" : "keyword"
                      },
                      "uuid" : {
                        "type" : "keyword"
                      },
                      "version" : {
                        "type" : "keyword"
                      }
                    }
                  },
                  "metrics" : {
                    "properties" : {
                      "system" : {
                        "properties" : {
                          "load" : {
                            "properties" : {
                              "1" : {
                                "type" : "double"
                              },
                              "15" : {
                                "type" : "double"
                              },
                              "5" : {
                                "type" : "double"
                              },
                              "norm" : {
                                "properties" : {
                                  "1" : {
                                    "type" : "double"
                                  },
                                  "15" : {
                                    "type" : "double"
                                  },
                                  "5" : {
                                    "type" : "double"
                                  }
                                }
                              }
                            }
                          }
                        }
                      },
                      "beat" : {
                        "properties" : {
                          "handles" : {
                            "properties" : {
                              "limit" : {
                                "properties" : {
                                  "hard" : {
                                    "type" : "long"
                                  },
                                  "soft" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "open" : {
                                "type" : "long"
                              }
                            }
                          },
                          "cpu" : {
                            "properties" : {
                              "total" : {
                                "properties" : {
                                  "ticks" : {
                                    "type" : "long"
                                  },
                                  "time" : {
                                    "properties" : {
                                      "ms" : {
                                        "type" : "long"
                                      }
                                    }
                                  },
                                  "value" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "system" : {
                                "properties" : {
                                  "ticks" : {
                                    "type" : "long"
                                  },
                                  "time" : {
                                    "properties" : {
                                      "ms" : {
                                        "type" : "long"
                                      }
                                    }
                                  }
                                }
                              },
                              "user" : {
                                "properties" : {
                                  "ticks" : {
                                    "type" : "long"
                                  },
                                  "time" : {
                                    "properties" : {
                                      "ms" : {
                                        "type" : "long"
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          },
                          "memstats" : {
                            "properties" : {
                              "rss" : {
                                "type" : "long"
                              },
                              "gc_next" : {
                                "type" : "long"
                              },
                              "memory_alloc" : {
                                "type" : "long"
                              },
                              "memory_total" : {
                                "type" : "long"
                              }
                            }
                          },
                          "cgroup" : {
                            "properties" : {
                              "memory" : {
                                "properties" : {
                                  "mem" : {
                                    "properties" : {
                                      "usage" : {
                                        "properties" : {
                                          "bytes" : {
                                            "type" : "long"
                                          }
                                        }
                                      },
                                      "limit" : {
                                        "properties" : {
                                          "bytes" : {
                                            "type" : "long"
                                          }
                                        }
                                      }
                                    }
                                  },
                                  "id" : {
                                    "type" : "keyword"
                                  }
                                }
                              },
                              "cpu" : {
                                "properties" : {
                                  "cfs" : {
                                    "properties" : {
                                      "period" : {
                                        "properties" : {
                                          "us" : {
                                            "type" : "long"
                                          }
                                        }
                                      },
                                      "quota" : {
                                        "properties" : {
                                          "us" : {
                                            "type" : "long"
                                          }
                                        }
                                      }
                                    }
                                  },
                                  "stats" : {
                                    "properties" : {
                                      "periods" : {
                                        "type" : "long"
                                      },
                                      "throttled" : {
                                        "properties" : {
                                          "ns" : {
                                            "type" : "long"
                                          },
                                          "periods" : {
                                            "type" : "long"
                                          }
                                        }
                                      }
                                    }
                                  },
                                  "id" : {
                                    "type" : "keyword"
                                  }
                                }
                              },
                              "cpuacct" : {
                                "properties" : {
                                  "total" : {
                                    "properties" : {
                                      "ns" : {
                                        "type" : "long"
                                      }
                                    }
                                  },
                                  "id" : {
                                    "type" : "keyword"
                                  }
                                }
                              }
                            }
                          },
                          "info" : {
                            "properties" : {
                              "ephemeral_id" : {
                                "type" : "keyword"
                              },
                              "version" : {
                                "type" : "keyword"
                              },
                              "uptime" : {
                                "properties" : {
                                  "ms" : {
                                    "type" : "long"
                                  }
                                }
                              }
                            }
                          }
                        }
                      },
                      "libbeat" : {
                        "properties" : {
                          "output" : {
                            "properties" : {
                              "read" : {
                                "properties" : {
                                  "bytes" : {
                                    "type" : "long"
                                  },
                                  "errors" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "type" : {
                                "type" : "keyword"
                              },
                              "write" : {
                                "properties" : {
                                  "bytes" : {
                                    "type" : "long"
                                  },
                                  "errors" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "events" : {
                                "properties" : {
                                  "batches" : {
                                    "type" : "long"
                                  },
                                  "duplicates" : {
                                    "type" : "long"
                                  },
                                  "total" : {
                                    "type" : "long"
                                  },
                                  "dropped" : {
                                    "type" : "long"
                                  },
                                  "toomany" : {
                                    "type" : "long"
                                  },
                                  "active" : {
                                    "type" : "long"
                                  },
                                  "failed" : {
                                    "type" : "long"
                                  },
                                  "acked" : {
                                    "type" : "long"
                                  }
                                }
                              }
                            }
                          },
                          "pipeline" : {
                            "properties" : {
                              "clients" : {
                                "type" : "long"
                              },
                              "events" : {
                                "properties" : {
                                  "total" : {
                                    "type" : "long"
                                  },
                                  "filtered" : {
                                    "type" : "long"
                                  },
                                  "dropped" : {
                                    "type" : "long"
                                  },
                                  "active" : {
                                    "type" : "long"
                                  },
                                  "failed" : {
                                    "type" : "long"
                                  },
                                  "published" : {
                                    "type" : "long"
                                  },
                                  "retry" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "queue" : {
                                "properties" : {
                                  "acked" : {
                                    "type" : "long"
                                  }
                                }
                              }
                            }
                          },
                          "config" : {
                            "properties" : {
                              "module" : {
                                "properties" : {
                                  "running" : {
                                    "type" : "long"
                                  },
                                  "stops" : {
                                    "type" : "long"
                                  },
                                  "starts" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "reloads" : {
                                "type" : "long"
                              }
                            }
                          }
                        }
                      },
                      "apm-server" : {
                        "properties" : {
                          "server" : {
                            "properties" : {
                              "request" : {
                                "properties" : {
                                  "count" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "response" : {
                                "properties" : {
                                  "valid" : {
                                    "properties" : {
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "accepted" : {
                                        "type" : "long"
                                      },
                                      "ok" : {
                                        "type" : "long"
                                      }
                                    }
                                  },
                                  "count" : {
                                    "type" : "long"
                                  },
                                  "errors" : {
                                    "properties" : {
                                      "ratelimit" : {
                                        "type" : "long"
                                      },
                                      "toolarge" : {
                                        "type" : "long"
                                      },
                                      "internal" : {
                                        "type" : "long"
                                      },
                                      "method" : {
                                        "type" : "long"
                                      },
                                      "unauthorized" : {
                                        "type" : "long"
                                      },
                                      "forbidden" : {
                                        "type" : "long"
                                      },
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "closed" : {
                                        "type" : "long"
                                      },
                                      "decode" : {
                                        "type" : "long"
                                      },
                                      "queue" : {
                                        "type" : "long"
                                      },
                                      "validate" : {
                                        "type" : "long"
                                      },
                                      "concurrency" : {
                                        "type" : "long"
                                      }
                                    }
                                  }
                                }
                              },
                              "concurrent" : {
                                "properties" : {
                                  "wait" : {
                                    "properties" : {
                                      "ms" : {
                                        "type" : "long"
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          },
                          "acm" : {
                            "properties" : {
                              "request" : {
                                "properties" : {
                                  "count" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "response" : {
                                "properties" : {
                                  "valid" : {
                                    "properties" : {
                                      "notmodified" : {
                                        "type" : "long"
                                      },
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "accepted" : {
                                        "type" : "long"
                                      },
                                      "ok" : {
                                        "type" : "long"
                                      }
                                    }
                                  },
                                  "request" : {
                                    "properties" : {
                                      "count" : {
                                        "type" : "long"
                                      }
                                    }
                                  },
                                  "count" : {
                                    "type" : "long"
                                  },
                                  "unset" : {
                                    "type" : "long"
                                  },
                                  "errors" : {
                                    "properties" : {
                                      "ratelimit" : {
                                        "type" : "long"
                                      },
                                      "internal" : {
                                        "type" : "long"
                                      },
                                      "toolarge" : {
                                        "type" : "long"
                                      },
                                      "method" : {
                                        "type" : "long"
                                      },
                                      "unavailable" : {
                                        "type" : "long"
                                      },
                                      "forbidden" : {
                                        "type" : "long"
                                      },
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "decode" : {
                                        "type" : "long"
                                      },
                                      "unauthorized" : {
                                        "type" : "long"
                                      },
                                      "notfound" : {
                                        "type" : "long"
                                      },
                                      "closed" : {
                                        "type" : "long"
                                      },
                                      "invalidquery" : {
                                        "type" : "long"
                                      },
                                      "queue" : {
                                        "type" : "long"
                                      },
                                      "validate" : {
                                        "type" : "long"
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          },
                          "decoder" : {
                            "properties" : {
                              "deflate" : {
                                "properties" : {
                                  "content-length" : {
                                    "type" : "long"
                                  },
                                  "count" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "reader" : {
                                "properties" : {
                                  "size" : {
                                    "type" : "long"
                                  },
                                  "count" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "gzip" : {
                                "properties" : {
                                  "content-length" : {
                                    "type" : "long"
                                  },
                                  "count" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "uncompressed" : {
                                "properties" : {
                                  "content-length" : {
                                    "type" : "long"
                                  },
                                  "count" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "missing-content-length" : {
                                "properties" : {
                                  "count" : {
                                    "type" : "long"
                                  }
                                }
                              }
                            }
                          },
                          "processor" : {
                            "properties" : {
                              "metric" : {
                                "properties" : {
                                  "transformations" : {
                                    "type" : "long"
                                  },
                                  "decoding" : {
                                    "properties" : {
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "errors" : {
                                        "type" : "long"
                                      }
                                    }
                                  },
                                  "validation" : {
                                    "properties" : {
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "errors" : {
                                        "type" : "long"
                                      }
                                    }
                                  }
                                }
                              },
                              "sourcemap" : {
                                "properties" : {
                                  "counter" : {
                                    "type" : "long"
                                  },
                                  "decoding" : {
                                    "properties" : {
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "errors" : {
                                        "type" : "long"
                                      }
                                    }
                                  },
                                  "validation" : {
                                    "properties" : {
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "errors" : {
                                        "type" : "long"
                                      }
                                    }
                                  }
                                }
                              },
                              "error" : {
                                "properties" : {
                                  "stacktraces" : {
                                    "type" : "long"
                                  },
                                  "frames" : {
                                    "type" : "long"
                                  },
                                  "transformations" : {
                                    "type" : "long"
                                  },
                                  "decoding" : {
                                    "properties" : {
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "errors" : {
                                        "type" : "long"
                                      }
                                    }
                                  },
                                  "validation" : {
                                    "properties" : {
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "errors" : {
                                        "type" : "long"
                                      }
                                    }
                                  },
                                  "errors" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "transaction" : {
                                "properties" : {
                                  "stacktraces" : {
                                    "type" : "long"
                                  },
                                  "spans" : {
                                    "type" : "long"
                                  },
                                  "frames" : {
                                    "type" : "long"
                                  },
                                  "transformations" : {
                                    "type" : "long"
                                  },
                                  "transactions" : {
                                    "type" : "long"
                                  },
                                  "decoding" : {
                                    "properties" : {
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "errors" : {
                                        "type" : "long"
                                      }
                                    }
                                  },
                                  "validation" : {
                                    "properties" : {
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "errors" : {
                                        "type" : "long"
                                      }
                                    }
                                  }
                                }
                              },
                              "span" : {
                                "properties" : {
                                  "transformations" : {
                                    "type" : "long"
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  },
                  "tags" : {
                    "type" : "keyword"
                  },
                  "timestamp" : {
                    "format" : "date_time",
                    "type" : "date"
                  }
                }
              },
              "beats_state" : {
                "properties" : {
                  "beat" : {
                    "properties" : {
                      "host" : {
                        "type" : "keyword"
                      },
                      "name" : {
                        "type" : "keyword"
                      },
                      "type" : {
                        "type" : "keyword"
                      },
                      "uuid" : {
                        "type" : "keyword"
                      },
                      "version" : {
                        "type" : "keyword"
                      }
                    }
                  },
                  "state" : {
                    "properties" : {
                      "output" : {
                        "properties" : {
                          "name" : {
                            "type" : "keyword"
                          }
                        }
                      },
                      "input" : {
                        "properties" : {
                          "names" : {
                            "type" : "keyword"
                          },
                          "count" : {
                            "type" : "long"
                          }
                        }
                      },
                      "service" : {
                        "properties" : {
                          "name" : {
                            "type" : "keyword"
                          },
                          "id" : {
                            "type" : "keyword"
                          },
                          "version" : {
                            "type" : "keyword"
                          }
                        }
                      },
                      "module" : {
                        "properties" : {
                          "names" : {
                            "type" : "keyword"
                          },
                          "count" : {
                            "type" : "long"
                          }
                        }
                      },
                      "beat" : {
                        "properties" : {
                          "name" : {
                            "type" : "keyword"
                          }
                        }
                      },
                      "host" : {
                        "properties" : {
                          "hostname" : {
                            "type" : "keyword"
                          },
                          "os" : {
                            "properties" : {
                              "build" : {
                                "type" : "keyword"
                              },
                              "family" : {
                                "type" : "keyword"
                              },
                              "version" : {
                                "type" : "keyword"
                              },
                              "platform" : {
                                "type" : "keyword"
                              }
                            }
                          },
                          "name" : {
                            "type" : "keyword"
                          },
                          "architecture" : {
                            "type" : "keyword"
                          }
                        }
                      }
                    }
                  },
                  "timestamp" : {
                    "format" : "date_time",
                    "type" : "date"
                  }
                }
              },
              "timestamp" : {
                "format" : "date_time",
                "type" : "date"
              }
            }
          }
        },
        "aliases" : { }
      },
      ".monitoring-es" : {
        "order" : 0,
        "version" : 7140099,
        "index_patterns" : [
          ".monitoring-es-7-*"
        ],
        "settings" : {
          "index" : {
            "format" : "7",
            "codec" : "best_compression",
            "number_of_shards" : "1",
            "auto_expand_replicas" : "0-1",
            "number_of_replicas" : "0"
          }
        },
        "mappings" : {
          "_doc" : {
            "dynamic" : false,
            "date_detection" : false,
            "properties" : {
              "interval_ms" : {
                "type" : "long"
              },
              "enrich_coordinator_stats" : {
                "properties" : {
                  "executed_searches_total" : {
                    "type" : "long"
                  },
                  "queue_size" : {
                    "type" : "integer"
                  },
                  "remote_requests_current" : {
                    "type" : "long"
                  },
                  "remote_requests_total" : {
                    "type" : "long"
                  },
                  "node_id" : {
                    "type" : "keyword"
                  }
                }
              },
              "ccr_auto_follow_stats" : {
                "properties" : {
                  "number_of_failed_remote_cluster_state_requests" : {
                    "type" : "long"
                  },
                  "recent_auto_follow_errors" : {
                    "type" : "nested",
                    "properties" : {
                      "auto_follow_exception" : {
                        "type" : "object",
                        "properties" : {
                          "reason" : {
                            "type" : "text"
                          },
                          "type" : {
                            "type" : "keyword"
                          }
                        }
                      },
                      "leader_index" : {
                        "type" : "keyword"
                      },
                      "timestamp" : {
                        "type" : "long"
                      }
                    }
                  },
                  "number_of_failed_follow_indices" : {
                    "type" : "long"
                  },
                  "number_of_successful_follow_indices" : {
                    "type" : "long"
                  },
                  "auto_followed_clusters" : {
                    "type" : "nested",
                    "properties" : {
                      "cluster_name" : {
                        "type" : "keyword"
                      },
                      "last_seen_metadata_version" : {
                        "type" : "long"
                      },
                      "time_since_last_check_millis" : {
                        "type" : "long"
                      }
                    }
                  }
                }
              },
              "ccr_stats" : {
                "properties" : {
                  "write_buffer_size_in_bytes" : {
                    "type" : "long"
                  },
                  "leader_global_checkpoint" : {
                    "type" : "long"
                  },
                  "read_exceptions" : {
                    "type" : "nested",
                    "properties" : {
                      "exception" : {
                        "type" : "object",
                        "properties" : {
                          "reason" : {
                            "type" : "text"
                          },
                          "type" : {
                            "type" : "keyword"
                          }
                        }
                      },
                      "retries" : {
                        "type" : "integer"
                      },
                      "from_seq_no" : {
                        "type" : "long"
                      }
                    }
                  },
                  "follower_index" : {
                    "type" : "keyword"
                  },
                  "leader_max_seq_no" : {
                    "type" : "long"
                  },
                  "last_requested_seq_no" : {
                    "type" : "long"
                  },
                  "follower_settings_version" : {
                    "type" : "long"
                  },
                  "successful_write_requests" : {
                    "type" : "long"
                  },
                  "remote_cluster" : {
                    "type" : "keyword"
                  },
                  "outstanding_write_requests" : {
                    "type" : "long"
                  },
                  "total_read_time_millis" : {
                    "type" : "long"
                  },
                  "outstanding_read_requests" : {
                    "type" : "long"
                  },
                  "total_write_time_millis" : {
                    "type" : "long"
                  },
                  "failed_write_requests" : {
                    "type" : "long"
                  },
                  "failed_read_requests" : {
                    "type" : "long"
                  },
                  "fatal_exception" : {
                    "type" : "object",
                    "properties" : {
                      "reason" : {
                        "type" : "text"
                      },
                      "type" : {
                        "type" : "keyword"
                      }
                    }
                  },
                  "bytes_read" : {
                    "type" : "long"
                  },
                  "leader_index" : {
                    "type" : "keyword"
                  },
                  "follower_max_seq_no" : {
                    "type" : "long"
                  },
                  "operations_written" : {
                    "type" : "long"
                  },
                  "write_buffer_operation_count" : {
                    "type" : "long"
                  },
                  "successful_read_requests" : {
                    "type" : "long"
                  },
                  "shard_id" : {
                    "type" : "integer"
                  },
                  "follower_mapping_version" : {
                    "type" : "long"
                  },
                  "follower_aliases_version" : {
                    "type" : "long"
                  },
                  "follower_global_checkpoint" : {
                    "type" : "long"
                  },
                  "total_read_remote_exec_time_millis" : {
                    "type" : "long"
                  },
                  "operations_read" : {
                    "type" : "long"
                  },
                  "time_since_last_read_millis" : {
                    "type" : "long"
                  }
                }
              },
              "cluster_state" : {
                "properties" : {
                  "shards" : {
                    "type" : "object"
                  },
                  "nodes_hash" : {
                    "type" : "integer"
                  },
                  "nodes" : {
                    "type" : "object"
                  },
                  "master_node" : {
                    "type" : "keyword"
                  },
                  "state_uuid" : {
                    "type" : "keyword"
                  },
                  "version" : {
                    "type" : "long"
                  },
                  "status" : {
                    "type" : "keyword"
                  }
                }
              },
              "type" : {
                "type" : "keyword"
              },
              "indices_stats" : {
                "properties" : {
                  "_all" : {
                    "properties" : {
                      "primaries" : {
                        "properties" : {
                          "search" : {
                            "properties" : {
                              "query_total" : {
                                "type" : "long"
                              },
                              "query_time_in_millis" : {
                                "type" : "long"
                              }
                            }
                          },
                          "docs" : {
                            "properties" : {
                              "count" : {
                                "type" : "long"
                              }
                            }
                          },
                          "indexing" : {
                            "properties" : {
                              "index_time_in_millis" : {
                                "type" : "long"
                              },
                              "index_total" : {
                                "type" : "long"
                              }
                            }
                          }
                        }
                      },
                      "total" : {
                        "properties" : {
                          "search" : {
                            "properties" : {
                              "query_total" : {
                                "type" : "long"
                              },
                              "query_time_in_millis" : {
                                "type" : "long"
                              }
                    ...

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[thecoop]

See #109003

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test testMinVersionAsOldVersion (0.4% fail rate)
• [main] 3 failures in step 7.17.23_bwc-snapshots (0.6% fail rate)
• [main] 3 failures in pipeline elasticsearch-pull-request (0.7% fail rate)

Build Scans:

• elasticsearch-periodic #3235 / openjdk23_7.17.23_java-matrix-bwc
• elasticsearch-pull-request #24033 / 7.17.23_bwc-snapshots
• elasticsearch-periodic #3229 / openjdk23_7.17.23_java-matrix-bwc
• elasticsearch-pull-request #23936 / 7.17.23_bwc-snapshots
• elasticsearch-pull-request #23926 / 7.17.23_bwc-snapshots
• elasticsearch-periodic #3223 / openjdk23_7.17.23_java-matrix-bwc
• elasticsearch-periodic #3220 / openjdk23_7.17.23_java-matrix-bwc
• elasticsearch-periodic #3217 / openjdk23_7.17.23_java-matrix-bwc
• elasticsearch-periodic #3196 / openjdk23_7.17.23_java-matrix-bwc
• elasticsearch-periodic #3187 / openjdk23_7.17.23_java-matrix-bwc