[BUG] [Remote State] Data in remote state repository is stored outside of the base_path

[ikorchynskyi]

Describe the bug

After the migration to remote cluster state the corresponding S3 bucket contains data not only under the specified base_path prefix, but also under the root.

Related component

Storage:Remote

To Reproduce

1. Setup an empty S3 bucket for a remote state
2. Configure the remote state for a cluster:

   
   # Enable Remote cluster state cluster settings:
   cluster.remote_store.state.enabled: true

Remote cluster state repository settings

node.attr.remote_store.state.repository: state node.attr.remote_store.repository.state.type: s3 node.attr.remote_store.repository.state.settings.bucket: my-remote-store node.attr.remote_store.repository.state.settings.base_path: state node.attr.remote_store.repository.state.settings.region: us-east-1

4. Complete a rolling restart of a cluster
5. Check S3 bucket content:

$ aws s3 ls s3://my-remote-store/ --recursive 2024-10-22 21:16:52 1872 -9U1S2V-Rzg/state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/index/5t0Qx14-S7eHxPjp2M-tiQ/5t0Qx14-S7eHxPjp2M-tiQ/metadata922337203685477578292233703072301646962 2024-10-22 21:27:06 2235 -A9zMTlJJm4/state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/index/ZFMebiN2SjS2HUoZ35iZxg/ZFMebiN2SjS2HUoZ35iZxg/metadata922337203685477577692233703072295501192 2024-10-22 21:18:40 2324 -TJ76-0S4XA/state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/index/vmNdhpLBQiK0-uBG18Ut7A/vmNdhpLBQiK0-uBG18Ut7A/metadata922337203685477577892233703072300559702 2024-10-22 20:37:34 20920 -gHMlojzrB4/state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/index/XtSLmDsXTJ6zaEm8qqnwSw/XtSLmDsXTJ6zaEm8qqnwSw/metadata922337203685477577492233703072325220922 2024-10-23 02:02:55 1872 -qj0gdpE0bg/state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/index/XMZF_KcgSr-2aezWtpKx9Q/XMZF_KcgSr-2aezWtpKx9Q/metadata92233720368547757829223370307213001042__2

omitted

2024-10-22 19:59:49 263 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/coordination922337203685475469592233703072347889331 2024-10-22 19:59:49 32 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/custom--data_stream922337203685475469592233703072347888791 2024-10-22 19:59:48 25903 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/custom--index-graveyard922337203685475469592233703072347888321 2024-10-22 19:59:49 12419 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/custom--index_template922337203685475469592233703072347888531 2024-10-22 19:59:49 442 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/custom--ingest922337203685475469592233703072347888241 2024-10-22 19:59:49 293 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/custom--repositories922337203685475469592233703072347888831 2024-10-22 19:59:48 450 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/settings922337203685475469592233703072347890841 2024-10-22 19:59:49 40 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/templates922337203685475469592233703072347889061 2024-10-22 20:25:03 250 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/global-metadata/transient-settings922337203685475440492233703072332737711 2024-10-23 09:35:11 50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745279C92233703071858648924 2024-10-23 09:35:11 50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745279P92233703071858651234 2024-10-23 09:35:10 50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745280C92233703071858662984 2024-10-23 09:35:10 50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745280P92233703071858663514 2024-10-23 09:35:07 50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745281C92233703071858696344 2024-10-23 09:35:07 50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745281P92233703071858697534 2024-10-23 09:35:05 50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745282C92233703071858714844 2024-10-23 09:35:05 50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745282P92233703071858715624 2024-10-23 09:07:51 50071 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745283C92233703071875055404 2024-10-23 09:07:51 50071 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745283P92233703071875056014 2024-10-23 09:05:39 50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745284C92233703071876374394 2024-10-23 09:05:39 50077 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745284P92233703071876374914 2024-10-23 09:05:35 50049 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745285C92233703071876411324 2024-10-23 09:05:35 50049 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745285P92233703071876412264 2024-10-23 09:05:35 50071 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745286C92233703071876416034 2024-10-23 09:05:35 50071 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745286P92233703071876417184 2024-10-23 09:05:34 50071 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745287C92233703071876419004 2024-10-23 09:05:34 50071 state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/manifest/manifest92233720368547757689223372036854745287P92233703071876419864 2024-10-22 20:11:47 2393 tFrhHnC1gLI/state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/index/NmfqSC3cSQ2Rtw55NbwsYQ/NmfqSC3cSQ2Rtw55NbwsYQ/metadata922337203685477578092233703072340692452 2024-10-22 22:39:20 80504 thHTpF3i0OE/state/ZGlhbG9nLW9wZW5zZWFyY2gtc3RhYmxl/cluster-state/KI73-trAR1WFLGmX3SzYmQ/index/Sv_nGZxCTY-H46XUEc49eg/Sv_nGZxCTY-H46XUEc49eg/metadata92233720368547755969223370307225216796__2

omitted

In total, there is

$ aws s3 ls s3://my-remote-store/ | wc -l 182

The amount of extra folders keeps growing over time.

### Expected behavior

Remote state data is stored fully under the `state/` `base_path`

### Additional Details

_No response_

[ikorchynskyi]

The cluster was on 2.15.0 and has been migrated to 2.17.0 before the switch to the remote state store:

$ opensearch-cli --profile my-cluster curl get --pretty --path '/'
{
  "name" : "my-cluster-data-3",
  "cluster_name" : "my-cluster",
  "cluster_uuid" : "KI73-trAR1WFLGmX3SzYmQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.17.0",
    "build_type" : "tar",
    "build_hash" : "8586481dc99b1740ca3c7c966aee15ad0fc7b412",
    "build_date" : "2024-09-13T01:05:31.102264964Z",
    "build_snapshot" : false,
    "lucene_version" : "9.11.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

[dblock]

[Catch All Triage - 1, 2, 3, 4]