Unable to read/delete system data streams

[andreidan]

Elasticsearch Version

All 8.x series

Installed Plugins

No response

Java Version

bundled

OS Version

Darwin

Problem Description

Trying to read or delete a system data stream yields

GET .ds-.fleet-actions-results-2022.12.08-000001/_settings

DELETE _data_stream/.fleet-actions-results

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Data stream(s) [.fleet-actions-results] use and access is reserved for system operations"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Data stream(s) [.fleet-actions-results] use and access is reserved for system operations"
  },
  "status": 400
}

System indices are not affected.

Steps to Reproduce

We only have one feature state that has a system data stream - the fleet feature state - the .fleet-actions-results one.

To reproduce, enroll a fleet agent and have it send some data to fleet (maybe using the system integration). We'll need to perform an action for the .fleet-actions-results system data stream to be created - I navigated to Agents under Management > Fleet and added some tags to the agent (click on the ... on the right hand side for the agent you enrolled).

Have a custom role to be sure we can delete system resources:

POST /_security/role/everything_everywhere_all_at_once
{
  "run_as": [ "test" ],
  "cluster": [ "all" ],
  "indices": [
    {
      "names": [ "*" ],
      "privileges": [ "all" ],
      "allow_restricted_indices": true
    }
  ]
}

And assign this role and superuser to a user.

Using the user with these roles attempt to get the _settings for the backing index or to delete the .fleet-actions-results data stream:

GET .ds-.fleet-actions-results-2022.12.08-000001/_settings

DELETE _data_stream/.fleet-actions-results

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[stefnestor]

Linking earlier escalation: https://github.com/elastic/elasticsearch/issues/89261