Cannot update watcher index settings to set `index.routing.allocation.include.role` as described in the docs

[masseyke]

Problem Description

Our documentation says that you can run watcher only on specific nodes by doing this:

PUT .watches/_settings
{
  "index.routing.allocation.include.role": "watcher"
}

However, in 8.0 .watches became a system index so you get this error if you try that:

"error" : {
"root_cause" : [
{
"type" : "illegal_state_exception",
"reason" : "Cannot override settings on system indices: [.watches*] -> [index.routing.allocation.include.role]"
}
],
"type" : "illegal_state_exception",
"reason" : "Cannot override settings on system indices: [.watches*] -> [index.routing.allocation.include.role]"
},
"status" : 500
}

We need to either fix this feature or remove it (its documentation anyway). A possible fix would be to add index.routing.allocation.include.role, index.routing.allocation.exclude.role, and index.routing.allocation.require.role to the list of valid properties for the /_watcher/settings API.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[smartkathycat]

@masseyke May I know if any progress on this case? We hit the same issue with version 8.12. The current documentation still has the same instructions with updating watcher settings. It is very confusing. https://www.elastic.co/guide/en/elasticsearch/reference/current/how-watcher-works.html#watch-execution

[masseyke]

@smartkathycat I'm sorry you have run into this. We have not made any progress on this yet.